Embedded Textual Inversion Prompts Meet Snow White, Stable Diffusion txt2img

Embedding New Styles Into Your Stable Diffusion

Eric Richards
7 min readOct 6, 2022

--

(Not much playing around with Stable Diffusion this past week for me due to a sorrowful family gathering. Note: Richmond, Virginia is a wonderful city and it’s a big gift to yourself to stop by the Virginia Museum for Fine Arts. It’s amazing.)

The next new thing I’ve learned using the AUTOMATIC1111 WebUIi local Stable Diffusion is embedded textual inversion prompts. That’s a bunch of odd words put together. It basically means a file you can add that lets you quickly craft a specific style to your prompt using just one keyword. It’s kind of like visual extensibility for your Stable Diffusion install.

Sometimes it is for producing a very specific subject styled in a certain way. So, it can be the subject of your prompt or a way to style your prompt that Stable Diffusion doesn’t currently do… perhaps a particular artist it was never trained on.

Here’s a before and after, with the after adding just the text dark-forest to match an embedding style file added to the system:

Using the dark-forest Embedded Textual Inversion Prompt, Stable Diffusion txt2img

If you have a collection of images that you want to train and create your own textual inversion style: you can! There are existing textual inversions files created by others that you can download, though realize you’ll have to ignore a bunch of experimental ones (mainly, dudes training their own face for Stable Diffusion).

Get You Some Textual Inversions Locally

Here’s one example specific to the WebUI setup. In it, I download a textual inversion called “Midjourney-style” to see if it can Midjourney-up my Stable Diffusion art.

What I like to do at this point is find an existing txt2img that I’ve made and try the new style on it. Currently, it’s kind of easy:

  • Either find your txt2img output directory or in the current WebUI click the folder icon in the txt2img tab to open that folder.
  • In Stable Diffusion WebUI, switch to the PNG Info tab.
  • Find a txt2img file you like and drag it into the PNG Info tab. This will decode the prompt and the settings used to make the image.
  • Select “Send to txt2img”
  • Switch to the txt2img tab and generate to ensure you’re setup to regenerate that image as it was.
  • Now add any .bin file name that you’ve placed into your embeddings folder to see if it changes the image, e.g., if you copied midjourney-style.bin into your embeddings folder, add midjourney-style to your prompt.
  • Generate.
  • Did it make a noticeable difference? Eh, you might need to grab some different images and see if the embedding changes things for you or not. Or move the new style around inside of the prompt.

Note that I see in my command prompt, when I generate with a new file in the embeddings directory, that it confirms the number of textual inversion embeddings it has read in. It’s worth peeking in there to see if that’s happening for you, too, if things aren’t working quite right.

Ensure textual inversion embeddings are being loaded.

You’ll need to try the new textual inversion style in your prompt in different places to see where it can best have impact.

Now then, I had to ask myself: is this really actually doing anything or is changing the prompt just getting a different result? So I had an embedding called hewlett.bin and got a very interesting, modified Snow White from it (below). I moved that .bin file out of the embeddings directory and tried again and got a new (close to the original) result. So yes, it’s doing something. In this case, something major.

The Textual Inversion · AUTOMATIC1111/stable-diffusion-webui Wiki (github.com) wiki page has instructions for creating your own textual inversion .bin file. If you’re an artist, you might go through this to create your own style for Stable Diffusion to use. I’m looking at you, Sudi! There are other places to make this training, too.

Perhaps more important is that Wiki page has some additional textual inversion resources at the end… perhaps a real boon if you’re a, ah, anime fan. I mean… wow. Alright. Not my jam but rock on if it’s yours.

The main call-out is another view into the HuggingFace textual inversion library:

I ran into some trouble with one — I saw it was named glow-forest but the actual source files said dark-forest. I went to the repository and downloaded the .bin file, renamed it to be dark-forest.bin, put it in my embeddings directory, and tried this before and after, retaining the seed between the two:

  • A pacific northwest forest with sword ferns and fir trees and a path going off into the dark of the forest, midnight, full moon, realistic, trending on artstation, highly detailed, by artgerm
  • A dark-forest pacific northwest forest with sword ferns and fir trees and a path going off into the dark of the forest, midnight, full moon, realistic, trending on artstation, highly detailed, by artgerm

The before & after is the picture above at the start of this post.

My advice here is if you see something interesting off of the HuggingFace repository to go into the list of files and download the .bin and ensure that the correct style (what you’ll rename the file to) is confirmed in the token_identfier.txt. Looks like .pt files are available too and I regret that I haven’t tried those yet.

Well.

What the heck. Let’s try one.

Okay, going back to Stable Diffusion Textual Inversion Embeddings (cyberes.github.io) I downloaded the chen-1 style as a .pt file and then copied that file into my embeddings directory. It’s a style and I added it to my Snow White image. Okay, it works. Well, you know what? If you can find a .pt file it might be quicker & easier to get into your embeddings directory than those .bin files. Your mileage and your variance.

Here is a more extensive set of Snow White generations — starting with the original and then adding textual inversion styles downloaded into the embeddings folder:

Original Snow White and Her Embedded Textual Inversion Variations, Stable Diffusion txt2img
  • Original
  • Midjourney-style
  • Anime-AI-being
  • Concept-art
  • Agm-style
  • Chen-1
  • Hewlett
  • Nixeu
  • Pixel-toy

Now, one thought I have on this: if you find an embedded textual inversion prompt (oy) that you like and use often, others that you share your generation settings with won’t be able to reproduce it without installing the same in their embeddings directory.

Stable Diffusion & Such News of Interest

Things I’ve seen pop up during the past week that got my interest:

Techie:

All the best.

--

--

Eric Richards

Technorati of Leisure. Ex-software leadership Microsoft (Office, Windows, HoloLens), Intel Supercomputers, and Axon. https://www.instagram.com/rufustheruse.art