Generating Some Smiles With Stable Diffusion

Smile! Get Wild with Stable Diffusion — Wildcards and Facial Expressions

This is a quick share of two techniques: (1) using wildcards for variety in your prompt and (2) having specific expressions for anyone you’re creating images of.

Getting Wild with Wildcards

Another way to pump out a lot of variety in Automatic1111 is to use wildcards. This allows a random line from a specific file to be plopped into your prompt. You have to install an extension to A1111 and create some files, so there are a few basic steps to enable this.

First, get the wildcards extension installed in Automatic1111:

1. Within the A1111 WebUI, go to the extensions tab and see if you’ve already installed “stable-diffusion-webui-wildcards” — it’s not on by default so you probably haven’t. If you haven’t, do the following steps.
2. Within that same extensions tab, choose “Available” and then select “Load from:” to get a list of extensions.
3. Early in that list should be “Wildcards script” — select “Install” to put that into your A1111 environment. (AUTOMATIC1111/stable-diffusion-webui-wildcards: Wildcards (github.com) )
4. Go back to the “Installed” extensions area and select “Apply and restart UI.” This restarts your stable diffusion instance. The WebUI won’t be available until the model is reloaded and ready (check your command prompt).
5. At this point, you can reload your WebUI page.

Next, make some wildcards:

1. Within where you run the batch file to start Stable Diffusion running via the command line, there should now be the following sub directory: extensions\stable-diffusion-webui-wildcards\wildcards
2. Go there.
3. Create a few files: fantasyArtist.txt and expression.txt and timeOfDay.txt

Put into your fantasyArtist.txt your list of favorite fantasy artists. A small example:

Saturno Button
Antonio J. Manzanedo
Seb McKinnon
Annie Stegg Gerard
Donato Giancola
Ruan Jia

Put into expression.txt some facial expressions you’d like to see, like:

smiling
laughing
angry
shouting
frowning

Put into timeOfDay.txt some various periods of the day, for example:

midnight
sunrise
noon
golden hour
sunset
twilight

Here’s an example of a prompt that uses all three of those wildcards:

Epic RAW HQ huang-guang-jian [detailed color pencil:photo shoot:0.3] by __fantasyArtist__ and (Norman Rockwell:0.5) and (John Singer Sargent:0.5) of emotional battle between (evil ice goddess armored warrior woman) fighting with (god warrior armored of fire and coal man ), glowing ice woman magic vs burning golden spring man magic, Looking (__expression__ face:1.4), fight, battle, muscular, ornate armor, duo, winter mountain top at __timeOFDay__, Action, cinematic dramatic lighting, 8K, hyper detailed, symmetry, sharp focus, bokeh, shot on Canon 5D,masterpiece [oil painting:hyperrealism:0.3] in the style of (Mike Mignola:0.5)

My negative prompt:

solo, bad_prompt, malformed sword, ((naked)), ((nipples)), ((breasts)), ((midriff)), asymmetric, horns, spikes, bats, clubs, car, cars, Ugly, tiling, poorly drawn hands, poorly drawn feet, poorly drawn face, out of frame, mutation, mutated, extra limbs, extra legs, extra arms, disfigured, deformed, cross-eye, body out of frame, blurry, bad art, bad anatomy, blurred, text, watermark, grainy, cropped, diptych, triptych, 3D, back, frame, framed, robot eyes, disfigured, hands, horse, text, toy, figurine

Note: I have two textual inversion embeddings in use: huang-guang-jian and bad_prompt; you can use the prompts without having those in your embeddings directory, you’ll just get different results from me.

Select something like a batch size of 8 or higher and start generating images based on your favorite size / sampling / CFG. I personally will make portraits at 576x704, Euler A, CFG of 5 (really letting SD riff here) w/ 150 steps (that takes a while) and a batch size of 8 while I’m trying things out.

And you can use wildcards for anything. This is something where if you do a large batch you can get some perhaps very interesting variety. Maybe it’s keywords for visual styles or whether it’s a photo or a watercolor.

Taking It to Eleven — Wild Improved Matrix Mash-up

So previously, I wrote about another way to generate a lot of variety: Stable Diffusion + Improved Prompt Matrix = Crazy Loads of AI Art.

Can you combine wildcards and the improved prompt matrix? Yes, yes you can.

I have another wildcard file called splashArtist.txt — I can’t say these artists affect the results too much but let’s ignore that and try a small modification to the prompt to use the improved prompt matrix script (select from scripts on the main txt2img tab):

Epic RAW HQ huang-guang-jian [detailed color pencil:photo shoot:0.3] by <__splashArtist__ | __fantasyArtist__> and (Norman Rockwell:0.5) and (John Singer Sargent:0.5) of emotional battle between (evil ice goddess armored warrior woman) fighting with (god warrior armored of fire and coal man ), glowing ice woman magic vs burning golden spring man magic, Looking (__expression__ face:1.4), fight, battle, muscular, ornate armor, duo, winter mountain top at __timeOFDay__, Action, cinematic dramatic lighting, 8K, hyper detailed, symmetry, sharp focus, bokeh, shot on Canon 5D,masterpiece [oil painting:hyperrealism:0.3] in the style of (Mike Mignola:0.5)

Look for the angle brackets (< and >). I have it alternating between __splashArtist__ and __fantasyArtist__ as you can see. Now, this will be a discrete iteration through those, vs. the wildcards themselves being random selections. There is a difference. Using a wildcard doesn’t mean generate everything that’s in the file. It means randomly choose one line from the file to use. The prompt matrix syntax means go through everything in here.

Combining the two can lead to even more variety hijinks!

Facial Expressions… What Took Me So Long?

I was generating a lot of images over the 2022 holidays. Like… a lot. Thousands. And experimenting with the vast proliferation of custom models. And I reached a breaking point: I couldn’t take one more person being generated that looked like an over-medicated expressionless soccer dad or soccer mom with a hundred pounds of battle armor slapped onto them. Their vacant expressions were so out of place and just doing me in.

Finally: I started adding expressions to my prompt vs. waiting for random expressions to be generated for me. The above prompt has an example. I don’t know if it’s great syntax, but it gets the job done.

This post helped: I love SD’s ability to express emotions : StableDiffusion (reddit.com).

One thing I learned is that some models require extra strengthening for the expression to work. E.g., this might work for some models: “looking angry face” while others require “(looking angry face:1.6)”. Other models require very little expression hinting — too much and you get gross exaggerations, which can be a thing in and of itself.

Smiling and laughing tend to come out strong, I guess considering how many pictures might have been trained with that. Angry faces usually work as well.

Try experimenting with expressions — even a slight smile or slight frown improves the drugged vacuousness of a bewildered soccer parent warrior. At the minimum, I feel the eyes are now focused on something rather than blankly staring into an existential void… staring back into them.

What Models Have I Been Using Lately?

Certainly not anything related to Stable Diffusion 2.x, which still seems stranded on the sandbar of boringness to me.

I’ve been using:

• SynthwavePunk_V3Alpha
• Dreamlike Diffusion
• seek art MEGA
• protogenX32OfficialR_1

All of these should be on CivitAI.com or Hugging Face. Right now, there’s an explosion of custom models / remixes coming out. More than I can keep up with, for sure.

Next Up…

I’m ramping up the images I’ve been publishing on Rufus the Ruse Art (@rufustheruse.art) and really feel behind for not trying out InvokeAI. Their YouTube videos look quite interesting with-respect-to composition and iteration. And it would be good to have another Stable Diffusion tool environment handy and ready to use. This past week the Automatic1111 GitHub repository was offline due to some of the salacious content linked out of the project’s wiki being against GitHub’s Terms of Service. It’s back, but obviously on notice.

Okay, InvokeAI is in my queue…