Stable Diffusion + Improved Prompt Matrix = Crazy Loads of AI Art
In this post, I discuss how to create a large, varied number of images using a script with Automatic1111, and how to modify the script for even more large-batch variety. You’ll have the power to step away from your local Stable Diffusion UI while it creates hundreds of images for you to review later (or throw on Instagram, like me). I end touching on Stable Diffusion V2 news.
Enter the (Prompt) Matrix
So you found a great prompt. With each new seed you get delightful results. Now you want to generate and generate and generate to see boat loads of images. One way is, if you’re like me, to push the batch count up to 16 and keep pressing “Generate” in Automatic1111.
Work. Work. Work.
But what if you want hundreds of images generated while you’re away from keyboard or doing something else that doesn’t require your graphics card? There is a way, especially if you’d like some variety with your results.
We’ve all done things the hard way and then discovered there’s an easy way. If there’s a way to make computational life easier, you’d better believe there’s a script that does it.
If you’ve read about the custom scripts in Automatic1111, one you discover right off the top is Improved Prompt Matrix. Now then, a prompt matrix is useful when you’re investigating various tweaks to creating your image (like how much CFG you’re going to push). What Improved Prompt Matrix does is awesome: you can put in special lists to your text prompt of things you want to switch out, and it will generate images based on each item in those lists.
So, for instance, here is a before:
Epic painting by Bastien Lecouffe Deharme and Norman Rockwell and John Singer Sargent of (beautiful elegant femme fatale woman elf goddess in leather holiday flowing dress) , swords, character design, dungeons and dragons, action, beautiful eyes, Dramatic cinematic lighting, ornate leather armor, muscular, aesthetic, Santa hat, intricately detailed, Symmetry, snowy, Christmas, Winter, Hyper-Realistic, Ultra Resolution, gothic, witchcore, 8K, bokeh, shot on Canon 5D, masterpiece oil painting in the style of (ayami kojima)
Negative: Santa Claus, horns, spikes, bats, clubs, car, cars, cropped, diptych, triptych. ugly, 3D, back, frame, framed, robot eyes, disfigured, deformed, horse, watermark, text, toy, figurine, blurry, bad art, bad anatomy, blurred
Euler a, 150 steps, CFG 8, 576x704
Result:
Create a List of Artists to Generate With
Now, I like to try out lots of artists and so with improved prompt matrix selected from the Scripts dropdown (install directions later), I can do the following with angle brackets (<, >) and vertical bars (|):
Epic painting by Bastien Lecouffe Deharme and Norman Rockwell and John Singer Sargent of (beautiful elegant femme fatale woman elf goddess in leather holiday flowing dress) , swords, character design, dungeons and dragons, action, beautiful eyes, Dramatic cinematic lighting, ornate leather armor, muscular, aesthetic, Santa hat, intricately detailed, Symmetry, snowy, Christmas, Winter, Hyper-Realistic, Ultra Resolution, gothic, witchcore, 8K, bokeh, shot on Canon 5D, masterpiece oil painting in the style of (<Mike Mignola | Alphonse Mucha | Ayami Kojima | Don Maitz | Gaston Bussiere | Aaron Horkey | Michael William Kaluta>)
This will create seven images, one for each of the artists in the list (to confirm, you can look in your Web-UI command window because the script will tell you how many are coming):
Some will look quite similar because the same seed is being used. What I see is the 2nd, 3rd, and 6th images stand-out, meaning that those matching artists in the list have interesting and strong styles and, for me, are worth further focus.
If you increase the batch size (say to three) you’ll get 7 x 3 = 21 images. Three seeds over seven images.
You can see the artist style in the columns — the 2nd, Alphonse Mucha, stands out.
Multiple Lists in the Prompt
And you’re not limited to one list with those less-than greater-than angle-brackets. You can have multiple lists and every combination of those lists will be created for each batch. Okay, you can see how this adds up… actually, it multiplies up. If we replace “gothic, witchcore” with a list of keywords like “ <cyanotype | witchcore>”, we’re up to a lot more images (14 per batch):
Epic painting by Bastien Lecouffe Deharme and Norman Rockwell and John Singer Sargent of (beautiful elegant femme fatale woman elf goddess in leather holiday flowing dress) , swords, character design, dungeons and dragons, action, beautiful eyes, Dramatic cinematic lighting, ornate leather armor, muscular, aesthetic, Santa hat, intricately detailed, Symmetry, snowy, Christmas, Winter, Hyper-Realistic, Ultra Resolution, <cyanotype | witchcore>, 8K, bokeh, shot on Canon 5D, masterpiece oil painting in the style of (<Mike Mignola | Alphonse Mucha | Ayami Kojima | Don Maitz | Gaston Bussiere | Aaron Horkey | Michael William Kaluta>)
What’s this good for?
- Generating boat loads of images. Imagine if you have a three element list of one artist, and five element list of a second artist, and a seven element list of keywords to try. For one batch, that’s 3 x 5 x 7 = 105 images for the same seed. Do sixteen batches and yep that’s 105 x 16 = 1,680 images. Great googly moogly!
- Comparing technique — the most obvious being artist. Give a list of artists for a prompt, see which one stands out, and then focus on that artist for lots and lots of images.
- Learning keywords — how does a keyword change the look of your image?
- Comparing strengths — remember anything can be given a percent strength. For instance, if I say “masterpiece by (Mike Mignola:0.25) and Norman Rockwell” the result I get is a 25% strength of Mike Mignola. So you could create a list of strengths for an artist or for a keyword and see what differences you get.
Installing Improved Prompt Matrix
Are you ready? To try this, or other custom scripts, start here:
- https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Custom-Scripts
- Go to the source code for Improved Prompt Matrix: https://github.com/ArrowM/auto1111-improved-prompt-matrix
- If you’re a dev who likes git, clone that down.
- If you don’t know GitHub from gits in a pub, choose the “Code” drop down button and select “Download ZIP” and unzip somewhere.
- Copy the “improved_prompt_matrix.py” Python file into the scripts directory where you run the batch file that starts up the Automatic1111 Web UI.
- I have to do this: restart everything. Restart the batch file, reload the Web UI. There’s a button to scan the script file directory in settings but I’ve had zero luck with it.
Now that Improved Prompt Matrix is installed, you can go to txt2img and create a prompt with the list of items you want to vary between angle brackets. At the bottom of the page, before you select “Generate,” drop down scripts and choose “Improved Prompt Matrix” — now hit “Generate” for beaucoup images.
Confirm in your command line window that it knows how many pictures it intends to generate.
Tip: don’t dork with anything in the web page running the generate. Don’t touch the prompt, don’t switch to another tab or anything. If you do, you lose the ability to interrupt what could be a very long run and you’ll have to restart your command line batch file. Or wait.
Go crazy with the various things you can vary now in your prompt to create variations.
Improving Improved Prompt Matrix to Make It Seedy
The strength of the Improved Prompt Matrix script is that it allows you to compare the results of a lot of variation with the same seed. That’s all well and good for a true matrix, but what if you have a cool prompt and you just want to create a unique seed for each one?
That’s where I was. I had some cool prompts and I just wanted to create hundreds of images at a time. Click “Generate” and go to bed. See you in the morning. So I made two small changes to the original script:
- Changed the name so that it would be obviously different.
- Changed the seed generation not to create rows of the same seed for each batch but just go and create a list of unique seeds.
That second change looks like this:
p.seed = [item for item in range(int(p.seed), int(p.seed) + p.n_iter * len(all_prompts))]
So now, if I do the above 3 batch run with the seedy version of Improved Prompt Matrix, I get a unique seed for each image:
Source code here if you just want to copy the renamed script file into your scripts directory, too: https://github.com/ejrejr/auto1111-improved-prompt-matrix.
Oh and a prompt surprise. When I was doing one of my interations, I decided to throw in some emojis. Why not? One of the emojis was the skull emoji 💀. And I ended up getting skull faces in my results! Okay.
Stable Diffusion V2 Is… Here… er… woot?
To the surprise of absolutely no-one, Stable Diffusion V2 model arrived and it was purged of anything that might cause capital-infusing problems or bad press for Stability AI. The community analysis detailed the removal of celebrities, nakedness, and modern artists and digital art sites. I mused something like this would happen back in Stable Diffusion + MidJourney + DALL-E Have Rung the AI-Art Bell and Stable Diffusion Chaos — Stability AI Unscrambling Eggs.
In the meantime, lots of models have been trained so the community has the ability, if not the raw GPU time, to train with their own data to bring back what they might be missing.
Personally, I haven’t even tried V2 yet. First of all, I’m on an 8GB video card so I think it’d be pressing my luck to get it to run. Secondly, I use a lot of those artists to craft interesting styles. I mean, I’ve made three recent posts analyzing dozens of artists in Stable Diffusion V1.x. I don’t see something to walk towards so I’m staying where I am.
V2 onward will be different than V1. In time, models will come along to bring back some of the joy of V1. Or a whole new open source generative AI system might emerge. Things are move fast and at least in this case, I’m not moving forward for a bit. Eventually I’ll see some compelling reason to jump (hands never looking like a warped jumble of bananas being one).
Crazy Generations
In the meantime, I went crazy with this prompt matrix thing and created hundreds and hundreds of images. I took this as a lesson to learn what folks do to share on Instagram and Facebook and how to use Meta tools for managing and scheduling posts. My pages, if you’re curious:
- Instagram: https://www.instagram.com/rufustheruse.art/
- Facebook: https://www.facebook.com/profile.php?id=100087983326700
I’ve also discovered that about every Instagram art picture published gets comment hustles from folks wanting money to promote pictures. Geez. No thanks.
Anyway, at the moment I’m having fun with a Christmas theme (emissaries, their assassins, and their protectors) and some science-fiction themed images (with the occasional celebrity fan-art, so *pfffhttt!* to you, Stable Diffusion V2).
Cheers.
