Stable Diffusion AI Art — a Tweak to Technique / New Models / Pontifications
I’ve been generating way more images from Stable Diffusion than I need. But I’m having fun mixing up technique and models and seeing what kind of ability each model has. All of this has fed into a story that is short on words but long on pictures that I’ve been posting to my Instagram: Rufus the Ruse Art (@rufustheruse.art). It wraps up with Christmas this year, though it does kick off a second story.
As I wrapped up the current story, I started using certain textual inversion embeddings and a set of new Stable Diffusion models. Let me tell you all about it.
Return to Textual Inversions
First of all, the main strategy change I’ve done lately is to have a variety of textual inversion embeddings going with each model to give some extra boost to what I’m trying to achieve. What am I trying to achieve? I like results that look like illustrations from a story, with lots of characters present. So I’m going for that.
I am using my seed-heavy improved prompt matrix script that I talked about here: Stable Diffusion + Improved Prompt Matrix = Crazy Loads of AI Art.
The basic part of some of my prompts look like this (the angular brackets are used by the script to generate many iterations of the prompt):
Epic < piotr-jablonski | huang-guang-jian | ruan-jia | concept-art | midjourney-style | xyz> [detailed color pencil:photo shoot:0.3] by <Cyril Rolando | Saturno Button | Guillem H. Pongiluppi|Bastien Lecouffe Deharme | Ruan Jia> and (Norman Rockwell:0.5) and (John Singer Sargent:0.5), HQ, 8K, hyper detailed, symmetry, <<Subject text of Prompt Goes Here>> action, cinematic dramatic lighting, bokeh, shot on Canon 5D, masterpiece [oil painting:hyperrealism:0.3] in the style of (Mike Mignola:0.5)
The textual inversion embeddings that I’ve been using, which can be found in the Hugging Face Concepts Library, include the following (all for Stable Diffusion 1.x):
- piotr-jablonski
- huang-guang-jian
- ruan-jia
- concept-art
- midjourney-style
- Xyz
- Hewlett
- Hoi4
- Hoi4-leaders
- Kojima-ayami
- Midjourney-style
- Nixeu
- Wlop-style
If you are “embedded textual inversion whaaaa?” then read up here about adding these to your Automatic1111 environment so that you can use them in your prompt: Embedding New Styles Into Your Stable Diffusion.
There is also another textual inversion I’ve just discovered, bad_prompt, which is meant to go into your negative prompt. I haven’t given it a chance yet to run and show it can clean up messy things like goofy fingers. Proper fingers: crossed.
Many models are being released now. I honestly can’t keep up. Folks post these to Hugging Face and to CivitAI. You can read up here regarding grabbing a model (ensure it’s safe) and using it in your Automatic1111 environment: Stable Diffusion — Investigating Popular Custom Models for New AI Art Image Styles.
SynthWavePunk_V3Alpha — My New Go-to
Whoosh! A combination of three models: SynthWave, InkPunk, and JoMad Diffusion. I love this model for the kind of illustration images I like to go for. This is my 100% go-to for generating images right now. Note there are several version of SynthWavePunk on the CivitAI page — I’m using V3 alpha.
Seek Art MEGA
Yeah, this one does great, too. I’m mean, really, I’m super pleased with a lot of the models coming out lately. This one produces great looking dramatic illustrations and works well with textual inversions. There’s also an associated web site: seek.art.
NovelInkPunkF222
Another one based on InkPunk! Plus the F222 model. Okay, guess what? F222 is one of those full-on naughty-by-nature models again. So that’s why my negative prompt is pretty important for not having T&A everywhere. It’s still there, just not everywhere.
Eldreth’s OG 4060 mix
Elldreth’s OG 4060 mix | Civitai
Interesting results but I had to throw a lot out along the way to wheedle it down to some good images for my prompt that seemed unique compared to others.
Dreamlike
dreamlike-art/dreamlike-diffusion-1.0 · Hugging Face
I feel like I got great colors and great, rich contrast. And a lot of face schmutz for some reason. Maybe it’s a keyword or a textual-inversion, I don’t know, but something in my prompt resulted in a lot of schmutz. But quite striking results.
Stable Diffusion V2.1?
stabilityai/stable-diffusion-2–1-base · Hugging Face
Yeah, okay, I installed the V2.1 512 & 768 models to try them out. V2.1 is the “whoops, we were way too strict in limiting what we trained with out of fear of the bare body slipping in there” release. Even though I had a negative prompt containing all that Emad suggested, I get very mediocre results. I did remember it needs less steps and tried a 2nd time. I know that others are getting good results, but I don’t have the energy to spend here given so many V1.5 custom models are so outstanding out of the box. So until interesting V2.x textual inversions come out, I’ll keep my distance.
Note: You cannot use V1.x textual inversions with V2.x models. You get Python-esque exception barf in your console should you try.
“No” to AI Art? Pontification Ahead.
Skip these musings if you’re mainly interested in technique. Hold my beer, I’m about to pontificate.
All this AI image generation was flying under the general social media radar for the end of summer and most of fall. Not anymore! The mainstream fan has been hit with the capabilities of AI Art and generative AI and people have feels about it, expressed with all the eloquence that social media affords us.
I think the current chaos is fine — it’s a lot to work out. This is now widely available disruptive technology and it’s going to have a huge amount of wake damage. Things are shook-up. Income threatened. There are no easy answers or conclusions. Feelings will be hurt, indignant passions will be blurted, and bad decisions will be made. It’s going to be messy as the path leading out of this slowly emerges, with difficulty and blunders.
Will there be laws passed against AI training like this? Probably. Will they be appealed and abolished? Maybe. Will there be lawsuits? Oh yes, bunches no doubt. Will they have merit? Eh. That’s probably beside the point.
The 100%-for-sure prognosticating I’m willing to make is: generative AI is not going away. AI art image generation is not going to stop. In my opinion, there is no unringing this bell. The technology has too much value and has become part of some folks everyday toolkit.
(From before: Stable Diffusion + MidJourney + DALL-E Have Rung the AI-Art Bell .)
Generative AI is just going to get better and better.
Like, maybe one day it will be able to render good looking hands. Consistently. I swear when Stable Diffusion kicks out a boring picture with perfect hands I’m like, “I’m sharing this! Just look at those hands! So… so… normal!”
The messy chop we’re in currently got a big ker-splash from the Lensa app using Stable Diffusion to generate personalized AI avatar artwork of paying users, I assume by creating a textual inversion off of some of their submitted pictures. Hey, I know I was enamored when I created my likeness, let along the likeness of my sweetheart. None of that joy would have ever existed without Stable Diffusion.
(From before: You, the Multiverse of You, and Stable Diffusion, AI Art of Me — Textual Inversion vs. Dreambooth in Stable Diffusion, and Artful AI Visions of My Sweetheart .)
But once the everyday users sharing the AI image generation hit the mainstream, the protests began about how it was trained off of images from the internet. This level of protests rose enough to scuttle Unstable Diffusion’s Kickstarter to improve upon the limited Stable Diffusion 2.1 model, an effort intending to bring back both missing artists and salacious content. Then their Patreon got nuked. Now they are on plan C (see Unstable Diffusion — AI Art without Limits (equilibriumai.com) ) before, I guess, going full-on crypto-pirate taping GPU cards together on a renegade derelict oil rig-seated kingdom.
(Interesting policy statement from Kickstarter: Our Current Thinking on the Use of AI-Generated Image Software and AI Art (kickstarter.com) )
Growing pains, am I right? Stable Diffusion 2.x and onward will become more of a highly filtered vanilla platform, dependent on community models and textual inversions to add pizzaz to the images. Stability AI can conform and wash their hands of responsibility (’cause their hands are busy shoving investment money into their pockets). Independent model contributors will most likely do so anonymously once the first one gets sued for training on copyrighted images — even if the suit has zero merits. The cost of defense for some hobbyist just goofing around leads to quick capitulation and implied precedence.
I don’t see this converging anytime soon. A lot of people will say provocative meaningless things but the occasional rare person will have new, thoughtful perspectives about the future of creativity and AI. On all sides. I look forward to those insights.
Now, I’m going to take a bit of a holiday rest before illustrating the next untold tale of “Winter Everlasting.” Cheers.
