Stable Diffusion — Investigating Popular Custom Models for New AI Art Image Styles
While I’m currently ignoring Stable Diffusion V2, I did decide to look at some other models that people have been sharing that are trained to have different styles or impact.
Okay, let’s talk about these models. My goal, and maybe yours, is to come up with something that has amazingly different results or results you’ve been having a hard time getting in the standard model. There are models to give your image a MidJourney boost or create characters that look like they are from the Arcane Netflix series.
There are a lot of interesting, innocent models out there.
And. Well. There are whole buncha very not-innocent models out there, trained to give you the very best in NSFW capabilities. I actually use one to get some different, unique styling. And I have to give it a bunch of negative prompts so that it doesn’t go directly to Naughty Town. Space marines are not supposed to have exposed buttocks. I’m very sure about that, skidding on alien rocks and all…
So as you’re looking for unique models, just realize you’re going through a cornucopia of NSFW-this and NSFW-that kind of training. You know, these exceptionally motivated folks are moving the technology forward, optimizing the flow, and learning & sharing a lot. Never bet against inspired geeks, whether it’s ASCII art or AI art generators.
So, let’s talk about the following models:
Arcane-Diffusion
- Location: nitrosocke/Arcane-Diffusion · Hugging Face
- Usage: Use arcane style in the prompt.
Ink Punk Diffusion
- Location: Envvi/Inkpunk-Diffusion · Hugging Face
- Usage: Use nvinkpunk in the prompt.
Open Journey
- Location: prompthero/openjourney · Hugging Face
- Usage: Use mdjrny-v4 style in the prompt.
SamDoesArt UltMerge
- Location: jinofcoolnes/sammod · Hugging Face
- Usage: Use samdoesarts style in the prompt.
HassanBlend
- Location: Welcome to Hassans Page! NSFW!!! Like, very, very NSFW.
- Usage: None. Just use your prompt.
Anything V3
- Location: Linaqruf/anything-v3.0 · Hugging Face
- Usage: None. Just use your prompt.
When downloading a model, look for the ckpt file extension or (even better) the safetensor file extension.
Where To Find Models Like These
Personally, I like Reddit and seeing what folks post as examples of a given model. There’s also just going through Hugging Face to see what models are posted (especially the most popular). And then there’s the big list off of rentry.co that’s going to be 97% NSFW models.
- Read through Stable Diffusion Reddit: StableDiffusion (reddit.com)
- Hugging Face: Models — Hugging Face and to see what’s popular — Hugging Face — The AI community building the future.
- Civitai | Share your models
- Rentry: Stable Diffusion Models (rentry.co)
“Is It Safe?”
Having recently done dental surgery, that question is a little triggering.
If you’re worried about safety in downloading a model (yes, be worried, if it’s grabbing any random Python code and running on your machine) use Hugging Face as your model site and before you download, see if there’s an import warning (pickle). You can inspect what the model is pulling in and if it looks okay then you’re good to go.
If the extension is “safetensor” then know it is intended to be a safe model. This is something that Automatic1111 has recently started supporting.
Where Do I Put a Model?
In your Automatic1111 install, there should be a folder called models and a sub-folder called Stable-Diffusion. These models, and their VAEs, go into this folder. Hit the refresh button by the current model in use by Stable Diffusion and then drop down to select any newly arrived model.
VAE What?
(This part is optional.)
Some models have an associated or recommended VAE. A what?
Reddit: What’s a VAE? : StableDiffusion (reddit.com)
Blink. Blink.
Within the magic pipeline of Stable Diffusion, a representation of an image is going around. Within this there is encoding / decoding. A VAE is part of that process. A VAE can increase the aesthetics of a model so that you end up with better eyes and hands (ha!).
There’s a VAE from Stability AI that folks say make for better eyes / hands in V1.5 / V1.4: Updated VAE released by Stability, reproducible before and after + example differences : StableDiffusion (reddit.com).
Read that regarding what to do with a VAE. There’s a VAE directory in Automatic1111. I’ve continued using the old method of making it an adjacent file to the model I want the VAE for. For instance, if the model’s name is model.ckpt then the VAE’s file name, in the same directory, needs to be model.vae.pt .
Two models in this list have suggested VAEs to go with them.
V1.5 Reference Images
The following are reference images of the Sci-Fi and the Emissary Fantasy prompts (they are at the bottom of this post, for reference). This is from Stable Diffusion V1.5 with the Stability AI VAE loaded w/ the model.
Sci-Fi
Christmas Emissary / Fantasy
Arcane Diffusion
Whoooooooooaaaaaa. Awesome. Wow, what a difference this made in cities and in characters.
- Interesting buildings.
- Good looking characters, much like the series. Now, some subjects end up looking like characters from the series.
- In the Sci-Fi prompt, lots of atmospheric / cloud effects which obscures anything in the distance. The prompt can perhaps be modified for this.
- In the Emissary prompt: buckles. Lots of buckles. And the outfits all look similar. Good elf ears. Sometimes elf ears are a disaster. But I like how they look with the Arcane Diffusion model.
- No matching to “looks like” in this model. You get what you get.
I can see using Arcane Diffusion a lot more, especially for any steampunk prompts.
Sci-Fi
Christmas Emissary
Ink Punk
I love it. I saw Ink Punk presented on Reddit and I see it’s hit the popular download chart on Hugging Face. I like it.
- I like the bold style and the flourishes.
- The buildings end up having really interesting but not overbearing details.
- Use of soft gradients looks great.
- No matching to “looks like” in this model, either.
- The Sci-Fi model unfortunately ended up with lots of similar shots from a tight-alleyway point-of-view looking at a single tall building. Not as impressive there.
- Some of the character poses looked redundant image to image to image.
Sci-Fi
Christmas Emissary
Open Journey
This has been trained on MidJourney 4 images to help bring some MJ to SD.
Oh. Oh my. Well that’s not going well on my Sci-Fi prompt. It’s become fixated that I used “Guardians of the Galaxy” in my prompt and now is creating a variety of overblown GotG team-up images. Huh. Okay, I took that out. Now I’m getting more of what I expected — Sci-Fi city shots.
- Variety of city images, not a lot of people. City images typically have a dominate angle (like vertical) and sometimes smeary alignment with that angle.
- A lot of reddish / orange glows.
- Fixation on movies. Any people it brings in looks like Harrison Ford because I mentioned Blade Runner as an aesthetic keyword.
- No matching, therefore, to “looks like” in this model.
- For my Christmas Emissary prompt: really interesting subjects. Sometimes. Most of the times this prompt kicks out images of a snowy house all lit-up and warm looking. Super details, though, for what does come out.
Sci-Fi
Christmas Emissary
Samdoesarts Ultmerge
This is a merge of several models, leaning towards a more stylized soft-painting model. Especially of young ladies. (Note that Automatic1111 can merge models, should you find several you want to swish together and see what happens.)
- It didn’t get great results with my prompts.
- Probably, to use it properly, the prompts need to be reworked to be more inline with the model’s purpose.
- In the Sci-Fi prompt, the images end up with colorful views down on street level of a very wet street scene.
- There was never a wide-city view. It was down at street level.
- For the Christmas Emissary prompt, lots and lots of images of a quaint snow covered downtown street with reflections off of a wet street.
- A bit of a strike-out for these prompts, but I’ll try simpler prompts in the future.
Sci-Fi
Christmas Emissary
HassanBlend
Okay so this is one of those naughty models.
The plusses here is that female faces tend to be very crisp and clear. Also, somewhat airbrushed. I did not get a lot of contrast in colors / shadows with HassanBlend. Also, I had to obviously deal with a lot of new positive / negative prompts, like wanting the subject to be fully dressed and not naked in various ways. I started adding “glamour shot” to the negative in hopes of avoiding absurd hip angles.
The reason I tried a naughty model like HassanBlend is that people noted you can get very photorealistic, smooth results, in general, out of it. One thing I noticed with my Sci-Fi prompt is that the buildings became very smooth and crisp. In V1.5 the buildings are a bit messy and the neon lighting doesn’t quite line up. In HassanBlend, the buildings looked like something out of a video game like Halo. Much better.
Oh, and less banana hands. Still some horrific hands but less. That’s just observational.
So, if I was running into messy architecture or something else I wanted to be smooth or I wanted airbrushed photorealistic results, I’d be tempted to try HassanBlend and be ready to put in a lot of prompting work to suppress its nature.
- Very photo-realistic results.
- Probably can go super-naughty but I’ve done my best to reign-it in.
- Does (somewhat) incorporate in “looks like” prompts. This is a rarity compared to the other models.
- I like the Sci-Fi buildings out of this a lot — they are smooth and more realistic than some of the messy-noise that comes out of V1.5 directly.
- For the Christmas Emissary — not a lot of keepers. The folks look real and have interesting clothes, but the posing is a bit too much like they are taking a selfie moment for Instagram / TikTok.
Note: in its setup that there is an associated VAE (vae-ft-mse-840000-ema-pruned.ckpt) that you should download and have right next to your model, named the same, except instead of the ckpt extension it should have vae.pt.
Sci-Fi
Christmas Emissary
Anything V3
It’s popular, let’s try it. Well, that’s not a good excuse, but seems as though all these here anime folks are pretty popular right now and I just wanted to see what would happen.
- Yep, there are a bunch of anime women and a few men.
- Interesting results for the Christmas Emissary prompt — it really shined bringing impossibly endowed young ladies into the theme.
- Super disappointing results for the Sci-Fi prompt — the cities looked flat and bland and boring and the characters we’re in very uninteresting outfits.
- Does not incorporate “looks like” prompts.
- I think this model does better if the prompt is adjusted to an anime kind of space. I tried replacing the movies w/ anime science fiction and it’s a little better.
- Strangely, the hands tend to be awesome. I’ll see extra fingers from time to time but I rarely see bunches of gnarled banana-fingers.
Note: there is a VAE that goes with Anything V3.
Sci-Fi
Christmas Emissary
Conclusion?
I think I’ll continue trying models and keep them in mind when I’m looking for a certain sense that a model works best with. I’ll also experiment with crafting prompts to understand what each model needs to really shine.
I’m going to stick to Hugging Face for now because they at least provide a scan / warning regarding any imports a model might bring in.
And since I’m working through sharing a lot of images I’ve created in these batches, look for them on Instagram and Facebook:
- Instagram: https://www.instagram.com/rufustheruse.art/
- Facebook: https://www.facebook.com/profile.php?id=100087983326700
Backup Information — Prompts Used
My prompts for running all these different models are below. When I run these batches, I use Euler a, 8 CFG, 576x704, and 150 steps. Also, I’m using a variant of the Improved Prompt Matrix custom script — I talked about this here: Stable Diffusion + Improved Prompt Matrix = Crazy Loads of AI Art. If you don’t want to use the script, choose one element within each list surrounded by angle brackets (less-than and greater-than) and replace that angle bracket list with your selection.
Sci-Fi Soldier / City Prompt
My original Sci-Fi prompt and negative prompt are as follows — note that I had to remove “guardians of the galaxy” from the prompt later so that I didn’t have Peter Quill and fam everywhere. Oh, and note that it has a custom embedding: EricRi2 — replace that with the name of whomever you want the male soldier to look like.
Epic photoshoot of sci-fi (soldier looks like EricRi2 man) with (beautiful lady soldier looks like <Nataile Portman | Morena Baccarin >) at multicolor cyberpunk alien city, muscular, dramatic cinematic lighting, poster, dynamic, <Autochrome|Backlight|Nebulous|Darkwave|Witchcore|Powerful|lunarpunk|cgsociety|futurism|demoscene>, sunset, HQ, detailed face, serious, sci-fi, action, halo, mass effect, guardians of the galaxy, altered carbon, blade runner, the fifth element, blade runner 2049, alien cityscape, astrophotography, intricate, 8K, hyper detailed, foreboding, cybernetic, hyper realism, shot with Canon 5D, bokeh, masterpiece photograph by (<Ross Tran| Bastien Lecouffe Deharme| Travis Charest| Joao Ruas| Ruan Jia>) and (Norman Rockwell:0.5) and (Jeremy Mann:0.5) and (rudolf herczog:0.33) and (Liam Wong:0.25)
Negative:
Back, Backside, rear, 3D, cartoon, glamour shot, nude, naked, nipples, bikini, lingerie, street, road, cars, back, frame, framed, robot eyes, disfigured, hands, horse, watermark, text, toy, figurine, comic, diptych, triptych, ugly, out of frame, mutation, mutated, extra limbs, extra legs, extra arms, disfigured, deformed, cross-eye, body out of frame, blurry, bad art, bad anatomy, blurred, text, watermark, grainy, cropped
Fantasy Christmas Emissary Prompt
My fantasy prompt, which I’ve been using to create a loose story about Christmas emissaries, their assassin hunters, and their protectors looks like the following:
Stunning [handpainted:photograph:0.3] by <Ruan Jia | Jeremy Mann | Joshua Middleton | Jeff Simpson | Tran Nguyen | Anato Finnstark> and (Norman Rockwell:0.5) and (John Singer Sargent:0.5) of (fit handsome Santa-Workshop elf man warrior god) with (beautiful elegant femme fatale elf goddess fully clothed in leather flowing dress ) , dungeons and dragons, d&d, dnd, action, volumetric light, ornate leather armor, muscular, cinematic, menacing, fantasy, chaotic, intricately detailed, Symmetry, snowy, wet, Winter, Hyper-Realistic, Ultra Resolution, <moody lighting| atmospheric | beautifully lit | HQ| cosmic nebulae| paranormalpunk | darkwave | foreboding | iridesence>, 8K, Christmas, darkwave, goth, witchcore, Baldur’s Gate, bokeh, shot on Canon 5D, masterpiece [oil painting:hyperrealism:0.3] in the style of Mike Mignola
Negative:
Santa Claus, nude, naked, bikini, lingerie, nipples, horns, spikes, bats, clubs, car, cars, Ugly, tiling, poorly drawn hands, poorly drawn feet, poorly drawn face, out of frame, mutation, mutated, extra limbs, extra legs, extra arms, disfigured, deformed, cross-eye, body out of frame, blurry, bad art, bad anatomy, blurred, text, watermark, grainy, cropped, diptych, triptych, 3D, back, frame, framed, robot eyes, disfigured, hands, horse, text, toy, figurine
