Organizing Your Stable Diffusion LoRA Treasures
Hold on! What’s a LoRA? Well, if you don’t know and you use Stable Diffusion a lot it’s worth learning. You see, in the old days (2022 CE) if we wanted a new style or to generate images of myself, we had two options: embeddings and creating whole new fine-tuned models with DreamBooth. Then LoRAs came along as a way to take what’s unique about training a whole new model and package it up in a much smaller model that you can use with any other of your favorite base models. So they might be 200MB in size, which is better than 6GB. You can download LoRAs off of CivitAI and put them into your models\LoRA directory. Personally, I like ink art, and this is my favorite one:
I go in and out of using one or more LoRAs on the images that I generate with Stable Diffusion. Then when I’m in, I’m really in throwing up multiple LoRAs to fuse them together or seeing what’s new to try out.
Now, I’m using Automatic1111 for my Stable Diffusion jobs. The nice thing is that there is a LoRA section in the txt2image page where you can see all of your LoRAs, even organized by directory (which I do to at least put all the SDXL LoRAs into one place). Which is good to a point. Okay, there’s a LoRA labeled, let see, “rppc” and that makes the image… makes the image… hell, I don’t know. It might as well be labeled WTFIDKLOL. I do not remember what it does.
Furthermore, I don’t remember if it has special trigger words in the prompt to really kick in and do what it was trained to do. Oh man, why didn’t I do a better job of making notes here?!?
I have treasures here, I’m sure, but what they are eludes me. I have all these obscure names and default “Preview” thumbnails staring at me.
There are three things you can do for basic LoRA organization to make your life easier. I’ll cover the three things I did off-and-on over the course of a head-cold infused weekend. 🤧
Hunting the Mysterious LoRA
Information please: LoRAs are pretty slim when it comes to useful end-user metadata. Some might have more info than others. Okay, to check this yourself, hover over a LoRA and see if a little information icon appears (an “i” in a circle). If so, click on that. Up pops a lot of metadata, formatted in JSON. Let’s look at the start of rppc.
Uh huh. Uh huh. Yep, there’s “rppc”… scroll, scroll… ah geez. Lots of training information but nothing to help jog my mind what the heck this does.
Which is common when collecting LoRAs. I haven’t found a tool to keep your LoRAs tidy. Maybe there’s something out there. Until you’re using a LoRA-tidy-maker-upper tool, you can do it manually. Ideally when you first get the LoRA. Otherwise, prepare to do a lot of detective work.
So step one: know the pedigree of your LoRA. Ends up what you download and the name that shows up in Automatic1111 can be very different. There’s an oil painting LoRA on CivitAI that shows up as model_SDXL. Yeah, not helpful. So what kind of detective work can you do?
First, if you don’t know the LoRA quite yet, you can hover over it and select the tool icon. Looks like a hammer and a wrench overlapping each other (visions of The Wall right now). Click on that. Up pops a dialog. Let’s look at an optional area that might help for detective work. It may or may not be there.
For model_SDXL I see it does have an area called “Training Dataset Tags.”
Well that’s a big hint. One tag with a 41 count occurrence: oilpaintsdxl. It happens the number of times that tag shows up in the training set is equal to the dataset size, noted above in the full dialog. To me, that’s a hint that it’s an activation keyword for the LoRA. If I search for that text on CivitAI, I actually get the matching LoRA model:
Oh, interesting. The CivitAI page says that the trigger word is actually “style by nty” — hmm. Okay. Let’s go with that.
I didn’t have such good luck with “rppc” so I had to break out a regular search engine. I searched “rppc lora” in Bing (you read that right) and got a CivitAI article for the LoRA that points me to the full LoRA. Ohhhhh, “rppc” stands for “Real Photo Postcard” — here’s the page:
(Hmm. Bad news. The author has turned off downloads here and moved to generating on Tensor.Art instead. Too bad. I know Tensor.Art is out there but I’m sticking with CivitAI for as long as they are floating above the inky black water.)
So you get the idea. You might have to do some detective work for LoRAs you downloaded four months ago with cryptic names. You’d like to have:
- A nice description for those obscure LoRA names,
- Know the trigger activation text, if any, and
- Perhaps note the original URL of where it came from.
- An idea of what the heck kind of image this LoRA even makes, perhaps relevant to your specific prompting style.
It would be great to not have to look this stuff up every four months. Let’s see what we can do about that.
Demystifying Your LoRAs One-by-One
Here’s what I did one head cold weekend. I first decided to create a reference image that I would use as a basis for each LoRA. I wanted all of the LoRAs to use the same image as a starting point and save away how each LoRA changed the image. Then, when I looked at all of the LoRAs I could appreciate what they did per the base image.
Alright, I would have done things different in retrospect (but maybe not in retro-retrospect). Here’s my base image generated from the below prompt:
The full prompt you can paste into the prompt field and then parse to try yourself:
Professional (cinematic shot:1.6) dark Chiaroscuro effect three-quarter view of elegant goth woman dressed in black and white silk considering vase of dead flowers next to a window letting moonbeams through, BREAK detailed [eyes | pupils | face | teeth ] RAW DSLR High Angle photo shot on Hasselblad 500C Camera film grain, natural skin, [hyper realistic | detailed | intricate | best quality | hyper detailed | ultra realistic | HQ |8K | UHD | 16K | ornate | masterpiece art], sharp focus, bokeh, depth of field, rule of thirds
Negative prompt: (( turned back, face markings, jpeg, close-up, up close, head shot, perfect skin, long neck, mutated, deviantart, cartoon, 3D, poor quality, amateur, portrait, pose, boring, deformed, polydactyl, extra fingers, reversed hands, six fingers)) negativeXL_D
Steps: 37, Sampler: Restart, CFG scale: 8, Seed: 3911677023, Size: 1152x920, Model hash: c967070428, Model: copaxTimelessxlSDXL1_v9, Version: v1.6.0Things of note:
- I’m using the copaxTimeless SDXL model, which I find to be full of variety and LoRA friendly.
- I have that negativeXL_D negative prompt embedding.
My “in retrospect” thoughts on this prompt after having used it on a bunch of LoRAs I had (please read before doing your own work if you’re making a prompt based on the above):
- Turns out very photo realistic. I regretted that at first because a LoRA needs to be very strong to avoid photo-realism. E.g., the Pixar LoRA sure didn’t look like Pixar at all. So I thought I should remove all that “RAW DSLR…” through “…film grain” prompt text.
- Some LoRAs are lighting effects, so having “Chiaroscuro effect” in there conflicts.
- Goth. I like my images of Goth women sure-you-betchya but that keyword ended up making everything dark and dreary and a whole lot of grayscale in some cases. Not great for showcasing LoRA aspects.
- I’d probably lighten up on the negative prompt text some, too. Maybe not have that embedding as well.
- I should have done 1024x1024, a very native SDXL image size that for sure the LoRAs have trained on.
I was well over halfway working through my set of LoRAs when I had this retrospect kick in. And in the end, when I had my retro-retrospect, I decided that the prompt was representative of the random prompts I kick off in large batches, so it’s okay — for me — to see how the LoRAs work against something representative of the prompts I create.
You do you.
Okay, so we have our base prompt with its seed fixed and everything. Here’s what I do for each LoRA:
- Try to track down the origin of the LoRA.
- Find out what the trigger activation keywords are, either through the LoRAs download page or by inferring from any training data set tags that have especially high numeric counts (put them together as one phrase). There may not be any trigger.
- Click that tool icon for the LoRA to do some metadata editing:
- Description: add a one line describing the LoRA if needed (like, oh, “Real Photo Postcard” for rppc).
- Activation text: add the keyword text for activating the LoRA, if there is any. If this is from the “Training dataset tags” you can just click on the tag and it will show up there automatically. Otherwise, you can paste from the LoRAs download page (e.g., on CivitAI, copy the text after “Trigger Words”).
- Notes: add the URL of the download page and anything else you might want to have handy here.
Alright, ready to create your first LoRA image?
- Make sure there aren’t any other LoRAs listed in your prompt / negative prompt.
- Click on the LoRA itself in the LoRA tab of the txt2image page. This should add the LoRA and any entered trigger word (nice!) to the end of the prompt.
- “Generate”
- Once generated, click that LoRAs tool icon again.
- Select “Replace preview” at the bottom of that dialog.
- Your single image you just generated will show up cropped as the new preview image for the LoRA!
- If you click the LoRA again, it should remove the LoRA and any trigger word from the prompt (nice!).
And here’s what I do as well:
- While the current image is busy generating, I go to any new subsequent LoRAs in waiting and spend my time editing that description, activation text, and notes so that I have the activation text ready to generate later.
Okay, so you can go through your LoRAs one-by-one not only making sure that they activation text is now there with the LoRA but also you have a preview image for what the LoRA actually does.
One last edit you can do: preferred weight. Now, this is a slider and I know some LoRAs actually support negative weights. Ah, not going to get that here. But if your find the default of 1.0 not just right for you, can you set it on a scale from > 0 to <= 2. That way when you click on the LoRA in the future and it is added to your prompt it will automatically be set to that weight.
Oh, one more thing: some LoRAs are negative prompt LoRAs. So it will be up to you, if it’s added to the regular prompt, to move them down to the negative prompt. It’s worth noting in the description if it’s a negative LoRA so that you remember to do that.
Results
Okay, what about my results for that above image? Here are snaps from Automatic1111 of all my current preview images for how the LoRAs — at 1.0 weight — affected my base image. For some: not what I expected, but maybe due to my retrospective thoughts.
Note: rather than one big long image I’ve broken it into two rows at a time out of my LoRA gallery. The above image is the base image and all that is different is adding the given LoRA to the prompt at 1.0 weighting (plus any trigger words).
So I’ll pick a few of these for an X/Y/Z Plot script run and see what kind of good trouble I can run into.
Wrap-Up
Note that the edits you make for a LoRA become side-car files next to your LoRA file. E.g., next to rppc.safetensors I now have rppc.json and rppc.png. Make sure all three travel together if you’re moving your hard work around.
I haven’t dug into what kind of metadata LoRAs can have. I should do that. Because I want to complain. There should be a lot more user-facing metadata in there. Like more descriptive text of what this LoRA does. How about a URL to a homepage for the LoRA? Specific trigger words. It’d even be nice to have the ability to embed a thumbnail so that it could have a default thumbnail. Right now LoRA management is messy, especially with obscure and sloppy file names. Maybe it’s all considered super transient and we’ll be using something else soon. If so: let’s learn from this mess. Now I’m off to throw away a bunch of Kleen — oops — facial tissues. Cheers.
