Generative AI models are becoming increasingly popular, with notable examples such as DALL-E 3, Midjourney, Firefly, and Stable Diffusion XL leading the way in image generation. The release of Stable Diffusion XL (SDXL) marks a significant milestone by enabling the local generation of high-resolution images up to 1024x1024. While this may initially seem like a limitation for those simply seeking to create beautiful images, the true advantage lies in its ability to be fine-tuned for generating images in styles or subjects it was not originally trained on. LoRA, or Low‑Rank Adaptation, represents a breakthrough in machine learning, particularly in the realm of Generative AI. This technology offers an efficient approach to customizing AI models. Unlike traditional methods that require extensive resources for training or fine‑tuning, LoRA focuses on modifying only a small part of an existing model.
For this tutorial, we decided to use Kohya's GUI as it is quite popular in the Stable Diffusion community and easy to use. Go and get Kohya's GUI from Github. First, select images that are tied to a subject. You can have images for multiple subjects, but there should be at least a couple of pictures per subject. When training a LoRA to generate images of real people, we found it can be beneficial to train multiple subjects with one Lora if the images of a particular person are of little quality (e.g. Instagram). This way the model can use the data from other subjects to, for example, increase the skin details of a person's face. A downside is, of course, that representation is less correct, and that in some cases also other features of the person's face can spill over into your generated image. Just keep that in mind during image selection.
Dataset and image quality for SDXL LoRA
In this HowTo, we train a Lora for SDXL so the images should have a resolution of no less than 1024x1024. A higher resolution is desirable, but you will potentially need more VRAM. The source images folder should start with a number, followed by an underscore, followed by the name of your subject. The name of your subject should be the same as the trigger word you want to use for telling SDXL to create an image with the subject in it. We found that it is best to use a combination of characters that do not have a real meaning, to not confuse the model. log / model / source: These folders are used by Kohya's GUI to store the training data. You can leave them empty for now.
Now we need to create a description of each image, called a caption. In Kohya's GUI navigate to Utilities, Caption, WD14‑Captioning. Paste the path to the folder with the images you want to create the captions for and type in .txt as "Caption file extension". Now you need to specify your trigger word again. Type in your trigger word in the "Prefix to add to W14 caption" section, followed by a semicolon and a space. It should look like this: "trigger, ".
Training workflow with Kohya GUI
At the top of the Kohya page, navigate to LoRA, then Training. Source model: pick an SDXL checkpoint you downloaded. Stability AI's SDXL base model can be found on Hugging Face. Then navigate to Folders to specify the folders you created earlier. In the Parameters tab, set embedding space, UNet learning rate to 0.0001, and Network Rank (Dimension) to 128. Neurons in hidden layer of the small neural net in LoRA can be adjusted according to the dataset and hardware. Now hit the Start training button and wait for the training to finish. If you don't want to configure all the settings yourself, you can also download our config file here. Our dataset consists of one subject in three outfits in similar poses. For copyright reasons, we can't show the entire training data, as well as images generated from the trained model. These include the 3 outfits that were displayed in the training data. We used 23 unique images and 5 crops totaling 29 images.
The images you get from your model are highly dependent on your dataset, but also your prompts. We added "teeth" to our negative prompt, whenever we wanted a smile with no teeth or no smile at all. For our positive prompt we used a variety of prompts. We found that the results are highly dependent on the dataset you use, but also on the prompts you use, as well as the style of the image you want to generate. The ability of the model to produce images that display differing settings is evident, and many images we generated have corn in the background.
Practical dataset guidelines and tokens
Our method aims for a high-quality SDXL 1.0 LoRa model using the Kohya SS GUI. One key advantage of SDXL is its permissive license that comes with it. Training LoRAs can seem daunting at first, but when you dive into it, it becomes straightforward. You will need at least 24 GB of VRAM to use Kohya with our configuration. The training data is the backbone of any AI model and deserves careful attention. A single low‑quality image in a set of 20 can dramatically impact the output. Even with larger datasets, this can be a problem. If you have a 400 image style dataset, split between pictures of people, landscapes, etc., go through all images one by one and ask yourself if you’d be happy with an image like that. If yes, check whether the image has something odd that could confuse the model. In general, aim for a dataset of between 15 and 30 images. Adding more images can help increase the model's flexibility, but ensure each extra image provides something new or unique. For example, if you’re training for a style and lack landscape images in that style, adding a landscape image can improve flexibility.
Rare tokens are sequences of letters that Stable Diffusion doesn’t already associate with concepts, such as skw or ukj. These tokens act as blank sheets that can be associated with your concept during training. The location of the rare token inside the caption affects the meaning Stable Diffusion will associate with it. Save all captions in text files with the same names as the images they accompany. You can copy the configs directly in the Lora training tab of the Kohya user interface, without changing other parameters. With these parameters, you should be able to make a model with a good balance of likeness and flexibility in less than 30 epochs.
Training parameters and evaluation
The repeat rate in the training data folder should be set to 3, and the folder name should be “3_” because Kohya reads the repeat rate and class token from the folder name. The most important tweak between runs is the learning rate; for Lora training, values between 3e-6 and 8e-5 are typical. Start with 3e-5 and adjust in future runs if needed. Increasing the learning rate usually speeds up concept learning but can reduce future flexibility. After training, evaluate the model manually by using some captions as prompts. If generated images are too similar to training images for the same prompt, it may indicate overfitting. If the concept is not captured well, or looks different, it may indicate underfitting. Adjust epochs, or tweak data and parameters accordingly.
Compared to SD1.5, SDXL LoRA training generally requires more VRAM and longer training times, but can yield more detailed results thanks to higher resolution. There are multiple base model lineage options and LoRA compatibility considerations depending on which checkpoint families you plan to use. The key is to match the LoRA to a compatible lineage so that the model can utilize the training effectively.
Alternative tools and practical tips
Two major options for SDXL LoRAs are Kohya SS and OneTrainer. Both can work, but Kohya is widely used for its features and community support. Ensure you have Python 3.10 for compatibility. For larger models, Colab or GPU Cloud solutions like RunPod can help if local hardware is insufficient. You can also reference ready-made config jsons to accelerate setup. The general principle remains: a good dataset, careful captioning, and iterative tuning yield the best results. The goal is achieving a balance between likeness and flexibility while avoiding overfitting.
In addition to the image dataset, you can study several model lineages and their LoRA results to understand compatibility patterns. This helps choose a base SDXL model and related LoRA variants that maximize fidelity for your subject while preserving flexibility for variations in pose, lighting, and background.
