How to train SDXL LoRA model

LoRA (Low-Rank Adaptation) is an efficient Stable Diffusion fine-tuning technology, especially suitable for SDXL. It allows you to inject specific concepts, styles, or objects into an SDXL model using relatively few resources and time without having to retrain the entire model. This tutorial will detail how to use kohya-ss/sd-scripts to train the SDXL LoRA model on Windows systems.

Preparation

1. Hardware requirements :

• It is recommended to use an NVIDIA graphics card with at least 12GB of video memory. If you don't have enough video memory, you can try lowering the training resolution or using other optimization techniques.

2. Software environment :

• Python : It is recommended to use Python version 3.10.6 and ensure that the environment variables are configured correctly.

• Git : used to clone training scripts.

• Stable Diffusion WebUI (AUTOMATIC1111 or others) : Although LoRA training itself does not rely on WebUI, it is required for subsequent use of LoRA.

• Training script : It is recommended to use the script in the kohya-ss/sd-scripts warehouse. It is powerful and supports a variety of training methods and parameter configurations.

3. Training data :

• Data quality : Use high-quality, clear, and relevant images.

• Data volume : 50-200 images are usually enough to train a good LoRA.

• Image size : It is recommended to use a resolution of 1024x1024 or 768x1024 , which must be a multiple of 64. If the video memory is limited, the resolution can be reduced appropriately.

• Image Annotation : Carefully label each image using concise and clear labels. For example, if you are training character LoRA, you can use labels such as "person", "man", "woman", "long hair", and "blue eyes".

• Base model selection : Choose an SDXL model that is similar in style to your training data to get better results. For example, when training anime-style LoRA, choose an SDXL model that is mainly anime-style.

Install and configure the training environment

1. Clone the kohya-ss/sd-scripts repository : Open the command line or PowerShell and run the following command to clone the repository:

    git clone https://github.com/kohya-ss/sd-scripts.git
   cd sd-scripts

2. Install dependencies : Run the following command to install the required Python libraries:

    pip install -r requirements.txt

3. Install PyTorch : Select the appropriate PyTorch version to install based on your CUDA version. There are usually installation instructions in the sd-scripts directory.

Prepare training data

1. Create a training folder : Create a folder outside the sd-scripts directory to store training data, such as my_lora_dataset .

2. Organize the data : Create two subfolders under the my_lora_dataset folder:

• train : stores training images.

• log : stores training logs and output files.

3. Label images : Put the images into the train folder and create a corresponding text file (.txt). The file name is the same as the image file name. The text file contains the labels of the images, separated by commas. For example, if the image file name is image001.jpg , image001.txt is created with the content 1girl, long hair, blue eyes .

Configure training parameters

Use the training script sdxl_train_network.py provided by sd-scripts for training. You need to create a configuration file (eg my_lora_config.toml ) to specify training parameters. Here is an example configuration:

 [general]
output_dir = "log" # Output directory pretrained_model_name_or_path = "path/to/your/sdxl_base_model.safetensors" # SDXL base model path train_data_dir = "path/to/my_lora_dataset/train" # Training data path resolution = "1024,1024" # Training Resolution output_name = "my_lora" # LoRA model name [training]
network_module = "networks.lora" # Use LoRA to train network_dim = 128 # LoRA dimension, affects model size and effect network_alpha = 64 # LoRA alpha, affects training speed and effect learning_rate = 1e-4 # Learning rate train_batch_size = 1 # Training batch size , adjust max_train_steps = 1000 according to video memory # Maximum number of training steps save_every_n_steps = 500 # How many steps to save the model [optimizer]
optimizer_type = "AdamW" # Optimizer type

Start training

In the sd-scripts directory, use the following command to start training:

 python sdxl_train_network.py --config=my_lora_config.toml

Using the LoRA model

After training is completed, the LoRA model file (.safetensors format) will be saved in the log directory. Copy it to the models/Lora folder of Stable Diffusion WebUI , and then call it using <lora:model file name:weights> syntax in the prompt word.

Explanation of key parameters :

• network_dim : The dimension of LoRA. The larger the value, the larger the model and the effect may be better, but the training time and file size will also increase.

• network_alpha : LoRA alpha, affects training speed and effect, usually set to half of network_dim .

• learning_rate : learning rate, which affects the model convergence speed and effect.

• train_batch_size : training batch size, adjusted according to video memory.

• max_train_steps : The maximum number of training steps, which determines the training duration.

• resolution : training resolution.

Other tips :

• Regularized images : To prevent overfitting, regularized images can be used. These images are generated using the same base model, but do not contain the concepts you want to train.

• Monitor training using TensorBoard : You can use TensorBoard to monitor the training process to understand the convergence of the model.

• Multi-machine distributed training : For large-scale data sets, multi-machine distributed training can be used to speed up the training process.

I hope the above steps can help you successfully train the SDXL LoRA model . If you have any other questions, please feel free to continue asking.