Gallery
Stable Diffusion

SDXL-Lightning: How to Use It for the Best Details

SDXL-Lightning is a diffusion distillation method that allows the generation of images with an extremely low number of steps. In use, it is similar to LCM or Turbo solutions. It comes as open-source LoRA and full UNet weights. We will take a closer look at the LoRA version, which we can apply at any SDXL model. This article is for advanced users with a knowledge of A1111, Forge, and extensions.

You can test SDXL-Lightning within the SDXL model (for example, Cinematix Lightning), or you may download the LoRAs (the link is provided at the end of the article in the References section) into the "Lora" folder. Then, simply insert the LoRA into a prompt in the usual way. 

Usage of LoRAs

SDXL-Lightning comes as 2, 4, and 8 step versions of LoRAs and full models. It is trained at 1024×1024, so it works bes around this resolution. However, you can upscale really fast to a very large image.

SDXL Lightning test LoRAs
Lightning LoRAs in Forge: CFG 1.5, 'Euler A SGMUniform' , wrong number of steps can (and will) produce color artifacts or glitches

In complex scenes, full models perform better than LoRAs with lower steps:

SDXL-Lightning performance with low steps in Stable Diffusion
1st pass, performance of a 2-step model

Downsides

It has somewhat limited flexibility due to lower responsiveness to token weights and the need of the lower CFG. (you can fight this issue to some extent with 'Hires fix'). With higher steps and CFG you will her artifacts fast. On the other hand, it performs well even with several LoRAs at once.

Working in 2-passes in Forge

SDXL-Lightning with 2-pass workflow comparison
SDXL-Lightning with 2-pass workflow (2-step model example)

You will leverage the 'low number of steps feature' in the 2-pass workflow. This enables the creation of high-resolution images with excellent details at a rapid pace. As illustrated in the image below, you can control the overall image quality by balancing the number of steps in the 1st and 2nd passes.

  • Set your prompt, and keep the number of token low for your first experiments
  • Set 'Euler A SGMUniform' or 'LCM' Sampling methods
  • Set sampling steps to 6-9
  • Check 'Hires fix', set it to 1.5 or 2 upscale, Hires steps 6-9, Denoising strength 0.5-0.7 (you can go much lower for some scenes)
  • Set upscaler to "NONE" (you may experiment with Latent, 4x-ClearRealityV1 or other upscaler later)
Hires-fix-tweaks for A1111
Hires-fix-tweaks

At this point, it is useful to have 'Hires fix tweaks' extension ready. Find it in extensions or install it with this URL https://github.com/w-e-w/sd-webui-hires-fix-tweaks

This will allow you to add additional prompts int 2nd Hires pass (A). You may also move LoRAs into 2nd pass when needed (option B can remove them from the main prompt before Hires fix). You max change the final image a lot with this tool, because with this workflow the image is finished in the 2nd pass during 'Hires fix'.

Conclusion

SDXL-Lightning introduces another option to your Stable Diffusion workflow, particularly because of its capability for very fast sketching in the first pass, and lightning-fast upscaling in txt2img and img2img. You may achieve similar results as with LCM models, with a similar limitations, but SDXL-Lightmimg shines when using upscaling modes. When you adjust to different tools and workflows, the results are remarkable.

References

Updated:

Tagged in:

I am a multimedia artist, composer, computer scientist, and author. In my works, I often deal with the philosophy of art and science and its implications in the creative area. I focus on the research of quantum and particle physics and development of machine learning and artificial intelligence— and I like to write about it. My experimental multimedia projects tend to be on the frontiers of several scientific fields — neuroscience, applied physics, astrophysics, informatics, and data science. You will find my experimental AI/ML models on Civitai (airesearch) to freely test.

Subscribe

Stay connected to make sure you don’t miss anything. Join our newsletter community for artists, designers, and art and science enthusiasts.