Gallery
FLUX

FLUX.1 in Forge UI: Setup Guide with SD/SDXL Tips

FLUX.1 installation Forge UI: How to Install Forge WebUI for Stable Diffusion

In our previous dive into FLUX.1 on ComfyUI, we discovered the model's potential. Now, let's harness that power within Forge's streamlined interface. This guide walks you through setting up FLUX.1 on Forge and explores new options in the latest Forge version to enhance SD and SDXL image outputs. Let's get started!

How to Install Forge 

  1. Install free version control manager git from git-scm.com
  2. In target folder, open cmd terminal and run git clone https://github.com/lllyasviel/stable-diffusion-webui-forge.git  
  3. Run webui-user.bat to download and install dependencies, this finishes the installation and downloads a base SD 1.5 model for tests into \stable-diffusion-webui-forge\models\Stable-Diffusion (this is where you put your other SD/SDXL/FLUX models you download.
FLUX forge interior with people composition
FLUX forge character realistic portrait composition
FLUX forge character interaction realistic portrait composition
FLUX forge character realistic portrait composition
Stable Diffusion FLUX interior realistic render

Updating Forge

To update, run git pull in cmd terminal in your \stable-diffusion-webui-forge\ folder.

FLUX.1 in Forge

FLUX.1 Models and Encoders

There are several versions of FLUX.1 models. Beside schnell (fast distilled model) and dev (for local rendering) there are distilled versions for use with GPU with lower VRAM.

The models are referred to as FP16 or FP8 (floating point),  and quantized NF4 (4-bit non-floating point). BNB ((BitsandBytes means it is a low-bit accelerator) model.  GGUF format is a binary format that is optimized for quick loading and saving of models.

Installation of Models

FLUX dev nf4 in Forge the simplest setup
FLUX dev nf4 in Forge in the simplest setup

For a fast start, you can download  theFLUX model into "\stable-diffusion-webui-forge\models\Stable-diffusion", without the need to download the VAE and encoders separately:

FLUX dev in forge
FLUX dev in forge with encoders and VAE

Standard FLUX dev models need more files to work in Forge. Download the other models into the proper folders:

Comparison of selected FLUX.1 dev models.
Comparison of selected FLUX.1 dev models. Q4_K_S (gguf) has very good results at size of 6.7GB
FLUX schnell comparison of quantized models
Comparison: flux1-dev-bnb-nf4-v2.safetensors | flux1-dev-fp8 | flux1-dev-Q4_K_S.gguf | flux1-dev-Q6_K.gguf

Flux-schnell/dev Differences

Flux-dev has a different (non-commercial) license. This should not affect you if you are an end user running the model locally to generate images (read the license nonetheless). Flux-schnell is very fast (generating nice images in just 3-5 steps) and adheres slightly better to prompts, but Flux-dev offers more creative options since it can use both distilled CFG and normal CFG to affect image output.

ControlNet and FLUX

There are controlnet models for FLUX, which are not yet working in Forge ControlNet at the time of writing this article.

LoRAs and FLUX

To run LoRAs trained for FLUX in Forge, set "Diffusion in Low Bits" to Automatic (fp16 LoRA). When using GGUF models, go with "Q4_K, Q6_K" models (not K_S) for faster render (results may be affected by your VRAM size). Dev and Schnell FLUX models should work.

Hyper-SD LoRAs opens other possibilities for experimentation, read more in this recent article on Hyper-Flux (Hyper-SD).

Errors and Issues

I got black or grey outputs at the end, but the image renders during the steps

You need to add proper VAE for the base model  you use. For flux1-dev-fp8.safetensors model, it will be ae.safetensors from  https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/ae.safetensors

When trying to generate image with FLUX model, you get AssertionError: You do not have CLIP state dict!

"AssertionError: You do not have CLIP state dict!" or "AssertionError: You do not have T5 state dict!" or "AssertionError: You do not have VAE state dict!" means you need to add models to "VAE / Text Encoder" UI list (see Installation of Models). The models need to be in proper folders.

Diffusion in Low Bits and SDXL/SD Experiments

In the new version of Forge there is a dropdown in "Flux" and "All" UI profiles where you can adjust  Diffusion in Low Bits. Interestingly, it will affect all SD models (SD/SDXL/FLUX) and can output a better image with some experimenting. At the time beiing you can choose between Automatic. Automatic (fp16 LoRA), bnb-nf4,  float8-e4m3fn, float8-e4m3fn (fp16 LoRA), bnb-fp4,  float8-e5m2, float8-e5m2 (fp16 LoRA).

Diffusion in Low Bits, fixing consistency errors in the AI generation outputs
SDXL Diffusion in Low Bits - a mode can fix consistency errors (book is repaired in the image)

Note: Diffusion in Low Bits info does not save into the  image meta yet 

Conclusion

While the rapid recent evolution of the Forge UI and the novel FLUX.1 model (another addition to the Stable Diffusion family) has undoubtedly pushed the boundaries of AI art generation, it's essential to acknowledge that the journey is not yet entirely smooth. Users might encounter occasional hiccups due to the software's developmental stage, such as some issues with ControlNet models (which are currently also under development).

Additionally, the speed and hardware requirements of the novel models (FLUX, SD3, AuraFlow) are substantial. However, with the emergence of several distilled models that fit into the VRAM of consumer graphics cards, we can now experiment with the new base model. Fine-tuning and LoRA training further open up possibilities for this new SD model and we will soon see if the FLUX models will be as flexible as SD 1.5 and SDXL models.

FLUX Stable diffusion model in Forge UI
Try Forge. Try FLUX.

References

Updated:

Tagged in:

I am a multimedia artist, composer, computer scientist, and author. In my works, I often deal with the philosophy of art and science and its implications in the creative area. I focus on the research of quantum and particle physics and development of machine learning and artificial intelligence— and I like to write about it. My experimental multimedia projects tend to be on the frontiers of several scientific fields — neuroscience, applied physics, astrophysics, informatics, and data science. You will find my experimental AI/ML models on Civitai (airesearch) to freely test.

You may also like:

Subscribe

Stay connected to make sure you don’t miss anything. Join our newsletter community for artists, designers, and art and science enthusiasts.