Gallery
Stable Diffusion

Creative Composition With Regional Prompter and Latent Couple Manipulations

Creative Composition in Stable Diffusion Using Regional Prompter Extension, Jungle Temple AI Digital Art by Daniel Sandner

We have talked about the ControlNet extension for AUTOMATIC1111 web UI before, which allows you to control composition in Stable Diffusion by openpose, depthmap, normalmap, and other models.
What if you want a gradual blend, or separate a generated image into thematic segments?

Here we can use several available strategies, in this article, we will take a look at Regional Prompter and Latent Couple Manipulations.

Installation: Install both extensions from Extensions/Available/Load from. Find Regional Prompter and Latent Couple Manipulations in the list, install them, and restart AUTOMATIC1111 UI. If you have an older fork of Latent Couple installed, scroll down for the solution.

Do not expect precise results in every render, for more control combine these methods with ControlNet.

Regional Prompter

Regional Prompter divides the area into rectangular segments, which can have different subjects defined by prompts. It solves some issues with prompt engineering and token spilling.

  • Use AND or BREAK at the end of the segment.
  • Divide Ratio controls the number of segments and its fraction. (1,1 means 50/50 division, 1,4 means 20/80, 1,1,1 means 33/33/33) 
  • Base Ratio is the weight of the base segment versus weighth of the region segment. Higher values will tend to override regions.
Parameters (with default setup, dividing into two regions): red cube BREAK blue cube
red cube BREAK blue sphere BREAK green torus
Base setup for three columns

Use Common and Base Prompts

Common prompt is a property common for the objects in the regions. You can check the Use common prompt checkbox. You will enter it in front of the prompt as another region, without adding it into Divide Ratio.

shiny glass surface BREAK red cube BREAK blue sphere BREAK green torus

Base prompt  is a property added to the scene. You can check the Use base prompt checkbox, or use ADDBASE keyword at the beginning of the prompt and BREAK at the end of the base segment. Base prompt tend to override the content of the scene, using the other segments as details. Remember to keep Base Ratio low (0.2) to avoid this.

ADDBASE photo studio, BREAK red sphere BREAK blue cube BREAK green cylinder

Although the procedure is not successful every time, due to the nature of the diffusion process, it can be a powerful technique in your toolbox.
If the objects are of similar shape or theme they tend to blend into one, which is an issue you can use for creative purposes:

Base prompt serves as a definition of the style and lighting

The image is divided into just four regions: jungle, temple, waterfal, and rocky landscape...

More Complicated Compositions With Regional Prompter

Regional Prompter now allows 2D region assignment, dividing the image into more sophisticated shapes. The division into uneven rows and columns may seem simplistic but offers many opportunities for creative compositions. 2D region assignment is  activated by using semicolon (";") in Divide Ratio, first number always means ratio in Y axis:

2,2,2;1,1,1,1 and 2,1,2;2,1,1,1 and 3,2,1;1,2,3 divisions

Use Visualize and make template button to preview the divisions. Use keywords from the template instead of BREAKs, when using 2D region assignment:

photorealistic ADDBASE summer sea sky ADDCOL island ADDROW abandoned boat ADDCOL rocky cliffs

An example of divide ratio 3,2,1;1,2,3 on image above.

Base Ratio

Base Ratio affect the influence of the base prompt. Experiment with fractions when using Base Ratio parameter (there could be a huge difference in output image between 0.2 and 0.25). You can specifically adjust weights of the regions, divided by comma.

Using Loras

Add them at the very end of the prompt. 

Latent Couple Manipulations

How to uninstall the former version of Latent Couple?

Uninstallation of an extension is usually done by deleting the folder with it. Delete the old Latent Couple folder in /extensions/stable-diffusion-webui-two-shot and restart the UI. Then install the new fork version.

Using Color Mask in Latent Couple Manipulations

Similar idea as with Regional Prompter, but the areas can be defined arbitrarily by masks (originally, the segments were also rectangular, now it is an option). You can paint zones, where a prompt takes an effect.

Regions are roughly defined by colors, painted in the leading image

Both techniques are especially powerful when used in combination with ControlNet and many other creative strategies.

These methods are also very interesting when used like img2img controllers.

How to Make a Precise Composition With Latent Couple

In case you go for a precise region setup:

  • Create color mask
  • Enable Latent Couple
  • Fill in prompt regions in Latent Couple, push Prompt info update
  • Enable ControlNet, import the same color mask 
  • Select control_seg Model, leave the Preprocessor empty
Latent Couple and ControlNet setup
How to create texture in stable diffusion with Laten Couple manipulations
AND wooden texture block AND glass textured sphere AND rusty metal texture sphere

How to Create an Indexed Color Mask in GIMP

You may create an indexed color mask in a graphic editor (or in Canvas Editor extension in A1111). Krita does not support indexed color space, but you can also use GIMP for this very easily. 

  • Create a new image (512x512)
  • Paint your mask with vibrant colors. Keep the number of colors low, every color will provide a region mask for Latent Couple Manipulations
  • Change mode Image/Mode/Indexed
  • In the dialog window, set the maximum number of colors (2-x)
  • File/Export as .PNG
  • Import into Latent Couple Manipulations, it will detect the regions by colors

Conclusion

Do not expect definitive results when controlling diffusion this way, for more precise results use a combination of tools.
Sometimes unexpected results can be inspirational—experimenting with regional prompter is fun, especially when creating continuous blends of subjects, themes, or textures.

Updated:

Tagged in:

I am a multimedia artist, composer, computer scientist, and author. In my works, I often deal with the philosophy of art and science and its implications in the creative area. I focus on the research of quantum and particle physics and development of machine learning and artificial intelligence— and I like to write about it. My experimental multimedia projects tend to be on the frontiers of several scientific fields — neuroscience, applied physics, astrophysics, informatics, and data science. You will find my experimental AI/ML models on Civitai (airesearch) to freely test.

You may also like:

Subscribe

Stay connected to make sure you don’t miss anything. Join our newsletter community for artists, designers, and art and science enthusiasts.