Concept Fusion for Art: Exploring Conditioning CLIP Mixing

Combining and fusing concepts is like mixing colors on an artist's palette—it's where the magic happens and generative AI models shine. You will naturally use it for creating environments, illustrations, unique design forms, or surreal scenes—the principle is the same as in the classic media and design. In generative AI, of course, you may use various techniques of prompt engineering or directly ask for the effect in a single prompt. However, as your prompting in natural language becomes more complicated, especially for Flux models (or ELLA), you may lose track in the text of the prompt, and the model will do the same, combining stronger tokens in a way you may not want.

The method can be used for any model, preferably one which allows natural language prompting (this is a suggestion, not a requirement).

I have been experimenting with the fusion of concepts and styles in Comfy UI, and in this article, I will present an interesting method of controlling the output via separate text prompts, without ControlNet or an intricate prompt engineering.

Conditional Mixing: When the Hell Freezes

In Comfy UI, there are nodes called 'Conditioning (Combine)', 'Conditioning (Average)',  and 'Conditioning (Concat)', which are combining text encode node outputs. In the example workflow, I am using Comfyroll Studio node called 'CR Conditioning Mixer' which combines all three (there is not any difference in the output). This image shows the effects of these three ways of mixing conditions/prompts:

• Combine: If the subjects are too dissimilar in a composition, it often tends to create a double exposure-type image. This method is useful for tough combinations, and even if the output is not perfect, you can refine it in another pass (the double-exposure effect may appear only when the prompt is combined for the first time).
• Average: This tends to meld the concepts together to create a consistent visual. The average strength affects only the Average mode.
  
• Concatenate (or "Concat"): Joins the concepts, often with cinematic results. Interestingly, manually joining prompts does not yield the same output, perhaps because guidance plays some role here.
  

As you may see, the technique is useful for combining opposite effects, which could be hard to prompt. You may employ this to affect lighting conditions, camera settings, environment, style, etc. In this image, I am fusing autumn and winter environment to create an atmosphere for a portrait:

You may consecutively combine more than 2 prompts this way (combining styles and subjects is also interesting).

Example Workflows

• You will find example workflows on sandner.art github.

Step-by-Step Process

• Define Initial Prompts: Kickstart the process by crafting detailed and specific prompts that describe the scenes or concepts you want to combine. The more vivid the prompts, the better the AI can understand and blend them—but even very simple prompt will do for starters.
• Set Up Comfy UI Nodes: Create a workflow in Comfy UI with nodes for each initial prompt. These nodes are the building blocks of your concept fusion journey.
• Mix Conditions: Use the Comfy UI node workflow to mix the conditions of the prompts. Connect the nodes in a way that lets the AI model blend the underlying conditions. The specifics will depend on the idea you are working with.
• Generate Outputs: Run the workflow to produce outputs based on the mixed conditions. The AI model will interpret the blended conditions and generate visual combinations that reflect the fusion of the original concepts. Use LoRAs to support the final result (for example style or concept LoRA that fits your theme) .
• Iterate and Refine: Review the outputs and tweak the workflow settings as needed. Adjust strengths or experiment with mixing types. This iterative process allows for rapid experimentation and refinement, helping you create unique and striking combinations. You can create the combinations in "layers", adding one after another:

Benefits of the Method

• Quick Experimentation: Using "separate" prompting in the Comfy UI node workflow enables swift experimentation with concept fusion, making it a breeze to explore new ideas and visual combinations—and allowing to use very long descriptive prompts.
• Nuanced Outputs: By mixing conditions rather than editing a single prompt text, the method yields outputs that are more nuanced and coherent, seemingly reflecting a deeper understanding of the underlying concepts and forms.
• Testing Creative Concepts: Artists and designers can use this method to test a concepts, generating outputs that are both innovative and visually captivating.

Conclusion

The method of mixing conditions/prompts using the Comfy UI node workflow offers a powerful approach to concept fusion in generative AI. By leveraging this technique, artists and designers can quickly experiment with the fusion of concepts, leading to the creation of unique and visually appealing outputs. This approach opens up new possibilities for art and design, enabling the generation of novel ideas and visual combinations that go beyond manual text mixing and simple collage or retouching.

References and Other Resources

• Combining styles with concept fusion Flux method (Comfy workflows)
• Basics of Comfy UI