SDXL ComfyUI Stability Workflow - What I use internally at Stability for my AI Art
TLDRIn this video, Scott Weather demonstrates how to use SDXL within Comfy, detailing a core workflow for AI art at Stability. He starts with a basic Comfy graph for quality assurance, explains how to load checkpoints, condition clips, and use samplers. He then introduces advanced techniques like using a refiner and conditioning latent noise, encouraging viewers to experiment and get creative with their own workflows.
Takeaways
- 😀 Scott Weather introduces a tutorial on using SDXL within Comfy UI for AI Art at Stability.
- 🔍 The tutorial serves as a starting point for a workflow that can be customized and expanded upon.
- 📈 Scott demonstrates how to create a basic Comfy UI graph for quality assurance in AI Art generation.
- 🖼️ The process involves loading a checkpoint, using conditioners, and setting up positive and negative prompts for image generation.
- 🛠️ Scott explains the importance of setting the right width, height, and target dimensions for the image.
- 🔄 The use of an advanced sampler is crucial for sampling from the conditioned data, with specific settings for steps and seeds.
- 🔍 The refiner is introduced to enhance the base image, working with a different model to improve details.
- 🔄 A unique approach is presented where the refiner is used before the base sampling to condition the latent noise.
- 📝 Scott emphasizes the need to save and manage metadata carefully to protect the workflow when sharing images.
- 🎨 The video concludes with an invitation for viewers to share their own workflows and ideas for further exploration.
- 👍 Scott encourages viewers to engage with the content by liking, commenting, and subscribing for more tutorials.
Q & A
What is the main topic of the video by Scott Weather?
-The main topic of the video is how to use SDXL within Comfy UI for AI Art, specifically focusing on a core workflow that Scott uses internally at Stability.
What does Scott Weather suggest as a starting point for using SDXL in Comfy UI?
-Scott Weather suggests starting with a basic Comfy UI graph for SDXL, which he demonstrates in the video, and then encourages viewers to get creative and explore further.
What is the purpose of the 'checkpoint' in the workflow described by Scott?
-The 'checkpoint' in the workflow serves as the base model that is loaded into the system to start the AI Art generation process.
How does Scott Weather recommend setting the width and height for the AI Art generation?
-Scott Weather recommends setting the width and height to 4096 for the AI Art generation, as it provides a good balance between quality and computational efficiency.
What is the role of 'positive prompts' and 'negative prompts' in the AI Art generation process?
-Positive prompts are used to guide the AI towards generating the desired image, while negative prompts are used to exclude unwanted elements from the final output.
Why does Scott Weather suggest using a 'primitive' node for the prompts?
-Scott Weather suggests using a 'primitive' node for the prompts to simplify the workflow by allowing the reuse of the same text input in multiple areas of the graph.
What is the purpose of the 'advanced sampler' in the workflow?
-The 'advanced sampler' is used to sample from the conditioned data and generate the AI Art. It provides more capabilities and control over the sampling process compared to a basic sampler.
How does Scott Weather handle the refiner in the workflow?
-Scott Weather uses the refiner as an additional step after the initial sampling to improve the quality and detail of the AI Art by working off a different model trained on different content.
What is the significance of the 'steps' in the sampler and refiner?
-The 'steps' in the sampler and refiner represent the number of iterations the AI will perform to refine the image. Adjusting these steps can significantly affect the outcome and quality of the AI Art.
What is the innovative approach Scott Weather introduces with the 'three steps' using the refiner before the base sampling?
-The innovative approach of using the refiner for the first three steps conditions the latent noise before serious sampling begins, which can lead to different and potentially more refined results in the AI Art generation.
Why is it important to manage metadata when sharing images generated with the Comfy UI workflow?
-It is important to manage metadata when sharing images because the metadata can contain the entire graph used to generate the image, which may include proprietary or sensitive information that should not be shared.
Outlines
🚀 Introduction to SDXL in Comfy
Scott Weather introduces the video, focusing on how to use SDXL within Comfy, a tool for quality assurance and stability. He emphasizes that this is a foundational workflow that many users follow, suggesting viewers start here before getting creative. Scott plans to demonstrate a basic setup first, then delve into more complex configurations that are not yet possible in Automatic 1111. He mentions the importance of the core graph for quality assurance and stability, hinting at the complexity that will be built upon throughout the video.
🎨 Setting Up the Basic SDXL Workflow
Scott begins by guiding viewers through loading a checkpoint in Comfy, using nodes to set up the initial configuration. He explains the process of adding a checkpoint, refining the clip, and setting the width and height to 4096. He also discusses the use of positive and negative prompts, demonstrating how to convert these into node inputs for ease of use. Scott uses 'a robot shopping at Walgreens' as a positive prompt and 'rocks' as a negative prompt, explaining the concept of opposites in prompts. He then introduces the advanced sampler and explains its role in the workflow, detailing the steps involved in setting up the sampler and the importance of the start and end steps in the sampling process.
🔍 Enhancing the Workflow with Refinement
Scott continues by introducing the refiner into the workflow, explaining how to load it like another checkpoint. He discusses the differences in clip conditioning for the refiner and the importance of the aesthetic score. Scott sets up a second sampler to work with the refiner, detailing the steps to connect the positive and negative prompts and the model. He emphasizes the need for both positive and negative prompts to function correctly. Scott then demonstrates how to preview the image and discusses the differences between the base sample and the refined output, explaining how the refiner can add detail and improve the image.
🌟 Advanced Techniques and Final Thoughts
In the final part of the video, Scott introduces an advanced technique of adding an extra refining step before the base step, which he describes as a kind of initializer or latent noise conditioner. He explains how this step can influence the final output by conditioning the latent noise early in the process. Scott demonstrates how to set up this additional refiner and how it interacts with the base sampler. He encourages viewers to experiment with different configurations and share their findings. Scott concludes by reminding viewers to protect their work by stripping metadata from images before sharing, and invites feedback and further discussion in the comments.
Mindmap
Keywords
💡SDXL
💡ComfyUI
💡Workflow
💡Checkpoint
💡Refiner
💡Conditioning
💡Sampler
💡Latent Noise
💡VAE Decoder
💡Preview
💡Metadata
Highlights
Introduction to the core workflow used for AI Art at Stability, starting with the basic ComfyUI setup.
Explanation of the importance of starting with a solid foundation before getting creative with the AI Art process.
Demonstration of how to load a checkpoint in ComfyUI for AI Art creation.
Discussion on the use of conditioners in AI Art and how they affect the outcome.
Tutorial on setting up positive and negative prompts for AI Art generation.
The significance of choosing the right width and height for AI Art projects.
Technique of converting prompts to node inputs for efficiency in AI Art workflows.
Introduction of the advanced sampler and its role in the AI Art creation process.
Explanation of the relationship between start and end steps in the sampling process.
The use of DPM++ SDE GPU for sampling and its advantages in AI Art.
How to create a basic ComfyUI graph for AI Art, including the use of a VAE decoder.
The role of the refiner in enhancing the details of AI Art and its integration into the workflow.
Technique of using aesthetic scores in the refiner for better AI Art results.
The concept of passing leftover noise from the base sampler to the refiner for improved results.
Innovative method of conditioning latent noise before starting the main sampling process.
The idea of using an initializer or latent noise conditioner for unique AI Art outcomes.
Final thoughts on the core workflow and encouragement to explore and get creative with AI Art.
Advice on protecting AI Art projects by stripping metadata before sharing images.
Invitation for viewers to share their own workflows and ideas for AI Art creation.
Closing remarks and a reminder of the ongoing exploration and experimentation in AI Art at Stability.