Stable Diffusion - how to write the best Prompts… this will surprise you!

Levende Streg
14 Jan 202311:10

TLDRThe video script discusses strategies for crafting effective prompts for Stable Diffusion, an AI image generation tool. It introduces RunDiffusion and as alternatives to Google Colab for running Stable Diffusion and explores prompt composition for outpainting and inpainting. The speaker emphasizes the importance of the first words in a prompt and the use of parentheses and square brackets for weighting. They note the challenges of creating comic book illustrations and dynamic poses with AI, stating that AI is a tool rather than a replacement for artists. The script also covers the benefits of RunDiffusion's Creator's Club for model switching and the potential of for prompt engineering and model selection. The speaker shares personal experiences with using AI-generated art in their workflow and provides tips on prompt engineering, aspect ratio adjustments, and using img2img prompts for background creation and canvas extension.


  • 📝 **Prompt Creation**: The importance of crafting the best prompts for Stable Diffusion is emphasized, focusing on the use of curly brackets {} to highlight the main subject of the image.
  • 🔍 **RunDiffusion**: Introduced as a site that allows for quick setup of Stable Diffusion and is praised for its ease of use and functionality.
  • 🔑 **Prompt Templates**: Discussed as a useful tool for creating prompts, where users can copy and modify text to suit their needs.
  • 🎨 **Art Style Influence**: Mentioned that the first words in a prompt are the most weighted, and the longer the prompt, the less importance is given to the later words.
  • 📈 **Upweighting and Downweighting**: Explained the use of parentheses and square brackets to emphasize or de-emphasize certain aspects of the prompt.
  • 🎭 **Comic Book Illustrations**: Noted the difficulty in creating high-quality comic book illustrations with Stable Diffusion, suggesting it's more efficient to draw them manually.
  • 🤖 **AI as a Tool, Not a Replacement**: Asserted that AI will not replace artists, but rather serve as a tool to assist in the creative process.
  • 🔄 **Model Switching**: Highlighted the ability to switch between models on RunDiffusion, which is beneficial for different types of prompts and tasks.
  • 🌐 ****: Described as a helpful site for prompt engineering, allowing users to specify models and maintain private prompts.
  • 📐 **Aspect Ratio**: Discussed the impact of aspect ratio on the output of Stable Diffusion, with different styles looking better in different ratios.
  • 🖼️ **Img2Img Prompts**: Illustrated how artists can use img2img prompts to refine existing artwork, particularly when dealing with complex elements like hands and dynamic poses.
  • 🧩 **Inpainting and Outpainting**: Detailed the process of inpainting and outpainting, emphasizing the need for detailed explanation and careful handling of visible image parts.

Q & A

  • What is the topic of discussion in the provided transcript?

    -The topic of discussion is about creating the best prompts for Stable Diffusion, exploring alternatives to Google Colab, and understanding how to use AI in creative workflows.

  • What is RunDiffusion and what is its main selling point?

    -RunDiffusion is a site that allows users to set up Stable Diffusion quickly. Its main selling point is that it can set up Stable Diffusion in as little as 3 minutes.

  • How does one use prompt templates on Github?

    -To use prompt templates on Github, one should copy the text and then paste it into the prompt box, making any necessary changes as required.

  • What is the significance of the curly bracket {} in a prompt?

    -The curly bracket {} in a prompt is significant as it represents the main subject or the most important part of the image that the user wants to be depicted.

  • Why are the first words in a prompt considered the most important?

    -The first words in a prompt are considered the most important because they hold the most weight in determining the output of the AI. As the prompt gets longer, the importance of the latter words decreases.

  • What is the difficulty level in creating comic book illustrations with Stable Diffusion?

    -Creating comic book illustrations with clear outlines and colors using Stable Diffusion is more difficult and time-consuming compared to creating photorealistic or 3D styles.

  • Why does the speaker believe that AI will not replace artists?

    -The speaker believes that AI will not replace artists because it is challenging to get precisely what you want with AI, and it cannot replicate an artist's ability to make adjustments based on feedback or visualize strategic content on the fly.

  • What is the advantage of signing up for Creator’s Club on RunDiffusion?

    -The advantage of signing up for Creator’s Club on RunDiffusion is the ability to switch between different models, which can be beneficial for various types of tasks such as txt2img prompting, outpainting, and using trained models.

  • What is the main change in that the speaker found helpful?

    -The main change in that the speaker found helpful is the ability to create the right dimension, play around with aspect ratio, and keep prompts private.

  • Why is it important to use terms that refer to drawing style in prompts?

    -It is important to use terms that refer to drawing style in prompts because using terms like 4K, unreal engine, or aperture blur can confuse the AI. The AI needs detailed descriptions to understand what style it should create.

  • How does the aspect ratio affect the output of Stable Diffusion?

    -The aspect ratio significantly affects the output of Stable Diffusion. Some styles look better in certain aspect ratios, and the AI gives different results based on the aspect ratio specified in the prompt.

  • What is the speaker's current usage of AI art generation in their work for clients?

    -The speaker currently uses AI art generation for about 2% to 5% of their work for clients, but predicts that this percentage will increase as AI technology improves and they become more adept at using it.



🎨 Optimal Prompts for AI Art Generation

The first paragraph introduces the topic of creating effective prompts for Stable Diffusion, an AI art generation tool. It discusses exploring alternatives to Google Colab, assessing prompt performance in different settings, and delving into outpainting and inpainting techniques. The speaker also shares their perspective on AI's role in the creative process, emphasizing that AI is a tool for artists rather than a replacement. The paragraph highlights the ease of using RunDiffusion, a platform for running Stable Diffusion, and the importance of prompt structure, including the use of curly brackets and parentheses for emphasis. It also touches on the challenges of creating comic book illustrations and dynamic poses with AI, and the speaker's personal preference for traditional drawing methods.


🔍 Prompt Engineering and AI Art Tools

The second paragraph provides further insight into prompt engineering for AI art generation, with a focus on the RunDiffusion platform. It mentions an upcoming episode for more information on RunDiffusion and the speaker's experience with various prompt combinations. The paragraph also discusses the importance of specifying the desired art style in the prompt and the challenges of creating certain styles, such as anime, with AI. The speaker then transitions to discussing, another tool for AI art generation, and its features, including the ability to specify dimensions, aspect ratios, and model selection within the prompt. The paragraph concludes with a brief mention of img2img prompts and the speaker's use of AI for background creation and canvas extension in their work.


🖌️ Inpainting and Outpainting with AI Art

The third paragraph delves into the specifics of inpainting and outpainting prompts, which are distinct from img2img or text2img prompts. It emphasizes the need for careful explanation of the visible parts of the image and the importance of addressing each part of the image separately. The speaker shares their approach to outpainting, where they ensure the bounding box includes a significant portion of the area they want to reuse and only a small part of what needs fixing. The paragraph concludes with an invitation for audience feedback and a reminder to embrace creativity without waiting for the perfect moment.



💡Stable Diffusion

Stable Diffusion is an AI model designed for generating images from textual descriptions. It is a part of the broader field of generative AI and is used by artists and designers to create unique visuals. In the video, it is the central tool for exploring how to write effective prompts for image generation, as well as for various techniques like inpainting and outpainting.


Prompts are the textual instructions or descriptions given to AI models like Stable Diffusion to guide the generation of images. They are crucial for determining the style, content, and quality of the output. The video discusses strategies for crafting prompts that yield the best results, emphasizing the importance of the first words in a prompt and the use of brackets and parentheses to adjust the weight of different elements.


RunDiffusion is an online platform that allows users to run Stable Diffusion models easily. It is mentioned as an alternative to Google Colab and is praised for its quick setup and user-friendly interface. The video suggests that it could be a good fit for integrating Stable Diffusion into one's workflow.


Outpainting is a technique where AI is used to generate additional parts of an image that were not originally present. It is a form of image extension that can be used to expand the canvas of an artwork. The video discusses how to compose prompts for outpainting, emphasizing the need for clear instructions to the AI.


Inpainting is the process of using AI to fill in missing or damaged parts of an image. Unlike outpainting, it focuses on restoring areas within the existing canvas. The video explains that inpainting requires careful detailing in prompts to guide the AI in reconstructing the desired parts of the image.

💡AI in Creative Workflow

AI in creative workflow refers to the integration of AI tools like Stable Diffusion into the process of creating art or design. The video explores how AI can be used to enhance productivity and offers insights into how the creator personally uses AI for a small percentage of their client work, predicting an increase as AI technology improves.

💡Comic Book Illustrations

Comic book illustrations are a specific style of visual art characterized by distinct outlines and colors, often used to tell stories in comic books. The video notes that creating comic book-style illustrations with Stable Diffusion is challenging due to the precision required for clean outlines and colors, and it is currently more efficient and enjoyable for the creator to draw these manually.

💡Dynamic Poses

Dynamic poses refer to the active and energetic arrangements of figures in artwork, often depicting movement or action. The video mentions that Stable Diffusion struggles with generating hands and dynamic poses, suggesting that it is quicker and more reliable for the artist to draw these elements themselves.

💡Aspect Ratio

Aspect ratio is the proportional relationship between the width and the height of an image or canvas. The video discusses how different aspect ratios can affect the outcome of image generation with Stable Diffusion, noting that some styles and subjects look better in certain ratios, such as 3:2 for portraits.

💡Img2Img Prompts

Img2Img (image-to-image) prompts are used when an existing image is provided to the AI along with instructions to modify or transform it in some way. The video demonstrates how img2img prompts can be used to adjust elements of an illustration, like changing the background of a character design.

💡NDA (Non-Disclosure Agreement)

An NDA is a legal contract that establishes a confidential relationship between parties, often used to protect sensitive information shared between them. The video mentions signing NDAs in the context of working with clients, particularly when dealing with confidential or proprietary content.


The best prompts for Stable Diffusion can be created by focusing on the most important elements first and using specific formatting techniques.

Alternatives to Google Colab, such as RunDiffusion and, are explored for their unique features and ease of use.

Prompt templates can be found on Github and customized within the prompt box for specific needs.

The use of curly brackets {} in prompts is crucial for defining the key content desired in the generated image.

Parentheses and square brackets can be used to adjust the importance of different elements within the prompt.

Creating photorealistic or 3D styles is easier with Stable Diffusion than comic book illustrations with clear outlines and colors.

The speaker prefers to draw comic book characters themselves due to the current limitations of AI in capturing specific traits.

AI art generation is currently used for a small percentage of the speaker's client work but is expected to increase as AI improves.

AI is viewed as a tool for artists rather than a replacement, with human creativity and adaptability still being irreplaceable.

RunDiffusion offers the ability to switch between models and run Invoke, although it comes with a high subscription cost.

The importance of aspect ratio in achieving desired styles and the impact it has on the output of Stable Diffusion is discussed.

Prompt engineering involves detailed description and careful placement of style references within the prompt for optimal results.

Img2img prompts are used to tweak and fix existing artwork, leveraging the strengths of artists in areas where AI falls short.

Inpainting and outpainting prompts require careful explanation and separation of image parts to guide the AI effectively.

The speaker shares personal experiences using Stable Diffusion for background creation and canvas extension in comic books.

The limitations of AI in handling dynamic poses and hands in artwork are highlighted, suggesting manual drawing as a faster alternative.

The speaker emphasizes the importance of creativity and encourages action, regardless of the perfect moment.