Stable Diffusion Consistent Character Animation Technique - Tutorial

Tobias Fischer
3 Apr 202334:11

TLDRThe video script outlines a technique for creating animations using stable diffusion, focusing on character development and animation. The creator shares their process of developing non-programmer friendly scripts to assist in generating animation sprite sheets. They discuss the steps involved in refining character poses and appearances, including the use of a turntable Laura and open pose for consistency. The tutorial also covers the creation of a reference image and the iteration process to improve coherency and detail in the animations. The end result is a detailed sprite sheet ready for implementation in a game engine.


  • 🎨 The video discusses a technique for creating animations using stable diffusion, with a focus on character animation.
  • πŸ‘Ύ The creator spent hours developing and refining the technique, resulting in a pathfinding attempt and an orc character animation.
  • πŸ“š The tutorial is based on a method shared by XYZ disk on the stable diffusion subreddit, which was modified and expanded upon.
  • πŸ“ The process involves downloading and installing necessary packages, such as OpenCV, Python, NumPy, and REM BG.
  • πŸ–ΌοΈ The creator used a turntable Laura and open pose to achieve consistent characters without a specific character embedding.
  • πŸ› οΈ Helper scripts were created to streamline the animation creation process, making it accessible to non-programmers.
  • 🎬 The animation involves several stages, including initial character design, refining poses, and creating a turntable pose picture.
  • πŸ”„ The process includes iterations to improve coherency and detail, with the use of stable diffusion for polishing and refining the images.
  • 🎞️ The final step involves extracting, cleaning, and potentially interpolating the animation frames to create a smooth and coherent animation.
  • πŸ“± The creator implemented the final sprite sheet in Godot, showing the animation in a game environment.
  • πŸ’‘ The video serves as a guide for those interested in creating their own animations using stable diffusion and related tools.

Q & A

  • What is the main technique discussed in the video?

    -The main technique discussed in the video is creating animations using stable diffusion.

  • How long did it take the speaker to work on the character animation?

    -The speaker spent a few hours working on the character animation.

  • What was the purpose of the scripts created by the speaker?

    -The purpose of the scripts was to help create animations in a non-programmer friendly way, allowing users with no programming experience to utilize them.

  • What was the original idea for the technique discussed in the video?

    -The original idea came from XYZ disk on the stable diffusion subreddit, who showed a technique for using a turntable Laura and open pose to get consistent characters without a specific character embedding.

  • Which packages are required for the project and how to install them?

    -The required packages are opencv, Python, numpy, and REM BG. They can be installed using pip install commands.

  • What is the purpose of the 'workspaces' folder in the project structure?

    -The 'workspaces' folder is used to organize different workspaces for various animations or projects, with each workspace having its own folder named with a unique identifier like '000'.

  • The 'poses' folder contains open pose poses for the control net in the automatic 1111 UI, which are used as a basis for the actual animation generation.


  • How does the speaker improve the character's appearance?

    -The speaker improves the character's appearance by using paint software to draw over the areas that need modification and then running the images through stable diffusion again to refine the details.

  • What is the process for creating a turntable pose picture?

    -The process for creating a turntable pose picture involves generating an image that shows the character from multiple angles, which is then used as a reference for the stable diffusion process.

  • How does the speaker ensure coherency between frames in the animation?

    -The speaker ensures coherency between frames by using an iterate script that polishes the images, increasing the coherency between the left and right images, and by manually adjusting the poses and features as needed.

  • What is the final output of the animation creation process?

    -The final output of the animation creation process is a sprite sheet that contains all the frames of the animation, which can then be implemented into a game or other projects.



🎨 Introduction to Animation Creation with Stable Diffusion

The speaker introduces a technique for creating animations using stable diffusion, showcasing two different results: a character animation and an orc attack. They explain that the process took about two hours and involved developing various techniques. The speaker also mentions creating scripts to aid in the creation of animations, aiming to make them accessible to non-programmers. The inspiration for this project came from a technique shared on the stable diffusion subreddit, which was then modified and expanded upon with the help of additional scripts to generate larger scale animation sprite sheets.


πŸ“ Setting Up the Workspace and Initial Configuration

The speaker provides a step-by-step guide on setting up the workspace for the animation project. This includes creating a 'workspaces' folder, installing necessary Python packages like opencv, numpy, and REM BG, and initializing the workspace with specific settings. They also discuss creating a 'poses' folder within the workspace for storing pose images and mention the use of a reference image for the stable diffusion process. The speaker emphasizes the importance of proper naming and organization of files and folders to ensure the process runs smoothly.


🎭 Fine-Tuning Character Appearance with Stable Diffusion

The speaker delves into the process of refining the character's appearance using stable diffusion. They discuss using a specific Laura embedding and providing prompts to guide the generation process. The speaker also covers the use of control nets, open pose, and various settings to achieve the desired look for the character. They share their approach to iterating and refining the character's features, such as the face and clothing, by using image editing software and stable diffusion to clean up and polish the details.


πŸšΆβ€β™‚οΈ Morphing Poses for Smooth Animation Transitions

The speaker explains how to create smooth transitions between different poses for the animation. They recommend starting with a pose that has been refined and then using weight painting to morph it into the next pose. The process involves adjusting the denoising strength and running the images through stable diffusion to maintain the character's features while altering the pose. The speaker also discusses the use of masks to focus on specific areas, such as the legs, and the importance of ensuring the correct order and orientation of the limbs for a coherent animation.


πŸ”„ Iterating and Enhancing Animation Coherence

The speaker outlines the process of iterating the workspace to improve the coherence of the animation. They discuss the use of the 'iterate.pi' script to refine the images and control the order of the turntable, left, and right images. The speaker explains how to increase coherency by setting parameters in the 'settings.pi' file and running the script with specific configurations. They also share their approach to polishing the images by running them through stable diffusion with adjusted settings to achieve a more cohesive result.


🎨 Post-Processing and Creating the Final Sprite Sheet

The speaker covers the post-processing steps to finalize the animation. They discuss extracting the animations, cleaning them by removing the background, and preparing them for interpolation. The speaker also talks about the optional interpolation process to create smoother transitions and loops in the animation. They explain how to use different tools and settings to interpolate frames and create a sprite sheet. The speaker concludes by discussing the implementation of the sprite sheet in a game engine and shares the final result of their animation project.



πŸ’‘Stable Diffusion

Stable Diffusion is a term used in the context of machine learning and artificial intelligence, referring to a model that generates images from textual descriptions. In the video, it is the primary tool used for creating character animations and poses. The technique involves using textual prompts and a control net to guide the AI in producing consistent character images without a specific character embedding.

πŸ’‘Character Animation

Character Animation refers to the process of creating movement and lifelike motion for characters in digital media, such as video games, movies, or animations. In the context of the video, the creator is using Stable Diffusion to generate a series of images that depict a character's movement, such as walking and standing poses, which are then combined to create an animated sequence.

πŸ’‘Turntable Laura

Turntable Laura is a specific technique mentioned in the video that involves using a model to display a character from multiple angles, akin to a turntable rotating the character to capture different views. This method helps in achieving consistency in character appearance across various poses and animations.

πŸ’‘Control Net

A Control Net in the context of AI-generated images is a tool that guides the generation process by providing a reference or framework for the AI to follow. It helps in maintaining consistency and achieving desired outcomes in the generated images. In the video, the creator uses a Control Net to ensure that the character animations produced by Stable Diffusion adhere to specific poses and characteristics.

πŸ’‘Helper Scripts

Helper Scripts are small programs or scripts designed to assist in automating or simplifying a particular task. In the video, the creator developed a set of helper scripts to streamline the process of generating animations with Stable Diffusion, making it more accessible for users with little to no programming experience.

πŸ’‘Animation Sprite Sheets

Animation Sprite Sheets are collections of images representing the frames of an animation, arranged in a grid-like format. Each frame of the animation is a single image or 'sprite' in the sheet, and when these sprites are displayed in sequence, they create the illusion of movement. In the video, the creator generates a sprite sheet from the series of character animations produced with Stable Diffusion.


Interpolation in the context of animations is a technique used to create smooth transitions between frames by generating intermediate frames that did not originally exist. This process helps to enhance the fluidity and continuity of the animation. In the video, the creator uses interpolation to add more frames between the existing keyframes, resulting in a smoother animation loop.


Godot is an open-source game engine that allows developers to create 2D and 3D games and interactive applications. In the video, the creator mentions implementing the generated sprite sheet in Godot to see the final result of the character animation in a game-like environment.

πŸ’‘θΏ­δ»£ (Iteration)

迭代, or iteration in English, refers to the process of repeating a set of steps with variations to improve a product or result over successive cycles. In the video, the creator goes through multiple iterations of the animation process, refining the character's poses and details to achieve a desired outcome.

πŸ’‘εΌ€ζΊ (Open Source)

开源, or open source, refers to a type of software licensing where the source code is made publicly available, allowing anyone to view, use, modify, and distribute the software freely. In the context of the video, the creator mentions using open-source tools and scripts to facilitate the animation creation process.


The speaker has been working on a technique for creating animations using stable diffusion.

Two different results were showcased: a character animation and an orc attacking animation.

The character animation was a pathfinding attempt, while the orc animation was tutorial-based.

The process involved developing various techniques and utilizing scripts to aid in animation creation.

The original idea came from a technique demonstrated on the stable diffusion subreddit by XYZ disk.

The technique involves using a turntable Laura and open pose for consistent character generation.

The speaker modified and extended the technique, creating helper scripts for larger scale animation sprite sheets.

The GitHub repository mentioned contains all the scripts and a readme on how to run them.

The process requires downloading the code, installing necessary packages, and setting up a workspace.

A reference image and specific poses are used as a starting point for the stable diffusion process.

The speaker iterated over the character design, refining it through multiple stages of stable diffusion.

The speaker used various settings such as CFG scale, denoising strength, and control net weights to achieve desired results.

The process includes creating a turntable pose picture and a reference image named source.png in the workspace.

The speaker emphasized the importance of taking time to perfect the initial character design as it informs the rest of the process.

The speaker demonstrated how to use stable diffusion to polish and refine the character's face and other details.

The process of generating an initial version of each character in their poses was explained, including the use of image to image and pose.

The speaker discussed the use of an iterate script to clean up images and increase coherency between frames.

Interpolation was introduced as an optional step to smooth out cyclical animations and increase frame count.

The final step involved creating a sprite sheet from the generated images, ready for implementation in a game engine like Godot.

The speaker implemented the sprite sheet in Godot and showed the in-game result, demonstrating the practical application of the technique.