Creating Embeddings and Concept Models with Invoke Training - Textual Inversion & LoRAs

30 Mar 202430:41

TLDRThe video script discusses training custom models using open-source scripts for embeddings and concept models. It explains the tokenization process and the importance of model weights in determining the generation process. The script provides a detailed guide on creating datasets, configuring training settings, and using the training application interface. It emphasizes the difference between embeddings and concept models, and demonstrates how to import and use trained embeddings in prompts for desired outputs.


  • πŸ“š Training custom models involves understanding high-level concepts and practical examples.
  • πŸ€– There are two types of tools used in the generation process: embeddings and concept models.
  • 🧠 The generation process is controlled by the prompt, text encoding, model weights, and the interpretation of the prompt.
  • πŸ’‘ Tokenization breaks down the prompt into smaller parts that the system can analyze mathematically.
  • πŸ” Model weights define what is possible to generate, based on the world that has been seen before.
  • 🎨 Embeddings allow efficient manipulation of the prompt layer by creating a new tool to prompt for specific concepts.
  • πŸš€ Concept models extend the base model with new information and concepts, redefining the model's interpretation at a foundational level.
  • πŸ“ˆ Training involves creating a dataset and using open-source scripts to train embeddings or concept models.
  • πŸ› οΈ The training process is adjusted through configurations that control learning rate, data loading, and validation.
  • πŸ“Š Validation images are used to monitor the training progress and determine the most useful step for embedding.
  • πŸ”„ The training scripts and tools will continue to evolve based on user feedback and needs.

Q & A

  • What are the two main types of tools that can be trained using the open-source scripts mentioned in the transcript?

    -The two main types of tools that can be trained are embeddings and concept models.

  • What is tokenization in the context of the generation process?

    -Tokenization is the process of breaking down the prompt into smaller parts or pieces that can be mathematically analyzed by the system.

  • How does the model weights and text encoding influence the generation process?

    -The model weights and text encoding determine the relationship between the numerical tokens and the visual content, essentially shaping the output based on the prompt and model's understanding of those relationships.

  • What is the purpose of creating an embedding?

    -Creating an embedding allows for more efficient manipulation of the prompt layer, consolidating many prompt requirements into a single token that can be used across different models.

  • How does a concept model differ from an embedding?

    -A concept model extends or injects new information and concepts into the base model, redefining how prompts are interpreted at a foundational level, whereas an embedding focuses on manipulating the existing content within the model more effectively.

  • What is the role of pivotal tuning in the training process?

    -Pivotal tuning is an advanced technique that allows for the training of a new embedding specifically designed to work with a particular concept being trained in a concept model, effectively creating a complete structure for use in the generation process.

  • What is the recommended data set size for textual inversion training?

    -A relatively small data set of 10 to 20 images is typically sufficient for textual inversion training.

  • How does the training script's interface help in preparing the data set for training?

    -The training script's interface, or data set tab, is particularly useful for captioning certain data sets required for training concept models, and helps in organizing images even if they are not captioned.

  • What are the benefits of using the 'keep in memory' option during training?

    -Using the 'keep in memory' option allows for faster data loading during training, but it requires sufficient memory or GPU resources.

  • How can the 'shuffle caption delimiter' setting help in the training process?

    -The 'shuffle caption delimiter' setting helps introduce diversity in the captions processed by the system by reorganizing them randomly based on a specified delimiter, making the model's understanding of individual concepts more resilient.

  • What is the purpose of the learning rate in the optimizer configurations?

    -The learning rate determines how aggressively the system should learn new content during the training process; a higher learning rate may lead to quicker learning but with more volatility, while a lower learning rate may result in slower, more stable learning.



πŸ€– Introduction to Custom Model Training

The paragraph introduces the concept of training custom models using open-source scripts available for free. It emphasizes the importance of understanding high-level concepts and provides examples. The discussion focuses on two types of tools used in the generation process: embeddings and concept models. The names of these tools reflect the techniques used to train them. The video script explains the technical aspects of the generation process, including tokenization and text encoding, and uses an analogy of light sources passing through a lens to simplify the understanding of the process.


πŸ“š Understanding Embeddings and Concept Models

This paragraph delves deeper into the roles of embeddings and concept models. It explains that embeddings allow for efficient manipulation of the prompt layer, relying on existing content in the model, while concept models extend the base model with new information and concepts. The process of creating data sets for each type of model is discussed, highlighting the importance of captioning images for concept models and the variation in data set size requirements. The paragraph also touches on the user interface of the open-source script for training models.


πŸ› οΈ Training Configuration and Data Sets

The paragraph provides a step-by-step guide on configuring the training process, including setting up basic configurations, data configurations, and optimizer configurations. It explains the importance of selecting the right data source, using captions effectively, and adjusting settings like resolution and data loading workers. The paragraph also discusses advanced settings and the trade-offs between using more resources for speed or reducing memory requirements.


🎨 Training Progress and Validation

This section discusses the training progress, focusing on the validation process. It explains how to monitor the training run, evaluate the model's outputs, and save the model at different stages. The paragraph describes how to use the validation images to assess the training's effectiveness and choose the most useful step for further use. It also covers how to import the trained embedding into the invoke system for practical application.


🌟 Finalizing and Applying the Trained Embedding

The paragraph concludes the training process by demonstrating how to finalize and apply the trained embedding. It shows the process of selecting the best step from the validation images, importing the embedding into the invoke system, and using it in prompts to generate new content. The comparison between using the new embedding and a standard term like 'watercolor' is highlighted, emphasizing the improved definition and style achieved through the custom training.


πŸš€ Future Training Scripts and Tools

The final paragraph discusses the future of training scripts and tools, emphasizing the importance of user feedback for continuous improvement. It invites users to share their experiences, projects, and challenges in using the training interface, and highlights the ongoing development and evolution of the training scripts. The paragraph ends with a call to action for users to engage with the community for further support and insights.



πŸ’‘Custom Models

Custom Models refer to the unique machine learning models that users can train using open-source scripts provided for free. These models are tailored to specific tasks or styles, such as generating images or text, and can be run locally on a user's machine. In the context of the video, custom models are central to the theme of empowering users with the tools to create personalized AI applications.

πŸ’‘Open-Source Scripts

Open-Source Scripts are publicly available software codes that allow users to train their own models without needing to develop the scripts from scratch. These scripts are free to use and can be modified as per the user's requirements. In the video, the emphasis is on using open-source scripts to facilitate the training of custom models, highlighting the democratization of AI technology.


Embeddings are a type of AI model that can be trained to represent words, phrases, or concepts in a numerical form. These representations capture the semantic meaning of the input data, allowing the model to understand and generate content based on the learned embeddings. In the video, embeddings are used to efficiently manipulate the prompt layer, enabling the AI to generate content that reflects specific styles or subjects.

πŸ’‘Concept Models

Concept Models are AI models that are trained to understand and generate content based on specific concepts or styles. They are extended versions of base models,注ε…₯ or extended with new information and concepts. These models are designed to interpret prompts at a foundational level, allowing for the generation of content that reflects new or unique styles and subjects. In the video, concept models are a key focus, with the speaker discussing how they can be trained to include new information and concepts.


Tokenization is the process of breaking down text or prompts into smaller, mathematically analyzable parts, known as tokens. Each token is assigned an ID, which the AI model uses to understand the relationships between these tokens and generate content accordingly. In the video, tokenization is a crucial step in the generation process, as it allows the model to interpret and respond to user inputs.

πŸ’‘Model Weights

Model Weights refer to the parameters within a machine learning model that determine the output based on the input data. These weights are adjusted during the training process to improve the model's performance. In the context of the video, model weights are critical as they define the possible outputs of the AI, essentially shaping the world the model can generate based on the data it has been trained on.

πŸ’‘Text Encoding

Text Encoding is the process of converting text into a format that can be understood and processed by a machine learning model. It is closely related to tokenization and is a key aspect of how AI models interpret and generate content. In the video, text encoding is part of the generation process where the prompt is prepared for the model to analyze and generate the desired output.


A Prompt is an input provided to an AI model that guides the output it generates. In the context of the video, prompts are textual inputs that, when passed through the system, are tokenized and encoded to influence the model's generation process. The effectiveness of a prompt is influenced by the model's understanding and the structure of the model weights.

πŸ’‘Data Sets

Data Sets are collections of data used for training machine learning models. They typically include examples of the desired output, such as images or text, which the model learns from to improve its performance. In the video, the creation of data sets is emphasized as a critical step in training embeddings and concept models, with the speaker providing guidance on how to prepare and use data sets effectively.


Interface refers to the user-friendly medium through which individuals interact with a system or software. In the context of the video, the interface is the application or tool used to train custom models, organize data sets, and configure training settings. The speaker discusses the simplicity and functionality of the interface, which is designed to make the training process accessible to users.

πŸ’‘Training Process

The Training Process involves the steps taken to teach a machine learning model to perform a specific task. This includes preparing data, adjusting model parameters, and evaluating the model's performance. In the video, the training process is detailed, with the speaker explaining the technical aspects and providing practical advice on how to effectively train custom models.


The session focuses on training custom models using open-source scripts available for free.

Two types of tools can be trained: embeddings and concept models, each with their unique training scripts.

Textual inversion is used for training embeddings, while Laura and Dora training are for concept models.

Tokenization breaks down prompts into smaller parts that can be analyzed mathematically by the system.

Model weights determine the relationship between the tokens and the visual content they relate to.

An analogy of light sources passing through a lens is used to explain the generation process.

Embeddings allow for efficient manipulation of the prompt layer by consolidating prompts into a new tool.

Concept models extend the base model to include new information and concepts.

Pivotal tuning is an advanced technique for training a new embedding that works with a specific concept.

Creating a dataset for embeddings involves using images and training a conditioning reference, while concept models require captioning images.

The training process is monitored through the UI, which is a simple application designed to help prepare datasets and train models.

The UI allows for the organization of images, captioning, and the creation of JSONL files for training.

Training configurations include basic settings, data configs, textual inversion configurations, optimizer configurations, and advanced settings.

Validation prompts are updated to match the placeholder token or trigger words used in the training.

Checkpoints, logs, and validation folders are created within the output directory to track the training progress.

Embeddings can be imported directly into Invoke, allowing users to utilize them in prompts for generating content.

The session concludes with an invitation for feedback to improve the training interface and scripts.