Discovered this technique very recently so I wanted to share with you all how Prompt Editing works in Stable Diffusion Automatic1111. It is quite a powerful way to provide prompts to the Stable Diffusion model.
Prompt Editing allows you to start your prompt and while the model is diffusing the image, swap certain words with another, or introduce new words to the prompt while the image is being diffused ie. generated.
It is a great way to control when and what influences part of your prompt. Let us consider the very simple prompt and influence it using this technique. We will lock the seed so that the only variation is the prompt.
Prompt: A beautiful painting of a singular lighthouse, shining its light across a tumultuous sea, Trending on artstation
Seed: 1523531057
Steps: 60
Now let’s introduce an artist like Vincent Van Gogh at Step 21 of the diffusion which means for the last 40 steps this artist style should influence the image.
Prompt: A beautiful painting of a singular lighthouse [by Vincent Van Gogh:21], shining its light across a tumultuous sea, Trending on artstation
The syntax is defined using square brackets [] and the format is [keywords:<starting step>]. This means that this keyword is applied for the remainder of diffusion steps.
In contrast if we simply run the prompt as A beautiful painting of a singular lighthouse by Vincent Van Gogh, shining its light across a tumultuous sea, Trending on artstation which means that entire diffusion process considers Van Gogh’s influence we get something like this.
You can see how heavily the image is influenced by Van Gogh’s style, in comparison to above. Note: The seed is locked.
The Syntax
The way to use this kind of prompt editing is to use the syntax in 3 different ways:
- [keyword a:keyword b:<step>] – this allows you to define that keyword a is used from start until <step> but then at this <step> keyword b takes over.
- [keyword:<step>] – this keyword is introduced at the defined <step> and continues for remainder of the diffusion process
- [keyword::<step>] – note the double : in this case, which implies that keyword is applied until <step> is reached and then from <step> onwards this keyword is ignored.
Instead of steps you can also define percentage of the total steps by using 0.5 for 50% and 0.25 for 25% etc. The complete details are explained in the wiki but I explore more below with examples.
In Practice
Let’s continue and introduce another artist to this prompt: A beautiful painting of a singular lighthouse [by Georges Seurat:21], shining its light across a tumultuous sea, Trending on artstation
We get something quite different in terms of the result.
Now if introduce both artists where we start with Seurat and then bring in Van Gogh half way at Step 30, let’s see what we get.
Prompt: A beautiful painting of a singular lighthouse [by Georges Seurat:by Vincent Van Gogh:30], shining its light across a tumultuous sea, Trending on artstation
We can see that image is influenced by both artists, you have the brush strokes like Seurat which then starts to become more circular as Van Gogh takes over, but then one might argue that what if you use both names in the original prompt instead of this “prompt editing” technique. Well let’s see what we are getting if we keep both names and join them with an AND. Prompt will be: A beautiful painting of a singular lighthouse by Georges Seurat and by Vincent Van Gogh, shining its light across a tumultuous sea, Trending on artstation
Conclusion
I continue my experimentation with this kind of Prompt editing because it is like starting a canvas in one way and then changing the result by introducing new elements to your scene as you are painting. Its quite intriguing and there is no possible end to the combinations and variations you can introduce to the final image.
If you'd like to support our site please consider buying us a Ko-fi, grab a product or subscribe. Need a faster GPU, get access to fastest GPUs for less than $1 per hour with RunPod.io