When AI is operating as an assistant, like an editor, we need ways to let it interact with our content directly. Inpainting gives users the ability the let AI adjust parts of a piece of content without regenerating or impacting the whole.
This pattern applies across all types of content, and can be used for interacting with existing content or remixing AI-generated content during a session.
For example,
- When writing an essay, I may ask AI to adjust the tone of a paragraph that I feel I am not getting quite right to see how it would reflect back my overall tone to me.
- When AI generates text, I can direct it to make changes in situ to the conversation itself.
- When generating creative assets like visuals or audio, I can isolate a part of the interface and have the AI apply new tokens only to that area.
![](https://cdn.prod.website-files.com/65db6dd21591364dfcb8ae36/668412350e81f47d9678631d_Frame%2090.png)
Best practices
The most critical aspect of inpainting is that it keeps the user in the driver's seat. The AI is never directly manipulating the source - instead, it is offering suggestions for how to improve it based on the user's instruction. The most seamless implementations of this pattern maintain that sense that the user is in control, drawing from other patterns that make it easy to know how to improve what exists, and then generate options, without a ton of thought.
For example, Adobe Firefly and Midjourney offer four versions of the regeneration for the user to choose from. Notion allows the user to manipulate the in-painted regeneration before accepting it into the document.
TPerhaps there is a future approach that operates more like an editor: the user could give instructions or ask for feedback, the AI might review it and suggest multiple changes, and the user can accept, reject, or regenerate those comments.
The more AI can truly operate as a partner, the more users will rely on it to complete tasks.