Inpainting

When AI is operating as an assistant, like an editor, we need ways to let it interact with our content directly. Inpainting gives users the ability the let AI adjust parts of a piece of content without regenerating or impacting the whole.

This pattern applies across all types of content, and can be used for interacting with existing content or remixing AI-generated content during a session.

For example,

• When writing an essay, I may ask AI to adjust the tone of a paragraph that I feel I am not getting quite right to see how it would reflect back my overall tone to me.
• When AI generates text, I can direct it to make changes in situ to the conversation itself.
• When generating creative assets like visuals or audio, I can isolate a part of the interface and have the AI apply new tokens only to that area.

Best practices

The most critical aspect of inpainting is that it keeps the user in the driver's seat. The AI is never directly manipulating the source - instead, it is offering suggestions for how to improve it based on the user's instruction. The most seamless implementations of this pattern maintain that sense that the user is in control, drawing from other patterns that make it easy to know how to improve what exists, and then generate options, without a ton of thought.

For example, Adobe Firefly and Midjourney offer four versions of the regeneration for the user to choose from. Notion allows the user to manipulate the in-painted regeneration before accepting it into the document.

TPerhaps there is a future approach that operates more like an editor: the user could give instructions or ask for feedback, the AI might review it and suggest multiple changes, and the user can accept, reject, or regenerate those comments.

The more AI can truly operate as a partner, the more users will rely on it to complete tasks.

Inpainting is a micro-version of prompting in general. The user needs to identify that the capability exists in the first place, then apply it, direct or update their prompt, and then review the result.

• The interface clearly shows the capability exists: Adobe features a dominant call to action to "Edit" a photo when the user hovers on it in the main canvas. Generative fill (their branded term for inpainting) is listed at the very top. Notion has a similar CTA to "Ask AI" when the user selects text, and options to edit the selected text are also listed at the top.
• The app makes it easy for the users to make changes: Notion's default suggested actions are listed in the dropdown, or the user can type directly into the input box to give instructions. Adobe Firefly, like other token-based generators, has to account for both the ability to select the image area and the instructions for how to adjust the output. Their web-based interface uses a simple UI to let the user select the area to adjust using a familiar brush tool, and to suggest constraints by adding or removing tokens.
• The app makes it easy to review results: Both of these examples use the variants and regenerate patterns to give the users multiple options before they commit. Another example of an app that does this well is Github, which uses inline prompting to let the user add or adjust code within an existing file. Similarly, the AI can offer variants for the user to select from.