Deepmind introduces Pointer Engineering, a Gemini-powered approach that uses the mouse cursor to capture visual and semantic context, reducing reliance on precise prompts for everyday AI interactions.

Deepmind researchers are integrating Pointer Engineering into Gemini in Chrome, where users can select parts of a webpage and ask questions about them. The feature will launch on the upcoming Googlebook as 'Magic Pointer.'

Instead of dragging content into an AI tool via detailed prompts, users point at on-screen elements and issue shorthand commands like 'Fix this' or 'Move that here,' combined with speech and gestures. Pixels are converted into 'structured entities' such as places, dates, or objects—for example, a handwritten note becomes an interactive to-do list.

The approach streamlines short, chat-like everyday interactions but does not replace prompt engineering; more complex tasks still require precise descriptions, according to Deepmind.