Computer & browser use
Agents that operate a screen like a person: look at a screenshot, then click, type and scroll.
Read first: The agent loop, Tool use & protocols (MCP).
D1 · StoryFor young children and total beginners
Some apps have no special doorway for AI, so the AI uses them the way you do.
It looks at a picture of the screen, decides where to click or what to type, then looks again to see what changed.
Look, decide, do, look again
- Look: a picture of the screen
- Decide: where to click or type
- Do it: click or type
Like: Playing a video game by watching the screen and pressing buttons.
D2 · PictureFor curious beginners
A model that can read images is shown a screenshot. It replies with an action, such as "click at (412, 230)" or "type this text". The harness performs it in a virtual machine or browser and sends back a new screenshot.
Browser agents often read the page structure (the HTML or accessibility tree) as well as the pixels, which is cheaper and more precise.
It works with almost any app, including old ones with no API, but it is slower, costlier and more fragile than calling an API. Every page it reads is untrusted, so hidden text can try to take it over.
The see, decide, act loop
- Screenshot: what is on screen
- Look: model reads the image
- Decide: click, type, scroll
- Act: harness performs it
Like: Playing a video game by watching the screen.
Where it breaks: A player watches the screen all the time. The model sees separate snapshots, so pop-ups, slow loading and things that move between screenshots trip it up.
D5 · FrontierFor experts
Open problems: benchmarks on real desktops and websites, such as OSWorld and WebArena, found agents far below human success rates when they were published. Logging in safely, prompt injection from web content, and knowing when an API is the better route are all still open.