JustujuAI, layer by layer.Open in the interactive map

Computer & browser use

Agents that operate a screen like a person: look at a screenshot, then click, type and scroll.

Layer 8 · Agents · Run time · reviewed 2026-10-30

Read first: The agent loop, Tool use & protocols (MCP).

D1 · StoryFor young children and total beginners

Follow the loop: how often does the model look at the screen?

Some apps have no special doorway for AI, so the AI uses them the way you do.

It looks at a picture of the screen, decides where to click or what to type, then looks again to see what changed.

Look, decide, do, look again

  1. Look: a picture of the screen
  2. Decide: where to click or type
  3. Do it: click or type

Like: Playing a video game by watching the screen and pressing buttons.

D2 · PictureFor curious beginners

A model that can read images is shown a screenshot. It replies with an action, such as "click at (412, 230)" or "type this text". The harness performs it in a virtual machine or browser and sends back a new screenshot.

Browser agents often read the page structure (the HTML or accessibility tree) as well as the pixels, which is cheaper and more precise.

It works with almost any app, including old ones with no API, but it is slower, costlier and more fragile than calling an API. Every page it reads is untrusted, so hidden text can try to take it over.

The see, decide, act loop

  1. Screenshot: what is on screen
  2. Look: model reads the image
  3. Decide: click, type, scroll
  4. Act: harness performs it

Like: Playing a video game by watching the screen.
Where it breaks: A player watches the screen all the time. The model sees separate snapshots, so pop-ups, slow loading and things that move between screenshots trip it up.

D5 · FrontierFor experts

Working a screen like a person works with any app, but it is slow, brittle and exposed to everything on the screen.

Far from people. On real desktop tasks, people succeeded 72% of the time and the best agent 12% when the benchmark was published (OSWorld, 2024). Scores have risen since, but the tasks are still narrower than real work.

Finding the button. On high-resolution professional software, the best grounding model located the right element only 18.9% of the time (ScreenSpot-Pro, 2025).

Injection from the page. Injected web content partly hijacked web agents in up to 86% of cases, though agents often failed to complete the attacker's goal (WASP, 2025). Logging in on a user's behalf raises the stakes.

Examples as of 2026-10-30

Sources

  1. Computer use tool (Claude API documentation) (2026)
  2. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments (2024)
  3. WebArena: A Realistic Web Environment for Building Autonomous Agents (2023)
  4. ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use (2025-04-04)
  5. WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks (2025-04-22)
  6. Google shuts down Project Mariner, folds it into Gemini Agent (Android Authority) (2026)

How it connects

More in Layer 8 · Agents