# Computer & browser use

URL: https://justuju.org/concept/computer-use/
Layer 8 · Agents

Agents that operate a screen like a person: look at a screenshot, then click, type and scroll.

## D1 · Story

Some apps have no special doorway for AI, so the AI uses them the way you do.
It looks at a picture of the screen, decides where to click or what to type, then looks again to see what changed.
1. Look: a picture of the screen
2. Decide: where to click or type
3. Do it: click or type
Like: Playing a video game by watching the screen and pressing buttons.

## D2 · Picture

A model that can read images is shown a screenshot. It replies with an action, such as "click at (412, 230)" or "type this text". The harness performs it in a virtual machine or browser and sends back a new screenshot.
Browser agents often read the page structure (the HTML or accessibility tree) as well as the pixels, which is cheaper and more precise.
It works with almost any app, including old ones with no API, but it is slower, costlier and more fragile than calling an API. Every page it reads is untrusted, so hidden text can try to take it over.
1. Screenshot: what is on screen
2. Look: model reads the image
3. Decide: click, type, scroll
4. Act: harness performs it
Like: Playing a video game by watching the screen. Where it breaks: A player watches the screen all the time. The model sees separate snapshots, so pop-ups, slow loading and things that move between screenshots trip it up.

## D5 · Frontier

Working a screen like a person works with any app, but it is slow, brittle and exposed to everything on the screen.
Far from people. On real desktop tasks, people succeeded 72% of the time and the best agent 12% when the benchmark was published (OSWorld, 2024). Scores have risen since, but the tasks are still narrower than real work.
Finding the button. On high-resolution professional software, the best grounding model located the right element only 18.9% of the time (ScreenSpot-Pro, 2025).
Injection from the page. Injected web content partly hijacked web agents in up to 86% of cases, though agents often failed to complete the attacker's goal (WASP, 2025). Logging in on a user's behalf raises the stakes.

## Examples as of 2026-10-30

- Anthropic computer use
- ChatGPT agent (formerly Operator)
- Gemini Agent (Google; replaced the Project Mariner experiment)
- Browser Use (open source)

## Sources

- Computer use tool (Claude API documentation): https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool
- OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments: https://arxiv.org/abs/2404.07972
- WebArena: A Realistic Web Environment for Building Autonomous Agents: https://arxiv.org/abs/2307.13854
- ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use: https://arxiv.org/abs/2504.07981
- WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks: https://arxiv.org/abs/2504.18575
- Google shuts down Project Mariner, folds it into Gemini Agent (Android Authority): https://androidauthority.com/google-project-mariner-shutdown-3664323/


---
Justuju · https://justuju.org/ · Content CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote with a link to the page.