loading
loading
Letting the model drive a real GUI by taking screenshots and issuing mouse and keyboard actions.
The model receives screenshots and returns actions (click at x,y, type text, scroll); your harness executes them against an actual desktop or browser and sends the next screenshot back. It can be self-hosted (you run the environment) or Anthropic-hosted, and it's a client-side loop — Anthropic processes the images and action requests but doesn't run your environment. It's the right tool when there's no API and you genuinely need to operate a visual interface; modern models return coordinates that map 1:1 to image pixels, so no scale-factor math. Sending screenshots at ~1080p balances accuracy against token cost.
Plainly
Think of Computer Use as a tool from the builder toolbox. Letting the model drive a real GUI by taking screenshots and issuing mouse and keyboard actions.
In practice
Use it when you are installing, editing, testing, reviewing, or shipping code. In practice, define the owner, input, output, and failure mode before you rely on it.