Most desktop automation tools break as soon as an app updates a button label or layout. You end up maintaining brittle selectors, parsing undocumented DOM trees, or building fragile workarounds. The Coasty computer use API flips that model. It treats your app like a human user, see the screen via screenshots, understand instructions, and click, type, and drag exactly as you would. You describe what you want, not how to click a specific element ID. The agent handles the messy reality of modern UIs, including pop-ups, dropdowns, and resizing windows.
How the computer use API works
The core idea is a two‑step loop. The client sends a screenshot with an instruction and a CUA version. The server returns structured actions like click, type, move, and scroll. You repeat until the status is done. For persistent context you can start a session and let the server store the full trajectory. For one‑shot tasks you can rely on the simpler predict endpoint. The API also provides grounding to map an element description to x,y coordinates and a free parse endpoint that converts existing pyautogui scripts into structured actions you can feed into the agent. All requests are authenticated with an API key set in the COASTY_API_KEY environment variable.
curl -X POST https://coasty.ai/v1/predict \
-H "X-API-Key: $COASTY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"screenshot": "$(base64 -w 0 screenshot.png)",
"instruction": "Click the Search button on the top right.",
"cua_version": "v3"
}'Real-world automation scenarios
- Log into a legacy application that exposes only a browser UI and no official API.
- Run periodic reports by clicking through a dashboard, exporting CSVs, and uploading them to a storage bucket.
- Configure services on a cloud console that only exposes a web interface.
- Perform repetitive tasks in IDEs such as opening projects, running tests, and posting results to a Jira ticket.
- Execute scripts in terminal-based tools that lack programmatic control.
Call /v1/predict with a screenshot and instruction, then loop until status is done.
Where this beats brittle automation
Traditional automation relies on brittle selectors like CSS classes, XPath, or element IDs. When a UI changes, your script breaks. The computer use agent works with visual cues and natural language instructions. It can handle dropdowns, modals, tooltips, and layout shifts without you updating selectors. Because it sees the actual screen, you can automate anything that a human can do, including dragging windows, scrolling pages, and interacting with legacy systems that have no APIs.
Next steps
- Get an API key at https://coasty.ai/developers and store it in your environment as COASTY_API_KEY.
- Start a task run with POST /v1/runs to let the server drive an agent on a provisioned cloud VM.
- Define workflows with POST /v1/workflows to model multi-step processes, retries, and conditional branches.
- Use POST /v1/parse to turn existing pyautogui scripts into structured actions compatible with the agent.
You can now drive real browsers, terminals, and desktop apps with the Coasty computer use API. Build reliable automation for anything that exposes a UI. Get your key at https://coasty.ai/developers and start automating.
Want to see this in action?
View Case Studies