How to Automate Any Desktop App With the Coasty Computer Use API
Writing selectors for every UI element in a flaky desktop app is painful. You often need to rewrite tests when a designer changes a button class or layout. The Coasty computer use API lets you drive real desktops, browsers, and terminals by interpreting screenshots and natural language, so you can automate any app without brittle selectors.
How it works
The Coasty computer use API runs a model that sees a screenshot and reads your instructions. It returns structured actions such as clicks, typing, and scrolling. You can either loop capture → predict → act with the /v1/predict endpoint or use a stateful trajectory with /v1/sessions. You can also ground an element description to coordinates with /v1/ground or turn pyautogui code into structured actions with the free /v1/parse endpoint. For end‑to‑end runs, use /v1/runs so the server drives an agent to completion and streams events via Server‑Sent Events.
curl -X POST https://coasty.ai/v1/predict \
-H "X-API-Key: $COASTY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"screenshot": "$(base64 -w0 screenshot.png)",
"instruction": "Click the Save button and type a filename.",
"cua_version": "v3"
}'Key concepts and pricing
- ●POST /v1/predict costs $0.05 per call and returns actions + status. Loop until status is "done".
- ●POST /v1/sessions costs $0.10 and creates a stateful trajectory for multi‑step workflows.
- ●POST /v1/sessions/{id}/predict costs $0.04 and acts on the session.
- ●POST /v1/ground costs $0.03 and maps a screenshot + description to x,y coordinates.
- ●POST /v1/runs bills $0.05 per agent step and supports max_steps, deadline_seconds, and on_awaiting_human.
- ●POST /v1/workflows lets you express runs as a versioned JSON DSL with steps like task, assert, if, loop, parallel, and human_approval.
- ●POST /v1/machines provisions a cloud VM you can start, stop, and snapshot.
- ●Billing happens from a prepaid USD wallet where 1 credit = $0.01.
- ●Webhooks are HMAC signed (header Coasty-Signature: t=unix,v1=hex).
You pay $0.05 per agent step for a computer use agent, and you can drive real desktops, browsers, and terminals with natural language.
Where this beats brittle automation
With selector‑based tools you must rewrite selectors whenever a designer changes a class, layout, or text. The Coasty computer use API sees the actual screen content, so it can click a button even if its class name changes. It can also handle layouts that shift between columns, which often breaks fixed‑coordinate code. By using a computer use agent you eliminate selector maintenance and get a single instruction to drive any UI.
Start building an automated workflow for your desktop app. Get a key at https://coasty.ai/developers and read the docs at https://coasty.ai/docs.