How to Automate Any Desktop App with the Coasty Computer Use API
Desktop automation often means hunting for stable CSS selectors or maintaining fragile XPath expressions. When an app updates a layout, those selectors break. The Coasty computer use API flips that model on its head. Instead of guessing where elements are, you let a computer use agent see the screen and act. It captures a screenshot, interprets the instruction, clicks or types, and repeats until the task is done. You get robust automation that adapts to UI changes, no matter how small.
How it works
The Coasty computer use API uses two main patterns. The first is a stateless loop of capture, predict, and act. You send a base64 screenshot, an instruction, and the CUA version. The model returns actions. When the status is done, you stop. The second pattern uses sessions to keep a trajectory in memory. You create a session, then repeatedly call predict on that ID. This lets the agent build context across multiple steps. For precise click targets, you can map a screenshot plus element description to coordinates with the ground endpoint. For converting existing pyautogui scripts into structured actions, the parse endpoint is free.
curl -X POST https://coasty.ai/v1/predict \
-H "X-API-Key: $COASTY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"cua_version": "v3",
"screenshot": "$SCREENSHOT",
"instruction": "Click the Login button"
}'Automating a full task with runs
- ●POST /v1/runs provisions a task run on a cloud machine. It requires machine_id, task, and optionally cua_version (v3 or v4). v4 adds an independent pass/fail verifier.
- ●You can supply instructions to append to the base prompt, a system_prompt, max_steps, deadline_seconds, and an on_awaiting_human action (pause, fail, or cancel).
- ●Each agent step is billed $0.05. Runs follow states: queued, running, awaiting_human, succeeded, failed, cancelled, or timed_out.
- ●You can monitor runs via GET /v1/runs, GET /v1/runs/{id}, or stream events with GET /v1/runs/{id}/events. The events endpoint uses Server-Sent Events; reconnect with the Last-Event-ID header.
- ●If you need to pause or cancel mid-run, use POST /v1/runs/{id}/cancel or POST /v1/runs/{id}/resume.
Each agent step is billed $0.05, and you can cancel or pause any running task with a single request.
Where this beats brittle automation
Traditional RPA tools depend on selectors that assume a fixed DOM structure. When an app introduces a new class or ID, your scripts break. The Coasty computer use agent works by seeing what is actually displayed. It reads the layout from the screenshot, maps the instruction to the current visual state, and clicks or types accordingly. This makes automation resilient to minor UI churn. You also avoid writing boilerplate boilerplate for every interaction type. The agent generalizes across buttons, inputs, menus, and even non-web apps that lack an official API.
Using workflows for multi-step orchestration
- ●Workflows are a versioned JSON DSL you use to orchestrate runs. POST /v1/workflows creates a workflow definition.
- ●You can trigger a workflow run with POST /v1/workflows/{id}/runs or POST /v1/workflows/runs for ad-hoc inline workflows.
- ●Step types include task, assert, if, loop, parallel, human_approval, retry, succeed, and fail. You can guard steps with budget_cents, max_iterations, or deadline_seconds.
- ●Variables follow the form {{inputs.x}} or stepId.field, making it easy to pass data between steps.
- ●Each task step inside a workflow is billed $0.05, the same as a direct run.
Provisioning machines for real desktop interaction
- ●The machines endpoint lets you provision cloud VMs that the agent can drive. POST /v1/machines creates a machine.
- ●Use the machine_id in your runs so the agent has a real desktop environment, browser, or terminal to work with.
- ●You can start, stop, and snapshot machines through additional machine endpoints, giving you control over compute resources and persistence.
Now you have a clear path to automate any desktop app with the Coasty computer use API. Start with a simple capture, predict, and act loop, then move to runs with state, workflows for orchestration, and machine provisioning for full desktop control. The API is billed per agent step at $0.05, and you can cancel or pause tasks anytime. Get your key at https://coasty.ai/developers and build automation that adapts to the UI instead of fighting it.