Trying to automate a modern desktop app with pure UI selectors is a constant arms race. The layout changes, the IDs are dynamic, and you spend more time maintaining scripts than building features. The Coasty computer use API solves this by letting your agent see the screen, reason about what it sees, and act with a mouse and keyboard like a human user. You describe what you want in natural language, and the agent drives the machine through real UI elements.
How it works
The Coasty computer use API runs an agent that repeatedly captures the screen, reasons about what it sees, and emits actions that manipulate the desktop. The core flow uses a stateless predict endpoint that takes a screenshot, an instruction, and the version of the computer use agent to use. It returns a list of actions such as pointer moves, clicks, and keystrokes plus a status. You keep sending captures and predictions until the agent reports a done status. For higher fidelity and memory of past interactions, you can start a session and use the session predict endpoint, which keeps a trajectory of steps in memory. If you need to locate an element by description, you can ground a screenshot and element description to exact x,y coordinates. You can also turn Python code into structured actions with the free parse endpoint.
#!/bin/bash
# Stateless computer use example , loop capture, predict, act until done
COASTY_API_KEY=${COASTY_API_KEY}
BASE_URL="https://coasty.ai/v1"
# 1. Take a screenshot (replace with your screenshot capture command)
SCREENSHOT=$(base64 -i screenshot.png)
# 2. Initial prediction request
RESPONSE=$(curl -s -X POST "${BASE_URL}/predict" \
-H "X-API-Key: ${COASTY_API_KEY}" \
-H "Content-Type: application/json" \
-d '{
"screenshot": "'"${SCREENSHOT}"'",
"instruction": "Install the latest version of Python on the desktop",
"cua_version": "v3"
}')
# 3. Extract actions and status from the response
ACTIONS=$(echo $RESPONSE | jq -r '.actions[] | "pointer_move x='"$((.x))"' y='"$((.y))"' click" | head -1')
STATUS=$(echo $RESPONSE | jq -r '.status')
# 4. Execute the first action (example: move pointer and click)
echo "Executing action: $ACTIONS"
# In a real implementation, you'd map each action to pyautogui or your UI driver
# 5. Loop until done
while [[ "$STATUS" != "done" ]]; do
# Capture the updated screen
SCREENSHOT=$(base64 -i screenshot.png)
# Send another prediction request
RESPONSE=$(curl -s -X POST "${BASE_URL}/predict" \
-H "X-API-Key: ${COASTY_API_KEY}" \
-H "Content-Type: application/json" \
-d '{
"screenshot": "'"${SCREENSHOT}"'",
"instruction": "Install the latest version of Python on the desktop",
"cua_version": "v3"
}')
# Extract status and actions
STATUS=$(echo $RESPONSE | jq -r '.status')
echo "Current status: $STATUS"
# In a real implementation, you'd parse and execute all actions here
sleep 1
done
echo "Task completed with status: $STATUS"Key endpoints and pricing
- POST /v1/predict , $0.05 per prediction. Takes screenshot, instruction, and cua_version. Returns actions and status. Use for simple, repeatable tasks.
- POST /v1/sessions , $0.10 for a session. Then POST /v1/sessions/{id}/predict , $0.04 per prediction. Sessions store a trajectory of steps for longer workflows.
- POST /v1/ground , $0.03 per request. Maps a screenshot and element description to x,y coordinates for precise element interaction.
- POST /v1/parse , Free. Converts Python code into structured actions. Great for turning existing scripts into agent-compatible plans.
- Billing: 1 credit = $0.01. You use credits per prediction and per step of a task run.
For full task automation where the server drives the agent to completion, use POST /v1/runs, which bills $0.05 per agent step and supports states, awaits, webhooks, and deadlines.
Where this beats brittle automation
Traditional UI automation relies on stable selectors like IDs, classes, or XPath patterns. When those change, your tests break. The Coasty computer use agent works directly on visual data. It sees buttons, dropdowns, and dynamic layouts just like a human. You describe goals in natural language and let the agent decide the sequence of clicks and keystrokes. This approach adapts to UI changes automatically and works across different applications without rewriting selectors. You also get trajectory memory for sessions, so the agent can remember past actions and context across multiple steps.
Next steps
- Sign up at https://coasty.ai/developers to get an API key.
- Start with stateless predictions on simple tasks to test the flow.
- Move to sessions for multi-step workflows with memory.
- Use runs for fully autonomous tasks with server-side state tracking and webhooks.
- Explore workflows for versioned, reusable automation DSLs that include tasks, asserts, loops, and conditions.
You now have a practical path to automate any desktop app with the Coasty computer use API. Describe your goal, let the agent see and act like a human, and iterate until the task is done. Get your key at https://coasty.ai/developers and start building reliable desktop automation today.
Want to see this in action?
View Case Studies