How to Automate Any Desktop App with the Coasty Computer Use API
Most desktop automation tools rely on brittle selectors, hardcoded IDs, or an app's official API. When the UI changes, your script breaks. The Coasty computer use API solves this by running a vision-guided computer use agent that sees the screen and clicks, types, and drags just like a human. You can drive real desktops, browsers, and terminals without maintaining selectors.
How the computer use API works
The computer use API drives an agent through a series of steps. At each step, the agent receives a screenshot and an instruction. It predicts an action and the server executes it on a connected machine. You can run the agent in two modes. The stateless loop uses POST /v1/predict at $0.05 per call. The stateful mode uses POST /v1/sessions and POST /v1/sessions/{id}/predict at $0.04 per call. Both return actions and status. The server keeps running until status is done.
#!/bin/bash
# Run a stateless computer use agent on a connected machine
# Reads COASTY_API_KEY from the environment
COASTY_API_KEY=${COASTY_API_KEY}
if [ -z "$COASTY_API_KEY" ]; then
echo "Error: COASTY_API_KEY environment variable is required."
exit 1
fi
# Capture the screen as a base64 image (example)
SCREENSHOT=$(base64 -w 0 screenshot.png)
# Call the stateless prediction endpoint
RESPONSE=$(curl -s -X POST https://coasty.ai/v1/predict \
-H "X-API-Key: $COASTY_API_KEY" \
-H "Content-Type: application/json" \
-d "{\
\"screenshot\": \"$SCREENSHOT\",\
\"instruction\": \"Open a browser and navigate to https://example.com\",\
\"cua_version\": \"v3\"\
}")
# Parse actions and status from the response
STATUS=$(echo "$RESPONSE" | jq -r '.status')
ACTION=$(echo "$RESPONSE" | jq -r '.action')
echo "Status: $STATUS"
echo "Action: $ACTION"
# Loop until the server signals done
while [ "$STATUS" != "done" ]; do
SCREENSHOT=$(base64 -w 0 screenshot.png)
RESPONSE=$(curl -s -X POST https://coasty.ai/v1/predict \
-H "X-API-Key: $COASTY_API_KEY" \
-H "Content-Type: application/json" \
-d "{\
\"screenshot\": \"$SCREENSHOT\",\
\"instruction\": \"Continue your task\",\
\"cua_version\": \"v3\"\
}")
STATUS=$(echo "$RESPONSE" | jq -r '.status')
ACTION=$(echo "$RESPONSE" | jq -r '.action')
echo "Status: $STATUS"
echo "Action: $ACTION"
sleep 1
done
echo "Agent completed. Actions: $ACTION"Key fields and pricing
- ●Base URL: https://coasty.ai/v1
- ●Auth header: X-API-Key or Authorization: Bearer
- ●POST /v1/predict: $0.05 per call, takes screenshot (base64), instruction, cua_version
- ●POST /v1/sessions then POST /v1/sessions/{id}/predict: $0.04 per call, enables trajectory memory
- ●POST /v1/ground: $0.03 for mapping screenshot + element description to x,y coordinates
- ●POST /v1/parse: free, converts pyautogui code into structured actions
Run a computer use agent from $0.04 per step with POST /v1/sessions/{id}/predict.
Where this beats brittle automation
Traditional automation tools need stable selectors, which break when UI changes. The computer use API lets the agent see the screen and reason about where to click, type, or drag. This works for any desktop app, including those without official APIs. You can also use POST /v1/parse to translate existing pyautogui scripts into structured actions that the computer use API can execute.
Next steps
- ●Get a key from https://coasty.ai/developers and set COASTY_API_KEY
- ●Provision a machine with POST /v1/machines to drive a real desktop
- ●Write a task description and let the server run to completion with POST /v1/runs
- ●Build workflows with POST /v1/workflows for multi-step automation with variables, conditions, and retries
With the Coasty computer use API, you can automate any desktop app without brittle selectors. Build vision-guided agents that see and act like a human. Get a key at https://coasty.ai/developers and start automating.