Most automation tools rely on brittle selectors, xpath strings, or an API that only knows about a specific field. That breaks as soon as UI changes. The /v1/predict endpoint solves that by letting you send a screenshot and an instruction. The model sees the screen, picks the right element, and returns exact x, y coordinates and a Python-style action. You can then replay those actions with pyautogui. This is a computer use agent that truly watches and clicks.
How it works
The request payload contains three core fields. The model returns actions and a status. The agent runs in a loop: capture a screenshot, POST to /v1/predict, send the returned actions to the OS, then repeat until status is "done".
#!/usr/bin/env bash
set -euo pipefail
COASTY_API_KEY="${COASTY_API_KEY:?COASTY_API_KEY must be set}"
BASE_URL="https://coasty.ai/v1"
# Encode a sample screenshot as base64 (replace with an actual image path)
SCREENSHOT=$(base64 -i screenshot.png)
# Send a predict request
RESPONSE=$(curl -s -X POST "$BASE_URL/predict" \
-H "X-API-Key: $COASTY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"screenshot": "'$SCREENSHOT'",
"instruction": "click the first button that says confirm",
"cua_version": "v3"
}')
echo "Response: $RESPONSE"
# Parse actions and status
STATUS=$(echo "$RESPONSE" | jq -r '.status')
ACTIONS=$(echo "$RESPONSE" | jq -r '.actions')
echo "Status: $STATUS"
echo "Actions: $ACTIONS"
# If done, you could then use pyautogui to replay actions.Request and response fields
- screenshot: base64 encoded image (PNG, JPEG, or similar).
- instruction: natural language description of the task, e.g., 'click the confirm button'.
- cua_version: string, default 'v3'. The version controls behavior.
- status: string, one of: running, done, error.
- actions: array of actions, each with type, x, y, and any required parameters.
Loop capture, predict, act until status is "done", then replay actions with pyautogui.
Where this beats brittle automation
With selectors you must track class names, ids, and DOM paths. A single change in the UI can break your script. The computer use API reads the screen directly. It does not need a stable class name or ID. It only needs a clear instruction and a visible element. This makes your agents more robust across web apps, desktop tools, and custom internal UIs. You can also combine this endpoint with /v1/sessions for stateful trajectory memory, or /v1/runs for full multi-step agent orchestration.
Pricing and billing
- POST /v1/predict costs $0.05 per request.
- POST /v1/sessions costs $0.10 per session creation.
- POST /v1/sessions/{id}/predict costs $0.04 per prediction inside a session.
- POST /v1/runs bills $0.05 per agent step.
- POST /v1/ground costs $0.03 per request to map a screenshot and element description to x, y coordinates.
You now know how to send screenshots to the computer use API and turn them into real mouse and keyboard actions. Build agents that see and click like a human, then wire them into workflows, sessions, or full task runs. Get your key at https://coasty.ai/developers and start building.
Want to see this in action?
View Case Studies