Many web automation tools rely on brittle selectors or APIs you do not control. They break when a site changes a class name or hides an element. The Coasty computer use API solves this with a /v1/predict endpoint that takes a screenshot and an instruction, then returns the precise actions the agent should take. You run this in a loop, capturing the next screenshot, predicting actions, and executing them until the status is done. This is how you build a computer use agent that sees the screen and acts like a human.
How /v1/predict Works
The /v1/predict endpoint is a single HTTP POST to https://coasty.ai/v1/predict. It requires these fields in the JSON body. The endpoint costs $0.05 per prediction. The response contains an actions array, a status field, and a request_id you can use for logging or debugging.
#!/bin/bash
# Example: screenshot to action with /v1/predict
# Save this as predict.sh and run with COASTY_API_KEY set
API_URL="https://coasty.ai/v1/predict"
API_KEY="${COASTY_API_KEY}"
# Base64-encoded screenshot (replace with your own)
SCREENSHOT_BASE64="iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg=="
# Instruction telling the agent to click the first visible link
INSTRUCTION="Click the first visible link on the page."
# CUA version (use 'v3' for basic mode, 'v4' for autonomous with a pass/fail verifier)
CUA_VERSION="v3"
# Call /v1/predict
RESPONSE=$(curl -s -X POST "$API_URL" \
-H "X-API-Key: $API_KEY" \
-H "Content-Type: application/json" \
-d "{
\"screenshot\": \"$SCREENSHOT_BASE64\",
\"instruction\": \"$INSTRUCTION\",
\"cua_version\": \"$CUA_VERSION\"
}")
# Output the full response
echo "$RESPONSE"Request and Response Fields
- screenshot: a base64-encoded image of the current screen. It should match the format and resolution the model expects.
- instruction: a natural language description of what you want the agent to do. This is appended to the system prompt.
- cua_version: either 'v3' (basic) or 'v4' (autonomous with a pass/fail verifier). Defaults to 'v3' if not specified.
- actions: an array of action objects returned by the model. Each action includes type ('click', 'type', 'scroll', etc.), x and y coordinates, and optional text or modifiers.
- status: the current state of the model. It can be 'pending', 'in_progress', or 'done'. When status is 'done', the task is complete.
- request_id: a unique identifier for this prediction. Use it to track requests, handle retries, or log events.
Loop capture, predict, and act until status is 'done'.
Where This Beats Brittle Automation
With traditional selectors, you must know the exact class name, ID, or XPath ahead of time. If a developer changes the UI, your script breaks. The /v1/predict endpoint lets your computer use agent reason about the visual layout. It sees the screen, understands the context from your instruction, and produces the exact coordinates and actions to perform. This makes your automation resilient to UI changes, dynamic content, or sites that hide elements behind complex JavaScript interactions.
Next Steps
- Start a session with POST /v1/sessions to enable stateful trajectory memory for longer workflows.
- Use POST /v1/ground to map a screenshot and element description to x,y coordinates for fine-grained clicks.
- Run a complete task with POST /v1/runs to let the server drive an agent to completion with a pass/fail verifier.
- Try the free POST /v1/parse endpoint to turn generated pyautogui-style code into structured actions.
- Read the full documentation and get your API key at https://coasty.ai/developers.
The /v1/predict endpoint is the foundation of the Coasty computer use API. Use screenshots and natural language to drive automation that sees and acts like a human. Build your first computer use agent and start automating complex workflows today. Get your key at https://coasty.ai/developers.
Want to see this in action?
View Case Studies