Screenshot to Action: A Deep Dive Into the /v1/predict Endpoint
Most automation tools rely on brittle selectors, hardcoded IDs, and APIs that only expose thin hooks. You end up maintaining selectors when a UI changes. The /v1/predict endpoint flips the model: you send a screenshot and a human-like instruction, and the model returns actions, clicks, types, scrolls, to drive the desktop. You loop capture → predict → act until the server reports status done. No DOM parsing, no selectors to break.
How /v1/predict works
POST /v1/predict sends a base64 screenshot, a natural language instruction, and the cua_version. The endpoint returns a status and an array of actions. You keep sending fresh screenshots and instructions until the status is done. This is the core flow for a stateless vision-based agent. Pricing is $0.05 per prediction.
import base64
import os
import requests
API_KEY = os.environ.get('COASTY_API_KEY')
BASE_URL = 'https://coasty.ai/v1'
def predict_action(screenshot_bytes, instruction):
# Encode screenshot as base64
b64_screenshot = base64.b64encode(screenshot_bytes).decode('utf-8')
payload = {
'screenshot': b64_screenshot,
'instruction': instruction,
'cua_version': 'v3'
}
resp = requests.post(
f'{BASE_URL}/predict',
json=payload,
headers={'X-API-Key': API_KEY}
)
resp.raise_for_status()
return resp.json()
# Example
screenshot_bytes = open('screen.png', 'rb').read()
result = predict_action(screenshot_bytes, 'Click the Submit button')
print(result)Request and response fields
- ●screenshot: base64-encoded image bytes (PNG, JPG).
- ●instruction: natural language task description.
- ●cua_version: string. Use 'v3' for the current model, 'v4' for autonomous mode with a verifier.
- ●status: string indicating current state like pending, done, or error.
- ●actions: array of structured actions (click, type, scroll, etc.).
Loop capture → predict → act until status is done.
Why this beats brittle automation
With selectors you must keep up with every layout change. The model sees the actual screen content, so it can adapt to reordered elements, new labels, or changed IDs. It can also reason about context: "Click the first button that says Save" instead of relying on a static XPath. This makes your agents resilient to UI churn and lets you script complex workflows without writing brittle selectors.
Next steps
Once you master /v1/predict, explore stateful sessions with /v1/sessions and task runs with /v1/runs for autonomous workflows. Check the full docs and get an API key at https://coasty.ai/developers.