Screenshot to Action: A Deep Dive Into the /v1/predict Endpoint
Traditional automation relies on XPath, CSS selectors, or brittle APIs. When UI changes, scripts break. The /v1/predict endpoint solves this by letting your code send a screenshot plus a natural language instruction to an AI model, then receive actionable coordinates, clicks, keys, or text. You get visual perception without writing selectors. You can drive desktop apps, browsers, and terminals through a simple HTTP loop: capture screen, predict, act, repeat. It costs $0.05 per prediction and returns actions plus a status that tells you when the task is done.
How /v1/predict works
The endpoint is a POST to https://coasty.ai/v1/predict with three required fields. You send a base64-encoded screenshot, an instruction string, and the cua_version. The model analyzes the visual context and the instruction, then returns an actions array plus a status. Each action includes a type (click, key, text, etc.), an element description, and coordinates. The status field is either "done" or an intermediate value like "pending". You loop: capture new screenshot, POST to /v1/predict, execute actions, and stop when status is "done". This is the core loop of any computer use agent built on Coasty.
curl -X POST https://coasty.ai/v1/predict \
-H 'X-API-Key: $COASTY_API_KEY' \
-H 'Content-Type: application/json' \
-d '{
"screenshot": "$(base64 -i screenshot.png)",
"instruction": "Click the Cancel button in the popup dialog.",
"cua_version": "v3"
}'Request and response fields
- ●screenshot: base64-encoded PNG image. No file size limits in docs, but keep it under a few megabytes for speed.
- ●instruction: plain-language string describing what to do. The model uses it alongside the visual context.
- ●cua_version: either 'v3' or 'v4'. 'v3' is stateless prediction. 'v4' is autonomous with a pass/fail verifier when used in runs (not this endpoint alone).
- ●actions: array of objects. Each object has type, element_description, x, y. Types include click, key, text.
- ●status: string. 'done' means the model finished reasoning and returned all actions. Other values indicate it is still processing or waiting.
Loop: capture screenshot, POST to /v1/predict, execute actions, repeat until status is "done". Each prediction costs $0.05.
Where this beats brittle automation
With selectors, you must keep up with every UI change: new IDs, class names, or layout shifts. A single change breaks your script. /v1/predict lets your agent see the screen and describe what it sees. It does not care about an element's ID. It only needs to understand the instruction and locate the object visually. This makes your automation resilient to UI updates. You also avoid deep nesting of XPath expressions and brittle type checks. You can interact with third-party apps, browser popups, or legacy UIs that have no stable automation APIs. The model integrates visual context with language, so your agent can reason about multi-step workflows and recover from unexpected states.
Next steps
- ●Wrap the loop in a function that handles screenshots, requests, and action execution.
- ●Use sessions and stateful memory if you need to remember context across multiple predictions.
- ●Explore workflows and runs for orchestrating multi-step tasks with asserts, retries, and human approval.
- ●Get your API key at https://coasty.ai/developers and start building your computer use agent.
The /v1/predict endpoint is the heart of computer use automation on Coasty. You can build agents that see and click like a human, without fragile selectors. Start by capturing a screenshot, sending it with an instruction, and processing the actions until status is "done". Once you are comfortable, move to sessions and runs for longer workflows. Get your key at https://coasty.ai/developers and start turning screenshots into actions today.