Screenshot to Action: A Deep Dive Into the /v1/predict Endpoint
Most UI automation relies on brittle selectors or APIs that do not exist. You need a way to see the screen, understand what is there, and act like a human. The /v1/predict endpoint does exactly that. It accepts a base64 screenshot, a natural language instruction, and a CUA version, then returns a list of actions such as click, type, scroll, and wait. You run this in a loop: capture the screen, predict the next action, run the action, and repeat until the status is done. The endpoint costs $0.05 per call, and the loop terminates automatically when the server signals completion.
How it works
The /v1/predict endpoint handles vision-based computer use. You POST a JSON payload with a base64-encoded screenshot image, an instruction string describing what to do, and the CUA version you want to use (v3 by default). The server processes the visual context, generates a set of actions, and returns them along with a status field. The status can be pending, done, or error. When status is done, the server has finished reasoning about the current screenshot and you can stop the loop. You do not need a session ID for this endpoint; it works statelessly per screenshot.
curl -X POST https://coasty.ai/v1/predict \
-H "X-API-Key: $COASTY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"image": "$(cat screenshot.b64)",
"instruction": "Click the submit button.",
"cua_version": "v3"
}'Request and response fields
- ●image: a base64-encoded string of the screenshot. Do not include data URI prefixes.
- ●instruction: natural language describing the task for this screenshot. More specific instructions reduce action noise.
- ●cua_version: the version of the computer use agent. v3 is the default, v4 supports autonomous pass/fail verification.
- ●Response actions: an array of action objects, each containing type (click, type, scroll, wait), coordinates (x, y), and text (for type actions).
- ●Response status: a string indicating pending, done, or error. Stop the loop when status is done.
- ●Price: $0.05 per prediction call.
Loop capture, predict, act until status is done.
Where this beats brittle automation
With selectors and hard-coded IDs you must maintain a mapping that breaks whenever a UI changes. The /v1/predict endpoint gives you visual understanding. It sees the submit button even if its class name changes. It recognizes text, images, and layout without knowing CSS classes. You can automate desktop apps, web dashboards, and legacy tools that expose only a GUI. This approach scales across environments because the agent works on the rendered surface, not on unstable identifiers.
Start building a computer use agent that watches screens and acts like a human. Use /v1/predict to turn screenshots into actions, pay $0.05 per prediction, and loop until the server reports done. Get your API key at https://coasty.ai/developers.