Screenshot to Action: A Deep Dive Into the /v1/predict Endpoint
Most automation tools rely on brittle selectors, hardcoded IDs, or APIs that do not exist. A computer use agent watches the screen, understands the UI, and clicks. The /v1/predict endpoint is the core of that loop. It takes a screenshot, an instruction, and a version, then returns actions the agent can perform. You pay only $0.05 per prediction. This post covers the request shape, response fields, and how the endpoint differs from stateless and stateful paths.
How /v1/predict works
The endpoint is a vision model that maps a screenshot and instruction into actions. You POST a base64 image, a text instruction, and a cua_version. The response includes actions and a status. The status is either "done" or another value. If the status is not "done", you capture a new screenshot, predict again, and repeat. The endpoint is stateless, so it does not remember prior frames. Use /v1/sessions and /v1/sessions/{id}/predict when you need trajectory memory and persistent state.
curl https://coasty.ai/v1/predict \
-H "X-API-Key: $COASTY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"image": "<base64-screenshot>",
"instruction": "Click the green Submit button.",
"cua_version": "v3"
}'- ●POST https://coasty.ai/v1/predict
- ●Headers: X-API-Key or Authorization: Bearer <key>
- ●Body: image (base64), instruction (string), cua_version (string v3 or v4)
- ●Cost: $0.05 per call
- ●Returns: actions (array) and status (string)
- ●Status "done" means the model decided the task is finished
- ●If status is not "done", loop with a new screenshot
Loop capture, predict, act until the status is "done".
Where this beats brittle automation
A computer use agent does not need to know the ID of a button or the class name of a dropdown. It sees the visual state of the screen and decides which element to interact with. This makes the agent robust to UI changes, dynamic elements, and layouts that shift. The /v1/ground endpoint further helps you map a screenshot and element description to precise x,y coordinates. The free /v1/parse endpoint turns PyAutoGUI code into structured actions. Together these tools let you build agents that work across browsers, desktop apps, and terminals without brittle selectors.
Use the /v1/predict endpoint to build a computer use agent that watches the screen and acts. Create a loop that captures screenshots, predicts actions, and executes them. When you need memory across frames, switch to /v1/sessions and /v1/sessions/{id}/predict. Get a key and start building at https://coasty.ai/developers .