Classic UI automation expects stable selectors and stable element IDs. When a web app changes class names or layout, those selectors break and your script fails. The /v1/predict endpoint solves this by feeding the model a screenshot and an instruction, then returning real mouse and keyboard actions. You get a computer use agent that sees the screen and behaves like a human. This post covers the exact request format, the response fields, and a working example you can copy and run.
How /v1/predict Works
The endpoint accepts a base64-encoded screenshot, a text instruction, and the cua_version to indicate the model. It returns an actions array and a status field that tells you if the current prediction is done. The flow is simple capture, predict, act. You keep sending screenshots and instructions until the server reports status done. Each prediction call costs $0.05. This is the core primitive behind any computer use API implementation.
curl https://coasty.ai/v1/predict \ \
-X POST \ \
-H "X-API-Key: $COASTY_API_KEY" \ \
-H "Content-Type: application/json" \ \
-d '{
"screenshot": "$BASE64_SCREENSHOT",
"instruction": "Click the blue button labeled Submit",
"cua_version": "v3"
}'Request Fields
- screenshot: base64-encoded PNG/JPEG of the current screen.
- instruction: human-readable text describing what to do.
- cua_version: model identifier (e.g., v3, v4).
- The call returns an actions array and a status field.
Response Fields
- actions: array of mouse and keyboard actions generated by the model.
- status: either "done" or an intermediate state such as "pending".
Loop capture, predict, act until status is "done".
Where This Beats Brittle Automation
Traditional automation relies on CSS selectors or XPath which can break instantly when a team releases a new design. The /v1/predict endpoint lets your agent read the visual layout and infer where elements are. It works even when class names change, layouts shift, or content is dynamically injected. This makes it ideal for exploratory UI testing, browser workflows, and desktop automation where selectors are unreliable. You are building a computer use agent that sees and acts like a human, not a brittle selector matcher.
Use the /v1/predict endpoint as the foundation for a computer use agent that reads the screen and executes actions. Start with a simple task like clicking a button and then scale to complex workflows. Get your API key and start building at https://coasty.ai/developers.
Want to see this in action?
View Case Studies