Most UI automation tools rely on brittle selectors like XPaths, IDs, or CSS classes. When a product changes its markup, those selectors break. The Coasty computer use API solves this by giving your agent vision. You send a screenshot and a natural language instruction, and the model returns actions like click, type, scroll, and keypress. The /v1/predict endpoint is the core of this workflow. It charges $0.05 per call and returns a structured list of actions you can stream or decode into pyautogui code.
How /v1/predict works
The endpoint accepts a base64-encoded screenshot, an instruction, and the cua_version you want to run. It returns a JSON object with actions and a status. You repeat this loop: capture the screen (pyautogui.screenshot), encode it to base64, send it to /v1/predict, and then execute the actions. When the status is done, the task is complete. If it returns pending, keep looping until done. The model uses the screenshot as context to decide which element to click, what text to type, or where to scroll. This is the same loop used by the higher-level task runs, but the /v1/predict endpoint gives you fine-grained control and lower latency.
curl -X POST https://coasty.ai/v1/predict \
-H "X-API-Key: $COASTY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"screenshot": "<base64 screenshot>",
"instruction": "Click the button that says Submit",
"cua_version": "v3"
}'Request and response fields
- screenshot: base64-encoded PNG of the current screen. Required.
- instruction: natural language description of what to do on the screen. Required.
- cua_version: choose "v3" or "v4". v4 adds autonomous pass/fail verification.
- actions: JSON array of step objects like { type: "click", element: "button with text Submit", x: 420, y: 350 }.
- status: current state of the prediction. Possible values are pending or done.
- price: $0.05 per /v1/predict call. Credits are deducted immediately.
- You do not need to create a session for /v1/predict. It is stateless and billed per call.
Loop capture → encode → predict → act until status is done.
Where this beats brittle automation
With traditional tools, you must maintain selectors for every button, input, and menu. A single class name change breaks your script. The computer use agent sees the screen like a human. It understands context, handles layout shifts, and adapts to UI changes without selector updates. Because the endpoint returns actions you can execute directly, you can integrate it into existing workflows, wrap it in a router, or use it as the foundation for more complex agents. You also get free parse operations via /v1/parse, which turns pyautogui code into structured actions for logging, replay, or storage.
You now know how the /v1/predict endpoint turns screenshots into actionable steps. Build a simple screenshot-to-click agent, then extend it to handle typing, scrolling, and multi-step workflows. For more endpoints, variables, and pricing details, get your API key at https://coasty.ai/developers.
Want to see this in action?
View Case Studies