Screenshot to Action: A Deep Dive Into the /v1/predict Endpoint
You want an agent that looks at your screen and clicks, types, and scrolls like a human does. The simplest way to start is with the /v1/predict endpoint. It takes a base64 screenshot and a natural-language instruction, then returns a list of actions like click, scroll, or type. You can call it in a loop, capture a new screenshot, predict again, and keep acting until the task is done. This endpoint is the core of vision-based automation.
How /v1/predict Works
The /v1/predict endpoint expects a POST request to https://coasty.ai/v1/predict. You provide a base64-encoded screenshot, an instruction string, and the cua_version string. The server returns an actions array and a status field. The status can be pending, done, or error. You keep calling predict on each new screenshot until status is done. The endpoint costs $0.05 per call.
curl -X POST https://coasty.ai/v1/predict \
-H "X-API-Key: $COASTY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"screenshot": "$(base64 -i screenshot.png | tr -d "\n")",
"instruction": "Click the blue button in the top right corner",
"cua_version": "v3"
}'Request and Response Fields
- ●screenshot (string, base64): the raw image of the desktop or browser.
- ●instruction (string): a natural-language description of what to do.
- ●cua_version (string): version of the computer use agent, defaults to v3.
- ●actions (array of objects): each action has a type and coordinates.
- ●status (string): one of pending, done, or error.
Loop capture → predict → act until status is done.
Why This Beats Brittle Automation
Traditional automation relies on selectors like IDs, classes, or XPath. These break when a website changes or when elements are dynamically rendered. A computer use agent sees the screen just like a human. It does not need stable selectors. It can handle layout shifts, new buttons, or unexpected layouts. This works for desktop apps, browsers, and terminals. You do not need to maintain selector maps or update them after every release.
Next Steps
- ●Create a workflow to chain multiple tasks together with /v1/workflows.
- ●Use /v1/runs to let a server-driven agent complete a whole task for you.
- ●Read the key from COASTY_API_KEY instead of hardcoding it.
- ●Check your wallet balance and webhook signatures in the docs.
Start with /v1/predict to turn screenshots into actions. Build a small agent that clicks, types, or scrolls. Once you are comfortable, explore workflows and server-driven task runs. Get your API key at https://coasty.ai/developers and start automating with a computer use agent.