Screenshot to Action: A Deep Dive Into the /v1/predict Endpoint
Most UI automation tools rely on brittle selectors, XPath rules, or fixed IDs that break when a product changes. The /v1/predict endpoint flips that model. You send a base64 screenshot and a natural language instruction, and the model returns click coordinates, text matches, and keyboard actions. This approach is the core of the computer use API, and it lets you build agents that drive real desktops, browsers, and terminals like a human would.
How it works
The endpoint is a POST request to https://coasty.ai/v1/predict. It requires a base64-encoded screenshot, an instruction, and a cua_version string. The request returns a JSON object with a status field, either pending or done, and an actions array that contains click events with x and y coordinates, text matches, and keyboard instructions. To drive a session, you capture a new screenshot, call /v1/predict, then act on the coordinates returned. Repeat until the status is done. Each prediction call is billed at $0.05.
import os
import base64
import requests
import json
API_URL = "https://coasty.ai/v1/predict"
API_KEY = os.getenv("COASTY_API_KEY")
def predict_action(screenshot_b64, instruction, cua_version="v3"):
payload = {
"screenshot": screenshot_b64,
"instruction": instruction,
"cua_version": cua_version
}
headers = {
"X-API-Key": API_KEY,
"Content-Type": "application/json"
}
resp = requests.post(API_URL, json=payload, headers=headers)
resp.raise_for_status()
return resp.json()
# Example: take a screenshot, encode, and call predict
# In practice, use a screen capture library like pyautogui
screenshot_bytes = b"""...
""" # replace with actual screenshot bytes
screenshot_b64 = base64.b64encode(screenshot_bytes).decode("utf-8")
result = predict_action(screenshot_b64, "Click the blue login button.", "v3")
print(json.dumps(result, indent=2))Key fields and pricing
- ●screenshot: base64-encoded PNG or JPEG image of the screen.
- ●instruction: natural language describing what to do (e.g., 'Enter email, then click submit').
- ●cua_version: either 'v3' (default) or 'v4' for autonomous mode with pass/fail verifier.
- ●status: 'pending' or 'done' indicating whether the model is still thinking.
- ●actions: array of click events, text matches, and keyboard instructions.
- ●Each /v1/predict call costs $0.05.
Loop capture, predict, and act until the status is 'done'.
Where this beats brittle automation
With selectors, a single UI change breaks your test. The computer use API sees the actual screen, understands context, and adapts. It works on web apps, desktop apps, and terminals without hard-coded IDs or XPaths. This is why our in-house model reaches 85.6% on OSWorld benchmarks and an independently verified 82.81% on the official leaderboard. You get robust automation that evolves with the application.
Start building agents that see and act like humans. Get your API key at https://coasty.ai/developers and try the /v1/predict endpoint today.