Automating Form Filling and Checkout Flows Over the Computer Use API
Checkout flows are a nightmare for traditional bots. They require typing into dynamic fields, clicking buttons that change position, and navigating through multi-step wizards. The Coasty Computer Use API lets you drive real browsers and desktop applications by seeing the screen and acting like a human, all via a simple REST API. You send a screenshot and a instruction, and the agent returns the next mouse and keyboard actions. No XPath selectors, no guessing DOM order, no brittle find-and-click logic.
How it works
The core primitive is POST /v1/predict. You send a base64-encoded screenshot, a natural-language instruction, and the cua_version. The endpoint returns a status and a list of actions such as move_to, click, scroll, and type. You loop: capture the screen, call /v1/predict with the screenshot, execute the returned actions, and repeat until the status is done. Each predict call is billed $0.05. Vision gives you real-time feedback, so the agent can handle layout shifts, animations, and accessibility features.
curl -X POST https://coasty.ai/v1/predict \
-H "X-API-Key: $COASTY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"cua_version": "v3",
"screenshot": "$(base64 -i screenshot.png)",
"instruction": "Fill the email field with [email protected] and click the sign in button."
}'Task Runs for multi-step flows
For multi-step flows, use POST /v1/runs. You provide machine_id (a cloud VM you provision via POST /v1/machines), task, cua_version (v3 or v4), instructions (appended to the base prompt), system_prompt, max_steps, deadline_seconds, on_awaiting_human (pause, fail, or cancel), and optional webhook_url. The server runs an agent to completion, streaming events via GET /v1/runs/{id}/events. Each agent step is billed $0.05. States include queued, running, awaiting_human, succeeded, failed, cancelled, and timed_out.
Grounding clicks with coordinates
When you need precise click coordinates, POST /v1/ground maps a screenshot and element description to x and y coordinates. You send a base64 screenshot and a human-readable description like 'email input field'. The endpoint returns pixel coordinates that you can use with actions.move_to and actions.click. This is billed $0.03 per call and works well when you want to combine vision with explicit targeting.
Use /v1/predict with base64 screenshots for the simplest form filling and checkout flows, and /v1/runs when you need the server to drive a full session to completion.
Where this beats brittle automation
Traditional automation relies on stable selectors like XPath or CSS class names. When a site reorders fields, changes classes, or uses dynamic IDs, your bot breaks. The Computer Use API lets the agent see the actual UI and decide where to click and type based on the current layout. It handles hover states, tooltips, and accessibility text. It also works for desktop apps that only expose UI, not APIs. This makes checkout flows, registration wizards, and form submissions far more robust.
Start building reliable form filling and checkout bots with the Coasty Computer Use API today. Get your API key at https://coasty.ai/developers and try the /v1/predict endpoint with your first screenshot.