Most automation assumes you know exactly where a button or input is on screen. In real apps, layouts shift, text changes, and fixed selectors break. The /v1/ground endpoint solves this by taking a screenshot plus a natural language description of an element and returning its precise X,Y coordinates for your agent to act on.
How /v1/ground works
To ground an element, send a base64 screenshot, an instruction that describes what you want to find, and the cua_version parameter. The endpoint returns a JSON body with x and y coordinates. You can use those values to construct a click or type action inside your loop of capture → predict → act.
curl -X POST https://coasty.ai/v1/ground \
-H "X-API-Key: $COASTY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"screenshot": "$(base64 -w0 screenshot.png)",
"instruction": "Find the login button",
"cua_version": "v3"
}'Request and response fields
- screenshot: base64 image of the current screen
- instruction: natural language description of the element you need (e.g., 'login button', 'email field')
- cua_version: API version to use (defaults to v3, but you can specify 'v4' if you want the autonomous verifier)
- Response body includes x and y coordinates as integers
x and y are the exact pixel coordinates you can pass directly to your agent's click or type actions.
Where this beats brittle automation
Traditional tools rely on fixed selectors like ID, class, or XPath. If an app reorders DOM elements, injects dynamic classes, or uses client-side rendering, those selectors break. By grounding on the visual state of the screen, your computer use agent adapts to layout changes, text updates, and even different browsers without rewriting selectors. This makes your agent more robust across products, demos, and internal tools.
Use /v1/ground to get precise coordinates from the visual state of a screen, then feed those into your computer use agent's click and type actions. Next, try building a workflow that grounds a login flow: capture, ground the username and password fields, then act. Get your API key at https://coasty.ai/developers.
Want to see this in action?
View Case Studies