Most desktop automation fails when the UI changes. Selectors break, IDs shift, and CSS classes get renamed. The /v1/ground endpoint solves this. You send a screenshot and a description of what you want to click, and Coasty returns the precise (x, y) coordinates. This grounding step is the bridge between high-level intent and low-level clicking. Use it inside your own loop to drive real desktops and browsers reliably.
How /v1/ground works
The endpoint takes a base64 screenshot, an element description, and a cua_version. It returns an action list with an action type of 'ground' and a coordinates object containing x and y. The request costs $0.03. Each returned action is usable directly by your agent or by other Coasty endpoints like /v1/predict. You loop: capture screen, ground the target, then execute the click.
# Example curl using the real /v1/ground endpoint
# Assume COASTY_API_KEY is set in your environment
API_KEY=$(cat COASTY_API_KEY)
# Capture a screenshot and convert to base64
SCREENSHOT_BASE64=$(base64 -i screenshot.png)
curl -s https://coasty.ai/v1/ground \
-H "X-API-Key: $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"screenshot": "'"$SCREENSHOT_BASE64"'",
"description": "the primary submit button in the top right",
"cua_version": "v4"
}' | jq .What you get back
- A JSON response with an actions array.
- Each action has type 'ground' and a coordinates object.
- Coordinates are pixel positions relative to the screenshot.
- You can validate the ground result before clicking.
Send the screenshot and element description to /v1/ground, then click the returned x and y coordinates.
Where this beats brittle automation
Traditional automation relies on CSS selectors, XPath, or element IDs. When the UI changes, these break. The /v1/ground endpoint uses vision to find elements by what they look like. You describe 'the blue download button on the left' instead of maintaining a broken selector map. This makes your agent resilient to layout shifts, design refreshes, and localized string changes. It also works across browsers, operating systems, and native apps where selectors often fail.
Ground UI elements to coordinates with /v1/ground and build robust computer use agents that adapt to changing interfaces. Start building today at https://coasty.ai/developers .
Want to see this in action?
View Case Studies