Ground UI Elements to Coordinates with /v1/ground
Every computer use agent needs to know where to click, type, or scroll. Hardcoded selectors break when layouts shift, and API calls often miss hidden controls. The /v1/ground endpoint solves this by turning a screenshot and an element description into precise x,y coordinates. You describe what you see, not how the DOM is built, and the model returns a location you can feed to a click or send function.
How it works
Send a base64 screenshot and a natural language description of the UI element to https://coasty.ai/v1/ground. Include the cua_version to match the agent's behavior. The endpoint returns a status and a location object with x and y integer coordinates. You loop over captures, ground each one, and act on the result until the task finishes. This endpoint costs $0.03 per request.
#!/bin/bash
# Read the API key from the environment, never hardcode it.
COASTY_API_KEY="${COASTY_API_KEY}"
# Example screenshot (replace with your own base64 image)
SCREENSHOT_BASE64="iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8/5+hHgAHggJ/PchI7wAAAABJRU5ErkJggg=="
# Describe the button you want to click
DESCRIPTION="Submit button in the top right corner of the form"
# Send the grounding request
curl -s -X POST "https://coasty.ai/v1/ground" \
-H "X-API-Key: $COASTY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"screenshot": "'"$SCREENSHOT_BASE64"'",
"description": "'"$DESCRIPTION"'",
"cua_version": "v3"
}' | jq .Request and response fields
- ●POST https://coasty.ai/v1/ground
- ●Header: X-API-Key: <your key> (read from COASTY_API_KEY env var)
- ●Body: { screenshot: base64 string, description: string, cua_version: string }
- ●Response: { status: string, location: { x: integer, y: integer } }
- ●Cost: $0.03 per grounding request
You describe what you see, the model returns x,y coordinates, you click.
Where this beats brittle automation
Traditional automation relies on CSS selectors, XPath, or ID attributes. When a designer changes a class name or moves a control, selectors fail and the test breaks. With /v1/ground, you describe the element in natural language. A change in the DOM does not matter as long as the visual appearance stays the same. This lets you build agents that see the screen and act like humans, adapting to layouts without selector updates.
Combine /v1/ground with /v1/predict or /v1/runs to build agents that inspect the screen, ground UI elements, and execute actions. Start building with a free key at https://coasty.ai/developers.