Ground UI Elements to Pixels with the /v1/ground Endpoint
UI automation breaks when selectors change. Class names shift, IDs get renumbered, or layouts reorder. With the Coasty computer use API, you can describe a UI element in plain language and get precise pixel coordinates. The /v1/ground endpoint takes a base64 screenshot and an element description and returns the top-left x,y coordinates of the first matching element. This lets you build computer use agents that see the screen and act like a human without maintaining fragile selectors.
How the /v1/ground endpoint works
The /v1/ground endpoint maps a screenshot and a text description to x,y coordinates. You send a POST request with a base64 screenshot string, an instruction that describes the element, and a cua_version. The server returns the element's top-left x and y pixels. This is a one-shot call, not a loop. It costs $0.03 per request. The endpoint is ideal for quick lookups before a computer use agent performs an action.
#!/bin/bash
# Ground a UI element to coordinates using /v1/ground
# Set your API key from the environment
COASTY_API_KEY=${COASTY_API_KEY:-''}
if [ -z "$COASTY_API_KEY" ]; then
echo "Error: COASTY_API_KEY environment variable is required"
exit 1
fi
# Example screenshot (replace with your own base64 image)
SCREENSHOT_BASE64=$(base64 -i screenshot.png | tr -d '\n')
# Request body
PAYLOAD=$(cat <<EOF
{
"screenshot": "$SCREENSHOT_BASE64",
"instruction": "the blue submit button",
"cua_version": "v3"
}
EOF
)
# Call /v1/ground
RESPONSE=$(curl -s -X POST "https://coasty.ai/v1/ground" \
-H "Authorization: Bearer $COASTY_API_KEY" \
-H "Content-Type: application/json" \
-d "$PAYLOAD")
echo "$RESPONSE"
Request and response details
- ●Endpoint: POST https://coasty.ai/v1/ground
- ●Header: Authorization: Bearer $COASTY_API_KEY
- ●Body: { screenshot: base64 string, instruction: string, cua_version: string }
- ●Response: JSON with x and y pixel coordinates
- ●Cost: $0.03 per request
- ●Returns the top-left x,y of the first matching element
The /v1/ground endpoint returns the top-left x,y of the first matching element for $0.03 per call.
When to use /v1/ground in your workflow
- ●Use it before a click or type action to locate elements dynamically
- ●Combine with /v1/predict or /v1/sessions for full computer use loops
- ●Works best when you have a clear, localized element description
- ●Services as a fast lookup layer over your computer use agent
Where this beats brittle automation
Traditional automation relies on static selectors like XPath, CSS selectors, or class names. These break when a UI updates, changes layout, or renumbers IDs. With a computer use agent that can see the screen, you can ground elements by natural language. The /v1/ground endpoint lets you describe what you want to click and get precise coordinates without maintaining fragile selectors. This makes your automation more resilient and easier to update.
Ground UI elements to coordinates with /v1/ground and build computer use agents that act like humans. For full automation loops, use /v1/sessions or /v1/runs. Get your API key at https://coasty.ai/developers to start building.