Grounding UI Elements to Coordinates with /v1/ground
Classic automation tools rely on brittle selectors: XPath, CSS, or hardcoded IDs. If the layout shifts, the selector breaks and the script fails. The Coasty /v1/ground endpoint solves this by taking a screenshot and a natural language description of a UI element, then returning the exact x, y coordinates for that element on screen. You can use these coordinates with tools like pyautogui to perform precise clicks, inputs, and drags. This enables robust computer use agents that adapt to real UI changes.
How /v1/ground works
The endpoint receives a base64-encoded screenshot, an instruction describing the UI element to locate, and the cua_version parameter. It returns a JSON payload with a status and the element's bounding box. The bounding box contains x, y, width, and height coordinates. You can pass any cua_version supported by your Coasty account, such as 'v3' or 'v4'. The service costs $0.03 per request.
#!/usr/bin/env bash
# Example: ground a button to coordinates using /v1/ground
export COASTY_API_KEY=$(cat ~/.coasty_key)
# Step 1: capture a screenshot (adjust path as needed)
png_file="screenshot.png"
scrot "$png_file"
# Step 2: encode the screenshot as base64
base64_file="screenshot.b64"
base64 -i "$png_file" > "$base64_file"
# Step 3: call /v1/ground with an element description
response=$(curl -s -X POST https://coasty.ai/v1/ground \
-H "X-API-Key: $COASTY_API_KEY" \
-H "Content-Type: application/json" \
-d "{\"screenshot\": $(cat $base64_file), \"instruction\": \"Click the login button\", \"cua_version\": \"v3\"}")
echo "$response"Key response fields
- ●status: The server's response status, typically 'success' or 'failed'.
- ●x: The horizontal coordinate of the element's top-left corner.
- ●y: The vertical coordinate of the element's top-left corner.
- ●width: The horizontal span of the element.
- ●height: The vertical span of the element.
- ●cost: The $0.03 charge for this request.
Call /v1/ground with a screenshot and element description to get an x, y bounding box, then click with pyautogui.
Where this beats brittle automation
If a site reorders buttons or changes class names, XPath or CSS selectors break. With /v1/ground, the agent sees the current layout and grounds the instruction to coordinates. Even if the UI changes, the agent can re-run grounding and get updated x, y values. This makes computer use agents resilient to layout shifts. Combine grounding with /v1/predict for end-to-end task runs: the agent sees, grounds, and acts, all while staying aligned with the real screen.
Grounding UI elements with /v1/ground gives you precise, layout-aware actions. Start building agents that adapt to real UIs. Get your API key at https://coasty.ai/developers.