Ground UI Elements to Coordinates with /v1/ground
Most automation tools rely on fragile selectors: class names, IDs, or XPath that break when a page updates. The Coasty computer use API solves this with /v1/ground, which takes a screenshot and a natural-language description of an element and returns exact x,y coordinates you can use with pyautogui or any other action library. This lets you automate any UI, even one you cannot control via APIs.
How /v1/ground works
The endpoint maps a screenshot plus a concrete element description to pixel-space coordinates. It is a free request that returns an x and y integer pair, plus a confidence score. You can then feed those coordinates into pyautogui click or a similar action library to reliably interact with the element.
#!/usr/bin/env bash
COASTY_API_KEY="${COASTY_API_KEY}"
# Base64-encoded screenshot (replace with a real base64 string)
SCREENSHOT="$(base64 -i screenshot.png)
# Element description: type a button text or role
ELEMENT_DESC="The blue submit button in the top right corner"
# Call the grounding endpoint
response=$(curl -s -X POST https://coasty.ai/v1/ground \
-H "X-API-Key: ${COASTY_API_KEY}" \
-H "Content-Type: application/json" \
-d '{
"screenshot":"'"$SCREENSHOT"'",
"description":"'"$ELEMENT_DESC"'"
}')
# Parse x, y and confidence
x=$(echo "$response" | jq -r '.x')
y=$(echo "$response" | jq -r '.y')
confidence=$(echo "$response" | jq -r '.confidence')
echo "Element coordinates: $x,$y (confidence: $confidence)"
# Use the coordinates with pyautogui (example)
python3 -c "import pyautogui; pyautogui.click($x, $y)"Request and response fields
- ●screenshot: base64-encoded image (PNG, JPG, or other formats supported by the vision model)
- ●description: natural-language element description (e.g., 'the submit button with text Login')
- ●x: integer pixel coordinate on the x-axis
- ●y: integer pixel coordinate on the y-axis
- ●confidence: float between 0 and 1 representing how confident the model is in the mapping
Ground element descriptions to coordinates with a single free call to /v1/ground, then act with pyautogui or your preferred action library.
Where this beats brittle automation
For APIs that let you POST or GET data, you can automate endpoints directly. But many real-world workflows require clicking, filling forms, navigating menus, or interacting with legacy UIs that expose no APIs. When selectors break or change, those tools fail. With grounded coordinates, you describe what you see, not how you find it. The model handles layout changes, dynamic classes, or reordering, and returns a stable pixel target that your automation can click. This makes computer use agents resilient to UI evolution and less sensitive to the exact markup of a page.
Combine /v1/ground with the other computer use API endpoints to build agents that handle any UI, from modern web apps to desktop tools. Start building reliable, UI-agnostic automation with a free trial key at https://coasty.ai/developers.