Ground UI Elements to Coordinates with the Computer Use API
You want your computer use agent to click a button, type into a field, or fill a form. You can use brittle CSS selectors, XPath, or arbitrary pixel offsets. Or you can be precise. The /v1/ground endpoint turns a screenshot and a natural description of an element into x,y pixel coordinates. This makes your agent aware of the UI it sees, not just a list of rules.
How it works
The /v1/ground endpoint takes a screenshot and a text description of the UI element. It returns the center x and y coordinates for that element. This is a pure vision task. The call costs $0.03 per use. You run it before any action on that element. The coordinates are relative to the full screenshot. You can feed them into pyautogui or any other click/typewriter library.
#!/usr/bin/env bash
# Read your API key from the environment
API_KEY="${COASTY_API_KEY}"
# Your screenshot file (base64 encoded)
SCREENSHOT_BASE64=$(base64 -i screenshot.png)
# Element description
decription="Save button in the top right corner"
# Call /v1/ground
response=$(curl -s -X POST "https://coasty.ai/v1/ground" \
-H "Content-Type: application/json" \
-H "X-API-Key: ${API_KEY}" \
-d "{
\"screenshot\": \"${SCREENSHOT_BASE64}\",
\"description\": \"${description}\"
}")
# Output the full response for debugging
echo "Response: ${response}"
# Parse the JSON (jq example)
if command -v jq &> /dev/null; then
echo "\nParsed result:"
echo "x: $(echo "${response}" | jq -r '.x // empty')"
echo "y: $(echo "${response}" | jq -r '.y // empty')"
fiRequest and response fields
- ●POST https://coasty.ai/v1/ground
- ●Headers: X-API-Key: <key> (or Authorization: Bearer <key>)
- ●Body: screenshot (base64 string of the full screenshot), description (text describing the UI element)
- ●Response: JSON with x (number, pixel coordinate), y (number, pixel coordinate)
- ●Cost: $0.03 per call
- ●If the element is not found, the endpoint returns an error object with code and message
Call /v1/ground once per element you intend to interact with, then feed the returned x and y coordinates into pyautogui.mouseClick or similar.
Where this beats brittle automation
Selector-based automation breaks when a class name changes, a layout shifts, or a framework injects dynamic IDs. Pixel offsets drift with window resizing. The /v1/ground endpoint works with the visual snapshot. It understands what a button looks like and where it appears. You describe the element in natural language. The model grounds that description to the exact pixel. This makes your computer use agent robust to UI changes and layout variations.
Use /v1/ground to ground your computer use agent in the actual UI. Build reliable click flows, form fillers, and workflows that adapt to the screen as it appears. Get a key at https://coasty.ai/developers and start grounding elements today.