Ground UI Elements to Coordinates with /v1/ground
You are building an automated agent that clicks buttons, types into inputs, and scrolls through a real desktop or browser. The elements you need to interact with do not have stable selectors. They change layout, names, or classes. You need a way to say 'click the search bar' and get an exact x,y coordinate you can use with pyautogui or another robot library. The /v1/ground endpoint solves this by mapping a screenshot and an element description to a precise location on the screen.
How it works
The /v1/ground endpoint takes a base64 screenshot and a natural language description of the element you want to click, type into, or inspect. It returns the x,y coordinates of that element on the screen. This is a stateless, low-level API. You call it, get coordinates, and then use those coordinates to drive pyautogui or your own automation logic. The endpoint costs $0.03 per call.
#!/bin/bash
# Ground an element to coordinates using the /v1/ground endpoint
COASTY_API_KEY=$(cat "$HOME/.coasty/key" 2>/dev/null || echo "$COASTY_API_KEY")
if [ -z "$COASTY_API_KEY" ]; then
echo "Error: COASTY_API_KEY environment variable not set or .coasty/key not found."
exit 1
fi
# Sample screenshot (replace with your own base64 image)
SCREENSHOT='''
iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg==
'''
curl -sS https://coasty.ai/v1/ground \
-H "X-API-Key: $COASTY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"screenshot": "'"$SCREENSHOT"'",
"description": "search input field"
}' | jq .import os
import base64
import json
import requests
# Read API key from environment
COASTY_API_KEY = os.getenv("COASTY_API_KEY")
if not COASTY_API_KEY:
raise RuntimeError("COASTY_API_KEY environment variable is required.")
# Sample base64 screenshot (replace with your own)
SCREENSHOT_B64 = "'''
iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg==
'''
url = "https://coasty.ai/v1/ground"
headers = {
"X-API-Key": COASTY_API_KEY,
"Content-Type": "application/json",
}
body = {
"screenshot": SCREENSHOT_B64.strip(),
"description": "search input field",
}
resp = requests.post(url, headers=headers, json=body)
resp.raise_for_status()
result = resp.json()
print(json.dumps(result, indent=2))
# The response contains x and y coordinates for the described element
if "coordinates" in result:
x, y = result["coordinates"]
print(f"Found element at ({x}, {y})")Request and response fields
- ●POST /v1/ground endpoint
- ●Request body: screenshot (base64), description (string)
- ●Response: coordinates as {"x": <int>, "y": <int>}
- ●Pricing: $0.03 per call
- ●Stateless: no session or token to manage between calls
One call to /v1/ground gives you precise x,y coordinates for any UI element you can describe.
Where this beats brittle automation
Traditional automation relies on brittle selectors like XPath, CSS selectors, or class names. UI changes often break these selectors, forcing you to update your test scripts or scripts. With /v1/ground, you describe what you see in plain language. The model looks at the screenshot and returns a coordinate. If the UI layout shifts, you just provide a fresh screenshot and updated description. The coordinate remains accurate even when selectors change. This approach is ideal for agents that interact with real desktops, browsers, and terminals, not just structured APIs.
You now have a reliable way to ground UI elements to coordinates with the /v1/ground endpoint. Use these coordinates in pyautogui or your own robot to click buttons, type into fields, and automate complex workflows. Build a computer use agent that sees the screen and acts like a human. Get your API key at https://coasty.ai/developers to start grounding elements and automating real desktops.