Grounding UI Elements to Coordinates with /v1/ground
Most automation fails when elements move, change IDs, or use dynamic classes. You can write brittle selectors, but they break with every UI update. The computer use API solves this by grounding UI elements to exact x,y coordinates. The /v1/ground endpoint takes a screenshot and a description, then returns the precise click location. This lets agents act on any visual element, not just HTML selectors.
How it works
Send a base64 screenshot and a short description of the element you want to click. The endpoint returns an x, y pair that you can pass to pyautogui. Input fields are screenshot (base64 string), description (string), and cua_version (string, default 'v3'). The response includes x (integer) and y (integer). The endpoint costs $0.03 per call.
#!/usr/bin/env python3
import base64
import os
import requests
API_KEY = os.getenv('COASTY_API_KEY')
BASE_URL = 'https://coasty.ai/v1'
# Encode a sample screenshot as base64 (replace with real path)
screenshot_path = 'screenshot.png'
with open(screenshot_path, 'rb') as f:
img_bytes = f.read()
screenshot_b64 = base64.b64encode(img_bytes).decode('utf-8')
url = f'{BASE_URL}/ground'
headers = {
'X-API-Key': API_KEY,
'Content-Type': 'application/json'
}
payload = {
'screenshot': screenshot_b64,
'description': 'the blue submit button at the bottom of the form',
'cua_version': 'v3'
}
resp = requests.post(url, headers=headers, json=payload)
resp.raise_for_status()
result = resp.json()
print('x:', result.get('x'))
print('y:', result.get('y'))Key details to remember
- ●POST /v1/ground costs $0.03 per request.
- ●The request requires screenshot (base64 string), description (string), and cua_version (string, default 'v3').
- ●Response contains x (integer) and y (integer) relative to the top-left corner.
- ●Grounding is stateless, you send a fresh screenshot each time you need coordinates.
POST /v1/ground maps a description and screenshot to x,y coordinates, billed at $0.03 per call.
Where this beats brittle automation
Traditional tools rely on selectors like ID, class, or XPath. When a designer changes a class name, everything breaks. With computer use agents, you describe what you see and ground actions to coordinates. This makes your automation resilient to UI changes, accessibility labels, and dynamic DOMs. You still get precise clicks, but you describe the intent, not the implementation.
Grounding UI elements to coordinates with /v1/ground is the foundation for reliable computer use agents. Use it to build workflows that click, type, and interact with any visual element. Get a key at https://coasty.ai/developers and start grounding your UI actions.