Ground UI Elements to Coordinates with the /v1/ground Computer Use API
Browser and desktop automation tools often rely on brittle selectors like XPath or CSS classes. Those break when a UI changes even slightly. The /v1/ground endpoint lets you send a screenshot and an element description then get back exact x,y coordinates you can pass to pyautogui. You pay $0.03 per request and get reliable, human-like control.
How /v1/ground works
The /v1/ground endpoint takes a base64 encoded screenshot and a text description of the UI element you want to click or interact with. It returns an object with an x and y coordinate. The request uses a standard POST with JSON body. The response contains the coordinates plus metadata. The endpoint is billed at $0.03 per call.
curl https://coasty.ai/v1/ground \
-H 'Content-Type: application/json' \
-H 'X-API-Key: $COASTY_API_KEY' \
-d '{
"screenshot": "<base64-encoded-image>",
"description": "The big blue login button on the top right"
}'Quick Python example
- ●Read COASTY_API_KEY from the environment for security.
- ●Use base64.b64encode on a raw bytes screenshot.
- ●POST to https://coasty.ai/v1/ground.
- ●Parse the JSON response to extract x and y.
- ●Each call costs $0.03 and returns a single coordinate.
import os
import base64
import requests
api_key = os.getenv('COASTY_API_KEY')
url = 'https://coasty.ai/v1/ground'
# Example: capture a screenshot as bytes
with open('screenshot.png', 'rb') as f:
image_bytes = f.read()
base64_image = base64.b64encode(image_bytes).decode('utf-8')
payload = {
'screenshot': base64_image,
'description': 'The big blue login button on the top right'
}
resp = requests.post(url, json=payload, headers={'X-API-Key': api_key})
resp.raise_for_status()
result = resp.json()
print('x:', result.get('x'))
print('y:', result.get('y'))
# Expected output:
# x: 842
# y: 156Remember: each /v1/ground request costs $0.03 and returns a single (x,y) coordinate from the provided screenshot.
Where this beats brittle automation
Modern web applications ship frequent updates. Incremental refactors often change class names, reorder elements, or restructure the DOM. Reliance on CSS classes or XPath quickly leads to flaky tests. The /v1/ground endpoint works on raw pixels. You describe what you see, "the big blue login button on the top right”, and you get coordinates that correspond to that visual element. This approach mirrors human mouse movement and is resilient to class-name churn. You can pair /v1/ground with pyautogui or any screen automation tool to drive real browsers, desktop apps, and terminals without brittle selectors.
Use /v1/ground to ground your UI actions on real pixels instead of fragile selectors. Build automation that survives UI changes and works across browsers and desktops. Get your API key at https://coasty.ai/developers to start grounding elements today.