Most desktop and browser automation relies on fragile selectors or hardcoded X,Y coordinates. When a UI changes layout or the browser tweaks margins, the script breaks. The Coasty computer use API solves this by grounding UI descriptions to real screen positions. The /v1/ground endpoint takes a screenshot and a natural-language element description and returns the precise X,Y coordinates you can feed directly into pyautogui or any other action library. You pay only $0.03 per grounding call.
How /v1/ground works
The /v1/ground endpoint is a stateless tool that bridges vision and action. You send a base64-encoded screenshot, an English description of the UI element, and the client’s cua_version. The server inspects the image and returns the bounding box of the element you described. The response includes the top-left X and Y coordinates, the width, and the height. This grounding step is free to call once per element per screenshot, costing $0.03. Use the coordinates to click, type, or drag with tools like pyautogui, or feed them into a trajectory-based agent for multi-step tasks.
#!/usr/bin/env python3
import base64
import os
import requests
COASTY_API_KEY = os.getenv('COASTY_API_KEY')
BASE_URL = 'https://coasty.ai/v1'
# Encode a local screenshot as base64
with open('screenshot.png', 'rb') as f:
img_b64 = base64.b64encode(f.read()).decode('utf-8')
payload = {
'image': img_b64,
'description': 'The primary CTA button at the top right',
'cua_version': 'v3'
}
resp = requests.post(
f'{BASE_URL}/ground',
json=payload,
headers={'X-API-Key': COASTY_API_KEY}
)
resp.raise_for_status()
print(resp.json())
Grounding vs brittle selectors
- No fragile selectors to break with UI updates
- Works on any visible browser or desktop element
- Returns exact pixel coordinates for direct pyautogui calls
- Reduces trial-and-error debugging in UI automation scripts
- Integrates cleanly into existing pyautogui workflows
Ground UI descriptions to coordinates once, reuse the coordinates across many steps.
Where computer use beats API-only tools
API-only tools often require you to maintain a separate mapping of element IDs to coordinates. When the layout changes, you must regenerate that mapping manually. With Coasty’s computer use API, the model sees the screen like a human does. You describe the UI element in plain English, and /v1/ground returns up-to-date coordinates for that exact moment. This keeps your automation resilient to layout shifts and lets you automate workflows at a higher level without manually maintaining selector databases.
Ground UI elements to coordinates with /v1/ground and build desktop automation that adapts to real UI changes. Combine grounding with trajectory-based sessions (POST /v1/sessions) or task runs (POST /v1/runs) to automate complex workflows. Get a key and start grounding at https://coasty.ai/developers.
Want to see this in action?
View Case Studies