Ground UI Elements to Coordinates with /v1/ground
Robust agents need to know where to click, not just what to say. Traditional automation tools rely on brittle selectors like XPath or CSS classes that break when a UI updates. The Coasty computer use API solves this by grounding UI elements to actual screen coordinates. The /v1/ground endpoint takes a screenshot and a natural language description of an element and returns x, y coordinates that your agent can use for precise clicks and drags.
How it works
Send a base64-encoded screenshot and a description of the element you want to act on. The API returns a JSON payload with the element's bounding box. You can then click at the center of that box using any library that understands screen coordinates. The endpoint costs $0.03 per request and is independent of session state, making it perfect for one-off lookups or pre-computation.
#!/usr/bin/env python3
import os
import base64
import requests
API_KEY = os.getenv("COASTY_API_KEY")
BASE_URL = "https://coasty.ai/v1"
# Read a local screenshot as base64
with open("screenshot.png", "rb") as f:
img_bytes = f.read()
img_b64 = base64.b64encode(img_bytes).decode("utf-8")
payload = {
"screenshot": img_b64,
"element": "Sign in button in the top right corner",
"cua_version": "v3"
}
resp = requests.post(
f"{BASE_URL}/ground",
json=payload,
headers={"X-API-Key": API_KEY}
)
resp.raise_for_status()
result = resp.json()
print(result)
# Click at the center of the bounding box
x = result["x"]
y = result["y"]
# Use pyautogui or your preferred library
distance = result.get("distance", 0)
print(f"Click at ({x}, {y})")Request and response fields
- ●screenshot: base64-encoded PNG image of the current screen.
- ●element: natural language description of the target UI element.
- ●cua_version: 'v3' or 'v4' to match your agent version.
- ●response: JSON with x and y coordinates (integer pixels).
- ●distance: optional pixel distance from the center of the bounding box (helpful for fuzzy targeting).
POST /v1/ground with screenshot and element description returns x, y coordinates for precise clicks.
Where this beats brittle automation
API-only tools scrape HTML or inspect classes, but those can change without warning. With /v1/ground, your agent sees the live UI and grounds actions on actual screen pixels. This means fewer false positives, clearer error messages, and a higher success rate when layouts shift. Combine this with the computer use API's vision-based prediction loop to build agents that explain their actions and recover from layout drift.
Grounding UI elements to coordinates is the missing piece for production-ready computer use agents. Use /v1/ground to turn natural language descriptions into actionable clicks. Get your API key and start building resilient automation at https://coasty.ai/developers.