Ground UI Elements to Coordinates with /v1/ground
Many automation tasks depend on finding the right element on screen. Traditional tools use CSS selectors or XPath, which break when UIs change or use dynamic classes. The /v1/ground endpoint solves this by taking a screenshot and a natural language description of what you want, then returning the exact x,y coordinates. This lets your agent act like a human, clicking and typing on real desktops instead of relying on fragile selectors.
How /v1/ground works
The server reads your screenshot and the element description you provide. It processes the image using vision models to locate the area that matches your description. It returns a JSON response with x and y pixel coordinates (0-based) and the width and height of the matched region. You can feed these coordinates directly into pyautogui or your own mouse emulation logic. The endpoint costs $0.03 per call.
curl -X POST https://coasty.ai/v1/ground \
-H 'X-API-Key: $COASTY_API_KEY' \
-H 'Content-Type: application/json' \
-d '{
"screenshot": "<base64-encoded PNG>",
"element_description": "the sign in button that says "Sign in""
}'import os
import base64
import requests
COASTY_API_KEY = os.getenv("COASTY_API_KEY")
endpoint = "https://coasty.ai/v1/ground"
with open("screenshot.png", "rb") as f:
screenshot_bytes = f.read()
screenshot_b64 = base64.b64encode(screenshot_bytes).decode("utf-8")
payload = {
"screenshot": screenshot_b64,
"element_description": "the sign in button that says \"Sign in\""
}
resp = requests.post(
endpoint,
headers={"X-API-Key": COASTY_API_KEY, "Content-Type": "application/json"},
json=payload
)
resp.raise_for_status()
result = resp.json()
print(result)
# Example output:
# {"x": 120, "y": 80, "width": 100, "height": 36}
Response fields
- ●x (int): The horizontal pixel coordinate of the matched element.
- ●y (int): The vertical pixel coordinate of the matched element.
- ●width (int): The width of the matched region in pixels.
- ●height (int): The height of the matched region in pixels.
Use /v1/ground to turn screenshots and element descriptions into reliable x,y coordinates.
Where this beats brittle automation
CSS selectors and XPath depend on the HTML structure. When a developer changes classes or reorders DOM nodes, your tests break. By grounding on what you see in an image, your agent follows the visual layout instead of the code. This works across browsers, native apps, and remote desktops. You get a robust computer use agent that reacts to the UI as a human would.
Start grounding UI elements with /v1/ground and build a computer use agent that clicks accurately on any screen. Get your API key at https://coasty.ai/developers.