Ground UI Elements to Coordinates with /v1/ground: A Computer Use API Guide
Selectors break when UI changes, layout shifts, or frameworks inject new classes. To run a computer use agent that feels like a human, you need to see the screen and act on real coordinates. The /v1/ground endpoint maps a screenshot and an element description to exact x,y coordinates so your agent can click, type, or scroll with pixel precision.
How /v1/ground works
Send a base64 screenshot and an element description to POST /v1/ground. The server scans the image, finds the region that best matches your description, and returns bounding box coordinates. Use these coordinates for pyautogui or any GUI automation library. This endpoint costs $0.03 per request and does not consume credits from sessions or task runs.
#!/bin/bash
# Install jq: brew install jq or apt-get install jq
# Replace with your actual screenshot file path
SCREENSHOT=$(base64 -i screenshot.png)
# Element description to ground on
description="The primary CTA button that says 'Sign up'"
curl -s -X POST https://coasty.ai/v1/ground \
-H "X-API-Key: $COASTY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"screenshot": "'"$SCREENSHOT"'",
"description": "'"$description"'"
}' | jq '.'import base64
import os
import requests
import json
# Read screenshot as base64
with open("screenshot.png", "rb") as f:
screenshot_b64 = base64.b64encode(f.read()).decode("utf-8")
description = "The primary CTA button that says 'Sign up'"
def ground_element(screenshot_b64, description):
url = "https://coasty.ai/v1/ground"
headers = {
"X-API-Key": os.environ["COASTY_API_KEY"],
"Content-Type": "application/json"
}
payload = {"screenshot": screenshot_b64, "description": description}
resp = requests.post(url, headers=headers, json=payload)
resp.raise_for_status()
return resp.json()
result = ground_element(screenshot_b64, description)
print(json.dumps(result, indent=2))Response shape
- ●Returns a JSON object with a bounding box for the matched element.
- ●Bounding box includes x, y, width, and height fields.
- ●Coordinates are in screen-relative pixels.
- ●Use these coordinates directly with pyautogui or other GUI automation tools.
/v1/ground costs $0.03, reads a base64 screenshot and an element description, and returns x, y, width, and height coordinates.
Where this beats brittle automation
Traditional automation leans on CSS selectors, XPath, or ID attributes. Those break when UI text changes, classes get renamed, or frameworks inject dynamic attributes. With a computer use agent, you describe what you want in plain text. The model sees the screen, understands context, and returns the exact coordinate box for that element. You can then click, type, or scroll at that position, no matter how the DOM changes. The /v1/ground endpoint is the bridge between a human-like description and reliable, pixel-accurate actions.
Combine /v1/ground with the /v1/predict endpoint to build agents that see and act like humans. Use the coordinates you ground to drive pyautogui or any GUI library. Ready to build? Get your API key at https://coasty.ai/developers.