Ground UI Elements to Coordinates with /v1/ground
Most UI automation relies on brittle selectors like XPath, CSS, or IDs. Those break when a layout changes or when an app uses dynamic classes. The Coasty computer use API gives you something better. The /v1/ground endpoint takes a screenshot and a natural language description of an element and returns the exact x, y pixel coordinates you need to click or tap. This lets your agent interact with any UI like a human, not a fragile script.
How it works
The /v1/ground endpoint is a mapping layer. You send a base64-encoded screenshot and a free-text description of the element you want to interact with. The model infers the bounding box of that element and returns its center coordinates along with confidence. This works on web pages, desktop apps, or any graphical interface your agent can see. Each request costs $0.03, billed to your prepaid USD wallet where 1 credit equals $0.01.
curl https://coasty.ai/v1/ground \
-H 'X-API-Key: $COASTY_API_KEY' \
-H 'Content-Type: application/json' \
-d '{ \
"image": "$(base64 -i screenshot.png)", \
"description": "the orange submit button at the bottom of the form" \
}'import os
import base64
import requests
url = "https://coasty.ai/v1/ground"
api_key = os.getenv("COASTY_API_KEY")
with open("screenshot.png", "rb") as f:
image_b64 = base64.b64encode(f.read()).decode()
resp = requests.post(
url,
headers={
"X-API-Key": api_key,
"Content-Type": "application/json",
},
json={
"image": image_b64,
"description": "the orange submit button at the bottom of the form",
},
)
resp.raise_for_status()
result = resp.json()
print(result['x'], result['y']) # e.g. 450 620Grounding is stateless and cheap
- ●Each /v1/ground request is $0.03 and does not require a session.
- ●No state is kept between requests, so you can run it as many times as needed.
- ●The response includes x, y coordinates and a confidence score.
- ●It works with any image your agent captures, not just web screenshots.
- ●Combine it with /v1/predict or /v1/predict session calls for full-automation loops.
The core idea: send a screenshot and a natural language description of the UI element, get back exact x, y coordinates, then click using those coordinates.
Where this beats brittle automation
Selector-based tools break when a framework changes classes, injects dynamic prefixes, or reorders DOM elements. XPath and CSS selectors can also fail on shadow DOM, canvas-based UIs, or custom widgets. Grounding removes those dependencies. Instead of asking for an ID that might not exist, you ask for what you see: 'the search input on the left side' or 'the menu icon in the top right.' The computer use agent understands the screenshot and returns a reliable coordinate, even if the underlying HTML changes. This lets you build automation that adapts to UI changes without rewriting selectors.
Use /v1/ground to turn screenshots and natural language into click coordinates. It is a stateless, low-cost way to ground your UI automation in what your agent can see. Build a web scraper that survives layout updates, a desktop automation tool that clicks real buttons, or a human-like UI tester. Get your API key and start grounding at https://coasty.ai/developers.