Most computer use agent tutorials show you how to send a screenshot and an instruction, but they stop short of telling you how to target a specific control. Clicking by ID or class breaks when UIs change or when elements are rendered via JavaScript. The /v1/ground endpoint solves this by taking a base64 screenshot and an element description and returning the exact x and y coordinates for that element on the screen.
How /v1/ground works
The endpoint accepts a base64 screenshot, an element description, and optional cua_version. It returns x and y coordinates that you can pass directly to pyautogui click. This is not a full task run. You capture the screen, ground the target, then use the coordinates in your automation loop.
- Endpoint: POST /v1/ground
- Price: $0.03 per call
- Request body: screenshot (base64), description (string), cua_version (string)
- Response: x (number), y (number), status (string)
import os
import base64
import requests
API_KEY = os.getenv("COASTY_API_KEY")
BASE_URL = "https://coasty.ai/v1"
def ground_element(base64_screenshot: str, description: str, cua_version: str = "v3") -> dict:
"""Send a screenshot and description to the grounding endpoint."""
url = f"{BASE_URL}/ground"
headers = {"X-API-Key": API_KEY}
payload = {
"screenshot": base64_screenshot,
"description": description,
"cua_version": cua_version,
}
resp = requests.post(url, headers=headers, json=payload)
resp.raise_for_status()
return resp.json()
# Example usage
# 1. Capture your screen with pyautogui or another screenshot tool
# screenshot_bytes = pyautogui.screenshot()
# screenshot_base64 = base64.b64encode(screenshot_bytes).decode()
# 2. Call grounding
# coords = ground_element(screenshot_base64, "The blue submit button")
# print(coords) # {'x': 123, 'y': 456, 'status': 'done'}
# 3. Click using the coordinates
# pyautogui.click(coords['x'], coords['y'])
# If you need to retry with a different description, just call ground_element again.
# The status field lets you confirm success. If status is 'done', you have coordinates.
# If it is something else, inspect the error or try again.
# You can also pass a different cua_version if your agent is using the 'v4' autonomous mode.
# The grounding endpoint does not depend on the task run version, but it respects the same versioning.
# You can chain this with /v1/sessions to maintain a stateful trajectory, but the grounding call itself is stateless.
# For more information, see the Coasty Computer Use API documentation at https://coasty.ai/docs.Grounding in a full task run
You can combine /v1/ground with a task run to create a complete workflow. The agent runs POST /v1/runs with machine_id, task, and cua_version. Inside the run, you can ground elements and use the coordinates to perform actions. The run steps are billed at $0.05 per agent step, while grounding is billed at $0.03 per call. This gives you the best of both worlds: a robust agent that can drive real desktops and fine-grained control over element targeting.
Error handling and retries
If grounding fails, the response includes an error object with code, message, and request_id. Common issues are a blurred screenshot, an inaccurate description, or a cua_version mismatch. Retry the call with a higher-quality screenshot or a more precise description. You can also use the /v1/predict endpoint to let the agent generate descriptions and ground elements in the same loop.
- Error response: {"error": {"code": "string", "message": "string", "request_id": "string"}}
- HTTP 401: invalid API key
- HTTP 402: insufficient credits (you need at least $0.03 for a grounding call)
- HTTP 429: rate limit exceeded
Where this beats brittle automation
Traditional automation tools rely on CSS selectors, XPath, or object models that break when the UI changes or when elements are injected dynamically. The computer use API grounds elements to screen coordinates using visual understanding, so your agent works even when selectors are unreliable. You can build tasks that navigate complex dashboards, fill forms with multiple steps, and handle dynamic content without rewriting your selectors every time the UI updates.
Next steps
Start building a computer use agent that can ground UI elements and perform precise actions. Use the /v1/ground endpoint to map screenshots to coordinates, then combine it with /v1/runs and /v1/sessions for stateful workflows. Get your API key and start experimenting at https://coasty.ai/developers .
Want to see this in action?
View Case Studies