Ground UI Elements to Coordinates with the /v1/ground Computer Use API
Web scrapers and bots often break when a site redesigns a class name or changes a layout. You could keep a massive library of brittle selectors, or you could let the agent see the screen and ground each UI element to a concrete x,y coordinate. The Coasty /v1/ground endpoint does exactly that. It takes a screenshot and a description of the element you want to target, then returns a precise x,y coordinate relative to the top-left corner of the viewport. Once you have that coordinate, you can click, type, or scroll with pyautogui or any other mouse/keyboard driver. This post shows you how to call /v1/ground reliably and integrate it into a computer use agent.
How /v1/ground works
The /v1/ground endpoint maps a screenshot and an element description to a coordinate pair. It does not drive the UI itself. It just reads the visual context and returns the best-fit location for the element you asked for. This is a one-shot call. The request body contains a base64-encoded screenshot, an element description, and the version of the computer use agent you are using (cua_version). The response is a JSON object with a single coordinate field, an x and y value in pixels. This value is relative to the top-left corner of the screenshot you sent. You can use it directly with pyautogui.click(x, y) to interact with the element.
Example using curl. This reads COASTY_API_KEY from the environment, encodes a screenshot as base64, and sends it to /v1/ground. Replace BASE64_SCREENSHOT with the actual base64 data of a screenshot you have taken.
COASTY_API_KEY="$COASTY_API_KEY"
BASE64_SCREENSHOT=$(base64 -w 0 screenshot.png)
curl -s https://coasty.ai/v1/ground \
-H "X-API-Key: $COASTY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"screenshot": "'"$BASE64_SCREENSHOT"'",
"element_description": "The search input field",
"cua_version": "v3"
}'Request and response details
- ●screenshot: base64-encoded image of the screen you want to analyze. It must be a full-screen capture or a viewport snapshot. The higher the resolution, the more precise the returned coordinate will be.
- ●element_description: A natural-language description of the UI element you want to target. It can be as specific or general as you like. The model uses this description plus the visual context to infer the best-fit location.
- ●cua_version: The version of the computer use agent you are using. Use "v3" for most cases. "v4" is the autonomous mode with a pass/fail verifier, but grounding still returns a coordinate from the visual analysis.
- ●Response object: { "x": 1234, "y": 5678 }. These pixel values are relative to the top-left corner of the screenshot you sent. You can use them directly with pyautogui.click(x, y) or pyautogui.moveTo(x, y).
Use the returned x,y directly with pyautogui.click(x, y).
Where this beats brittle automation
Bots that rely on CSS selectors, XPath queries, or hardcoded DOM IDs break as soon as a site changes a class name or reorders elements. You might spend hours maintaining a library of selectors. The /v1/ground endpoint lets you ground each action to a concrete screen coordinate derived from a real-time screenshot. This means your agent can still click the search input even if the class name changes to input-abc-123. You only need to keep the element description up to date, not the underlying selectors. Combined with the Coasty computer use API, you can build agents that see the screen, ground elements to coordinates, and perform actions with the same reliability as a human operator.
Grounding UI elements with /v1/ground gives you precise, human-like targeting without brittle selectors. It works with any screenshot, so you can apply it to web browsers, desktop apps, or arbitrary UI tooling. Start building robust computer use agents today. Get your API key at https://coasty.ai/developers.