Classic automation relies on brittle selectors like XPath or CSS IDs. If a page changes, your script breaks. The /v1/ground endpoint lets you describe UI elements in plain language and get exact x,y coordinates back. That means you can drive any app with a computer use agent that truly sees the screen and acts like a human.
How /v1/ground works
The endpoint takes a screenshot and a short description of an element and returns its bounding box coordinates. You send a base64 screenshot, a text instruction like 'the primary submit button', and the current cua_version. The response includes x, y, width, height. You can use those values directly with pyautogui or any other clicking mechanism.
curl -X POST https://coasty.ai/v1/ground \
-H "X-API-Key: $COASTY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"screenshot": "$(base64 -i screenshot.png | tr -d '\n')",
"description": "the primary submit button",
"cua_version": "v3"
}'Request and response fields
- screenshot: base64 encoded PNG image of the current view.
- description: plain-language description of the element you want to ground.
- cua_version: string, typically 'v3', matches the agent version.
- Response includes x: integer, y: integer, width: integer, height: integer representing the bounding box.
Grounding in a full workflow
You can combine /v1/ground with /v1/predict or /v1/sessions to build a full computer use agent. First capture the screen, then ground the element you need, then use the coordinates to click. This keeps your agent stateless yet precise. You can also parse existing pyautogui scripts with /v1/parse and then ground any hardcoded clicks to real UI elements.
Why this matters for computer use agents
- No fragile selectors: the agent relies on visual context, not HTML structures.
- Works on modern web apps and native desktop tools that expose no APIs.
- Coordinates are stable across page reloads as long as layout doesn't shift drastically.
- You can replace hardcoded mouse moves with grounded actions in your existing automation.
Grounding turns natural language descriptions into exact coordinates, making your computer use agent robust across any UI.
Where this beats brittle automation
Traditional automation tools need stable XPath or CSS selectors. When designers change class names or restructure HTML, tests break. A computer use agent that grounds elements to coordinates does not depend on that structural information. It looks at the visual layout and finds the button based on its description. That makes your automation resilient to UI changes and applicable to any app you can see.
Start grounding UI elements to coordinates with /v1/ground and build computer use agents that work on any app. Get your API key at https://coasty.ai/developers.
Want to see this in action?
View Case Studies