Most UI automation code relies on brittle selectors like XPath or CSS IDs that break when the layout changes. With the Coasty computer use API you can give the agent a natural-language description of what you want to click, and it returns the exact x,y coordinates on the current screen. That grounding step lets the agent act like a human, pointing and clicking where it can see, not where the DOM says it should be.
How /v1/ground works
Use POST /v1/ground to translate a screenshot and a text description of an element into a (x, y) coordinate pair. The request requires a base64-encoded screenshot, an element description, and a cua_version string. The response returns a JSON object with x and y coordinates in pixels relative to the top-left corner of the screen. This endpoint costs $0.03 per call. After grounding, you can pass those coordinates directly to your own UI automation stack or let the agent use them in subsequent task steps.
curl https://coasty.ai/v1/ground \n -H "X-API-Key: $COASTY_API_KEY" \n -H "Content-Type: application/json" \n -d '{
"screenshot": "$(base64 -w0 screenshot.png)",
"element_description": "the OK button at the bottom right",
"cua_version": "v3"
}'
Request fields
- screenshot: base64-encoded PNG of the current screen. The model sees exactly what your user sees.
- element_description: natural-language text describing the target element, e.g., 'the OK button at the bottom right'.
- cua_version: the version identifier for the computer use agent, e.g., 'v3'.
Response fields
- x: horizontal pixel coordinate from the left edge.
- y: vertical pixel coordinate from the top edge.
- Both coordinates are relative to the full screen, not a window.
The grounding endpoint returns the exact (x, y) coordinate for any element you describe, enabling pixel-perfect automation.
Where this beats brittle automation
Traditional automation tools rely on stable selectors that often break when a layout shifts or a framework reorders elements. By grounding actions to coordinates, your computer use agent acts like an actual user, it points to what it sees on the screen. This approach is resilient to UI changes, works across browsers and desktop apps, and lets you scale complex workflows with a single high-level instruction.
Start grounding your UI automation to coordinates with /v1/ground. Build computer use agents that can click, type, and interact with any visible element. Get your API key at https://coasty.ai/developers and start grounding.
Want to see this in action?
View Case Studies