Grounding UI Elements to Coordinates with the Computer Use API
Most automation scripts rely on brittle selectors like XPath or CSS classes. When the UI shifts, those selectors break and you end up with flaky tests or failed jobs. The Coasty Computer Use API solves this with /v1/ground, a free endpoint that takes a screenshot and a description of an element and returns exact x and y coordinates you can use with pyautogui or your own click logic. This lets you build a computer use agent that understands the current screen state and acts on it reliably.
How /v1/ground works under the hood
The /v1/ground endpoint is simple and stateless. You send a base64-encoded screenshot, an instruction describing the element you want, and the cua_version you are using. Coasty processes the image and text together, then returns a JSON object with an actions array. Each action contains an x and y coordinate for a click, plus a confidence score. This is the only way to get reliable, human-like input without managing DOM trees or selectors yourself.
curl https://coasty.ai/v1/ground \n -H "X-API-Key: $COASTY_API_KEY" \n -H "Content-Type: application/json" \n -d '{
"image": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA...",
"instruction": "Click the Submit button on the login form",
"cua_version": "v3"
}'Mapping UI to coordinates step-by-step
- ●Read the latest screenshot from your agent or a headless browser.
- ●Encode the screenshot as base64 and include the correct MIME prefix.
- ●Send a precise instruction describing the element, not a generic click.
- ●Pass the expected cua_version to ensure consistent behavior.
- ●Parse the returned actions array and extract x and y values.
- ●Use those coordinates with pyautogui.click or your own input logic.
Grounding is free, billed at $0.03 per request, and gives you pixel-perfect coordinates based on the actual rendered UI.
Where /v1/ground beats brittle automation
Selectors break when classes change, layouts shift, or frameworks re-render components. With /v1/ground you describe what you want in plain language and let the model find it on the screen. This works across browsers, desktop apps, and custom UIs as long as they render an image. You avoid maintaining fragile selector maps and get a computer use agent that adapts to real UI changes without code updates. It also frees you from parsing complex DOM structures and lets you focus on high-level task logic.
Use /v1/ground to turn any UI description into actionable coordinates and build reliable computer use agents that work on real screen states, not fragile selectors. Ready to start? Get a key at https://coasty.ai/developers and see the full docs at https://coasty.ai/docs.