Every UI automation tool eventually hits a roadblock. CSS selectors drift after a framework update. Locators break on a localized language pack. XPath expressions become unreadable as a component library changes. The Coasty computer use API solves this by grounding UI elements to raw coordinates. The /v1/ground endpoint takes a screenshot and a visual description and returns a precise x,y pair you can feed to pyautogui. You get visual accuracy without maintaining a brittle selector store.
How it works
The /v1/ground endpoint is a free tool that bridges the gap between what you see and where you click. You send a base64-encoded screenshot, a natural language description of the element, and the cua_version header. The model processes the image, matches the description, and returns a coordinates object with x and y integers. This pair is the exact pixel location you can reuse in your action pipeline.
curl https://coasty.ai/v1/ground \
-H "X-API-Key: $COASTY_API_KEY" \
-H "Content-Type: application/json" \
-d '{"screenshot": "$(base64 -i screenshot.png | tr -d "\n")","description": "a submit button with text Submit","cua_version": "v3"}'Request format and fields
- screenshot: base64-encoded image of the screen, no file extensions in the string.
- description: natural language description of the UI element you want to target.
- cua_version: "v3" or "v4" to select the vision model version.
Response format and fields
- coordinates: object with x and y integers representing the top-left pixel of the element.
- status: optional indication of how confident the model is in the match.
- The endpoint is free and does not consume credits.
Ground once, click many times. Use /v1/ground to resolve a visual description to x,y once, then reuse those coordinates in subsequent actions.
Where this beats brittle automation
Traditional automation relies on selectors that must be updated whenever a UI changes. A button might have a different class name, ID, or text after a hotfix. When that happens, your tests or scripts fail silently or explode with flaky errors. The /v1/ground endpoint uses the actual visual state of the screen. It does not depend on class names, IDs, or data attributes. It sees the UI as a human sees it. This makes agents robust across frameworks, languages, and localized interfaces. You can write a single agent that works on React, Angular, Vue, and native apps without maintaining a separate selector map for each.
Combine /v1/ground with the computer use API task runs and workflows to build agents that see and act. Start building resilient UI automation today at https://coasty.ai/developers to get your API key.
Want to see this in action?
View Case Studies