Desktop automation often breaks because code can't see the actual UI layout. You write CSS selectors or XPath paths, but those break on a single layout change. The /v1/ground endpoint solves this by turning a screenshot and a natural language description of an element into precise x,y coordinates. This gives your computer use agent the ability to click and interact with real UI elements even when selectors do not exist.
How it works
The /v1/ground endpoint maps a screenshot and a text description of a UI element to its screen coordinates. The request requires a base64-encoded screenshot, an element description, and the cua_version parameter. The response includes an element object with x and y coordinates and a bounding box. This grounding step is the bridge between visual perception and executable mouse actions. It costs $0.03 per call.
curl https://coasty.ai/v1/ground \
-H "X-API-Key: $COASTY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"screenshot": "$(base64 -w 0 screenshot.png)",
"element": "Submit button in the top right corner of the login dialog",
"cua_version": "v3"
}'Key request fields
- screenshot: base64-encoded PNG of the screen, captured by your agent
- element: natural language description of the UI component you want to locate
- cua_version: API version, use v3 for current behavior
- Authorization header can be X-API-Key: <key> or Authorization: Bearer <key>
Key response fields
- element.x: horizontal position of the element's top-left corner
- element.y: vertical position of the element's top-left corner
- element.width: width of the bounding box
- element.height: height of the bounding box
- status: success or an error describing why grounding failed
After grounding, use element.x and element.y with pyautogui or any mouse API to click, hover, or type at that location.
Where this beats brittle automation
Traditional automation relies on selectors that must match the DOM structure, class names, or IDs. A single layout update or theme change breaks your script. The /v1/ground endpoint grounds your agent in the visual reality of the screen. It does not need selectors. It sees the screenshot, interprets your description, and returns coordinates that always point to the correct element on that specific frame. This makes your computer use agent robust to UI changes, dark mode toggles, and localized text.
Use /v1/ground to ground your computer use agent in the visual world. Combine it with /v1/predict to build agents that see, interpret, and act on desktop UIs without brittle selectors. Get your API key at https://coasty.ai/developers and start building reliable automation.
Want to see this in action?
View Case Studies