Desktop automation often fails because selectors break or wait times drift. You want your computer use agent to point and click exactly where it should, not near it. The /v1/ground endpoint turns a raw screenshot and a human-readable description into precise x, y coordinates. It costs $0.03 per request and lets you feed pixel-perfect locations into pyautogui or any other automation tool. You get reliable clicks, drags, and input without brittle CSS selectors.
How /v1/ground works
The endpoint takes a base64-encoded screenshot, a natural language description of the UI element, and the cua_version to match the model. It returns a JSON object with a status and the coordinates. The status can be 'done' if the element is found and 'failed' if the model cannot locate the element. The response contains x and y integer fields that you can pass directly to pyautogui.click or other automation libraries. This gives you a programmatic bridge between visual context and action.
curl https://coasty.ai/v1/ground \ -H 'X-API-Key: $COASTY_API_KEY' \ -H 'Content-Type: application/json' \ -d '{
"screenshot": "$(base64 -i screenshot.png | tr -d "\n")",
"description": "the submit button in the top right corner",
"cua_version": "v3"
}'When to use /v1/ground
- Map dynamic UI elements that change classes or IDs
- Bridge between natural language and pixel-perfect clicks
- Combine with /v1/predict to generate a sequence of actions
- Use as a fallback when CSS selectors are unreliable
- Integrate into larger workflows that require precise coordinates
Call /v1/ground with a screenshot and element description to get x,y coordinates for pyautogui.click.
Where this beats brittle automation
Traditional automation relies on stable selectors like CSS classes or XPath, which break when frameworks change layouts or adds attributes. /v1/ground operates on the visual layer of the screen, so it works even when selectors change. You provide a description like 'the submit button in the top right corner' and the model returns the exact pixel position, letting you click with high precision. This approach is more robust for rapidly evolving UIs and aligns with how humans interact with desktops and browsers.
Add /v1/ground to your computer use agent to reliably convert visual context into actions. Combine it with /v1/predict for full-screen automation that mirrors human behavior. Get your API key and start building at https://coasty.ai/developers .
Want to see this in action?
View Case Studies