Most browser and desktop automation breaks when UI elements move, change text, or have nested structures. You might rely on brittle XPath or CSS selectors that break on a single class rename. The Coasty /v1/ground endpoint turns that pain into a solved problem. You send a screenshot and a human-like description of what you want to click, and it returns exact x,y coordinates you can pass to pyautogui or your own agent. This is how you build a computer use agent that works across real browsers and desktops.
How /v1/ground works
The /v1/ground endpoint costs $0.03 per request. It takes a base64-encoded screenshot, an element description, and returns the x and y coordinates on the image where the element appears. The request format follows the standard Coasty API pattern. Use your COASTY_API_KEY in the X-API-Key header. The endpoint returns JSON with an error object if something fails, or a result object with x and y coordinates. This lets you ground any element on screen without writing custom selectors.
#!/bin/bash
# Read API key from environment
COASTY_API_KEY="${COASTY_API_KEY}"
# Example screenshot (replace with a real base64 image)
SCREENSHOT="$(base64 -i screenshot.png -w 0)"
curl -s https://coasty.ai/v1/ground \
-H "X-API-Key: $COASTY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"screenshot": "'$SCREENSHOT'",
"element_description": "the blue login button at the top right corner"
}' | jq '.'What the response looks like
- A successful response includes an object with x and y coordinates, both integers.
- If the element cannot be located, the body contains an error object with code, message, and request_id.
- You can use these coordinates directly with pyautogui.click(x, y) or pass them into your own agent's move action.
Ground UI elements once, then act with coordinates: POST /v1/ground, then pyautogui.click(x, y).
Where this beats brittle automation
Traditional automation relies on selectors that break when you update your UI framework. Web frameworks inject dynamic IDs and change class names. Cross-browser e2e tools like Cypress point to virtual DOM elements that do not exist in the real browser. The /v1/ground endpoint works on the actual rendered screen. It does not care about internal class names, dynamic IDs, or framework quirks. You describe what you see in natural language, and the model returns the pixel coordinates where that description matches. This makes your computer use agent resilient to layout changes, theme toggles, and slight text variations.
Use cases you can build right now
- Automate login flows by grounding the login button, username field, and submit button on each environment.
- Click through dynamic dashboards where table columns reorder or icons change without updating selectors.
- Fill forms across many regions or themes where UI layouts shift based on user preferences.
- Create a general-purpose agent that navigates browsers and desktop apps by describing what to click, not hardcoding IDs.
Grounding UI elements to coordinates with /v1/ground gives you a reliable foundation for any computer use agent. Use it to build resilient automation that survives layout changes and framework updates. Get your API key at https://coasty.ai/developers and start grounding your UI elements today.
Want to see this in action?
View Case Studies