Most UI automation tools require you to pick a selector, CSS class, XPath, or ID. When an app refreshes, changes its layout, or uses dynamic classes, those selectors break. The /v1/ground endpoint solves this by taking a screenshot plus a natural language description of what you want to click and returning the exact X,Y coordinates, so the agent can act like a real user.
How /v1/ground works
You send a POST request to /v1/ground with a base64 screenshot, an element description, and a cua_version. The service returns a JSON object with x and y coordinates in pixels. This is a one-time lookup, not a trajectory. You ground an element once per step, then use pyautogui.click(x, y) or similar to act. Prices are $0.03 per grounding request.
curl -X POST https://coasty.ai/v1/ground \
-H "Authorization: Bearer $COASTY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"screenshot": "$(base64 -i screenshot.png | tr -d "\n")",
"element_description": "the blue submit button in the top right",
"cua_version": "v3"
}'Request and response fields
- screenshot: base64-encoded PNG image of the UI.
- element_description: natural language description of the target element.
- cua_version: version of the computer use agent model (e.g., v3).
- x: integer coordinate in pixels from the left edge.
- y: integer coordinate in pixels from the top edge.
POST /v1/ground costs $0.03 per call and returns x,y coordinates based on the screenshot and description.
Where this beats brittle automation
Traditional tools rely on stable selectors. When a framework changes class names, moves elements, or uses dynamic IDs, your script fails. With /v1/ground you describe what you want in plain language. The model sees the same UI you see, finds the element, and returns coordinates that work even under layout changes. You do not need to maintain a selector database or rework your tests when the UI evolves.
Combine /v1/ground with /v1/predict for a full computer use agent that sees, grounds, and acts. Build agents that click buttons, fill forms, and navigate complex apps like a human. Get a key at https://coasty.ai/developers and start grounding UI elements today.
Want to see this in action?
View Case Studies