Grounding UI Elements to Coordinates with /v1/ground
API-only tools break when layouts change or apps use dynamic classes. You need to see the screen and click exactly where a user would. The /v1/ground endpoint takes a screenshot and an element description, then returns the precise x, y coordinates. This lets your computer use agent rely on visual context instead of brittle selectors.
How it works
Send a POST request to https://coasty.ai/v1/ground. The request body needs a base64 screenshot and a text description of the element you want to target. The endpoint returns JSON with an error object if nothing matches, or a data object with an x and y field. Each ground call is billed at $0.03. This is the only cost for turning visual descriptions into clickable coordinates.
curl -X POST https://coasty.ai/v1/ground \
-H "Authorization: Bearer $COASTY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"screenshot": "" // paste base64 screenshot here
},
"description": "the primary submit button"
}'Request fields
- ●screenshot: a base64-encoded PNG or similar image. The server receives visual context.
- ●description: a plain-language description of the UI element. For example, 'the submit button' or 'the email input field'.
- ●Authorization: Bearer <key> header uses the key from COASTY_API_KEY environment variable. Do not hardcode keys.
Response fields
- ●x: the horizontal coordinate in pixels.
- ●y: the vertical coordinate in pixels.
- ●error: present if no matching element is found. If present, the agent should fall back to alternative strategies instead of blindly clicking.
Use /v1/ground to map a screenshot and element description to x,y coordinates, billed at $0.03 per call.
Where this beats brittle automation
CSS selectors and XPath break when developers change class names, reorder elements, or use dynamic IDs. Visual grounding lets your agent recognize the same element by its appearance and context, even if the underlying HTML changes. This is key for reliable computer use agents that must adapt to evolving UIs without constant maintenance.
Combine /v1/ground with /v1/predict and /v1/sessions to build agents that see the screen and act like humans. Navigate to https://coasty.ai/developers to get your API key and start grounding UI elements.