Grounding UI Elements to Coordinates with /v1/ground
Most desktop automation fails when a button shifts or a layout changes. A brittle selector like button[role='primary'] breaks without warning. Coasty solves this by grounding every action to a real visual element. The /v1/ground endpoint maps a screenshot and a visual description to precise x,y coordinates. You get a stable target for clicks, drags, and typing without maintaining fragile selectors.
How /v1/ground works
The /v1/ground endpoint turns a screenshot into a coordinate pair tied to a UI element. Send a base64-encoded screenshot plus a natural language description of the element you want. The service returns the x and y coordinates relative to the top-left corner of the screenshot. This lets you build reliable interaction flows that stay correct even when layouts change.
curl https://coasty.ai/v1/ground \n -H "Authorization: Bearer $COASTY_API_KEY" \n -H "Content-Type: application/json" \n -d '{
"screenshot": "BASE64_ENCODED_IMAGE",
"description": "a primary button labeled Submit"
}'
Example response:
{
"x": 312,
"y": 456,
"element": "Submit button",
"confidence": 0.96
}Grounding in a full agent loop
You can combine /v1/ground with /v1/predict to build a computer use agent that always targets the right element. First, capture the screen. Then ground the element you want to interact with. Use the resulting coordinates for pyautogui-style moves and clicks. The agent remains robust because it stays attached to the visual reality of the current window.
Ground every action to a real element. Use the returned x,y for clicks and drags. This keeps automation resilient across layout changes.
Why this beats brittle selectors and API-only tools
Selector-based automation depends on stable HTML structures. A single class change or DOM rearrangement breaks the entire flow.API-only tools cannot see the screen. They guess actions from a model or hardcoded mappings, which become outdated quickly.Grounding ties every action to a visual target. The system checks that the described element exists in the current screenshot before resolving coordinates.This approach works on real desktops, browsers, and terminals. You do not need to maintain different selector sets for each application.
Start building resilient agents that see and react to the real UI. Ground every click and drag with /v1/ground. Get your API key at https://coasty.ai/developers and build automation that survives layout changes.