Ground UI Elements to Coordinates with /v1/ground
You have a screenshot of a web page or desktop app. You want to click the Search button on the right side of the header. You could hunt for a class name or XPath that might change. Or you could let the model see the screen and tell you exactly where to click. The /v1/ground endpoint does exactly that. It takes a screenshot and a description of the element and returns x,y coordinates you can pass to pyautogui or another automation library. This endpoint costs $0.03 per call. It is the bridge between visual perception and action.
How it works
Send a POST request to https://coasty.ai/v1/ground with the following fields. The body is a JSON object. The X-API-Key header must contain your API key read from the COASTY_API_KEY environment variable. - screenshot: base64 encoded image of the screen. The base64 string should not include the data URI scheme prefix. The model will examine this image to locate the element. - element: a natural language description of the target element. Be specific. Use words like 'Search button', 'Submit button', 'First table row'. The model will search the image for visual features that match this description. - cua_version: the version of the computer use agent you want to use. Use 'v3' for the default agent. Use 'v4' for the autonomous agent with a pass/fail verifier. The endpoint returns a JSON response with a coordinates object. This object contains x and y keys with integer values. These are the pixel coordinates of the top-left corner of the centered element. You can use these values directly in pyautogui.click(x, y).
curl -X POST https://coasty.ai/v1/ground \
-H "X-API-Key: $COASTY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"screenshot": "$(base64 -i screenshot.png | tr -d "\n")",
"element": "Search button in the top right corner",
"cua_version": "v3"
}'What you get back
- ●A JSON response with a coordinates object.
- ●x and y numeric fields representing the top-left corner of the centered element.
- ●The coordinate system matches your screen resolution and DPI.
- ●You can use these coordinates in pyautogui or any other automation tool.
- ●The endpoint always returns an integer for each coordinate.
Pass the coordinates object directly to pyautogui.click(x, y).
Where this beats brittle automation
When you rely on selectors like class names or XPath, a UI update can break your automation. You might need to maintain a list of selectors and update them when the design changes. The /v1/ground endpoint uses visual understanding. It looks at the screenshot and finds the element based on its appearance. This makes your agents more resilient to design updates. You can describe what you want in plain language. The model handles the complexity of finding the element for you. This approach works for web pages, desktop applications, and any UI with a visible screenshot. It does not depend on a specific framework or DOM structure. It also avoids the need to manage browser driver sessions or server-side rendering. You just need the image of the screen and a clear description.
Putting it together
- ●Capture a screenshot of the current screen using your platform's screenshot API.
- ●Encode the image as base64 without the data URI prefix.
- ●Call /v1/ground with the screenshot and an element description.
- ●Extract the x and y coordinates from the response.
- ●Pass those coordinates to pyautogui.click(x, y) to click the element.
Use the /v1/ground endpoint to ground your UI actions in visual understanding. This endpoint helps you build computer use agents that are more robust and easier to maintain. Once you have a key, you can start building agents that see and click like a human. Get your API key at https://coasty.ai/developers and build reliable automation with Coasty.