Traditional automation relies on brittle selectors like XPath or CSS classes. When the UI changes, your script breaks. The /v1/ground endpoint solves this by taking a screenshot and a natural language description of an element and returning its precise x,y coordinates on the screen. This lets your computer use agent locate buttons, links, and inputs by what it sees, not by fragile selectors.
How it works
The /v1/ground endpoint accepts a base64 screenshot and a description of the target element and returns an object containing the coordinates. The request uses the following fields. The endpoint costs $0.03 per call. You send the screenshot as base64, describe the element in plain text, and the server returns x and y coordinates relative to the top-left corner of the image.
curl -X POST https://coasty.ai/v1/ground \
-H "X-API-Key: $COASTY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"screenshot": "/9j/4AAQSkZJRgABAQEASABIAAD/2wBDAAYEBQYFBAYGBQYHBwYIChAKCgkJChQODwwQFxQYGBcUFhYaHSUfGhsjHBYWICwgIyYnKSopGR8tMC0oMCUoKSj/2wBDAQcHBwoIChMKChMoGhYaKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCj/wAARCAHgA1ADASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwD3eeUdQDz6FF2rZ5W7kfH/9k=",
"description": "The blue Log in button in the top right corner of the page"
}'Request and response fields
- screenshot: base64-encoded image data from your agent’s screen capture
- description: plain text description of the target element, e.g. 'the submit button'
- x: horizontal pixel coordinate of the element’s center
- y: vertical pixel coordinate of the element’s center
- cost: $0.03 per call to /v1/ground
Ground UI elements with /v1/ground to get actionable x,y coordinates from a screenshot and description.
Where this beats brittle automation
With traditional automation, you rely on CSS selectors, IDs, or XPath. When a designer changes a class name or reorders elements, your script fails. A computer use agent that uses /v1/ground can identify the same button by what it looks like and where it appears on the screen. This makes your automation resilient to UI changes, language localization, and layout shifts. You describe what you want the agent to do in natural language and let /v1/ground find the right spot to click or type.
Combine with predict and sessions
You can chain /v1/ground with /v1/predict and /v1/sessions to build full computer use workflows. First, capture a screenshot. Then ground the target element. Finally, use the coordinates in your action sequence. This keeps your agent stateful and trajectory-aware over multiple steps.
Ground UI elements to coordinates with /v1/ground and build computer use agents that work like real users. Get your API key and start automating at https://coasty.ai/developers.
Want to see this in action?
View Case Studies