Building a reliable UI automation script means you have to click the right element every time. Traditional selectors break when the DOM changes or the layout shifts. You can end up with brittle test flakiness or scripts that silently fail. The /v1/ground endpoint solves this by taking a visual description of a UI element and returning the exact X,Y pixel coordinates on the current screen. It lets your computer use agent ground elements in real time and act with pixel-perfect precision.
How /v1/ground works
You send a POST request to https://coasty.ai/v1/ground. The body must include a base64 screenshot of the screen you want to analyze and a text description of the element you need to locate. The endpoint returns a JSON payload with a coordinates array that contains the x and y values for each matching element. This lets you feed those coordinates directly into a local automation tool like pyautogui or a browser automation library. The current price is $0.03 per request.
import base64
import os
import requests
# Read the API key from the environment (never hardcode it)
COASTY_API_KEY = os.getenv('COASTY_API_KEY')
# Encode a simple screenshot to base64 (replace with your own screenshot)
with open('screenshot.png', 'rb') as f:
screenshot_b64 = base64.b64encode(f.read()).decode('utf-8')
url = 'https://coasty.ai/v1/ground'
headers = {'X-API-Key': COASTY_API_KEY}
payload = {
'screenshot': screenshot_b64,
'description': 'search button in the top right corner'
}
response = requests.post(url, json=payload, headers=headers)
response.raise_for_status()
result = response.json()
print('Coordinates:', result.get('coordinates', []))
# Example output:
# Coordinates: [{'x': 850, 'y': 120}]Key fields and pricing
- POST https://coasty.ai/v1/ground with a base64 screenshot and element description.
- Response includes a coordinates array with x and y fields for each match.
- Each /v1/ground request costs $0.03.
- You can combine /v1/ground with a computer use agent loop that captures, grounds, and acts.
Call /v1/ground to turn visual descriptions into pixel coordinates, then feed those coordinates into your automation.
Where this beats brittle automation
Selectors like CSS classes, IDs, or XPath expressions rely on the HTML structure of a page. When a designer rearranges a layout or a framework updates its markup, your tests break. The /v1/ground endpoint works on the visual layer. It sees the same screenshot your user sees and grounds elements based on their appearance. This makes your computer use agent more resilient to UI changes and less prone to flakiness. You can also use it for accessibility testing or to verify that the right button is visible before clicking.
Add /v1/ground to your computer use agent workflow to ground UI elements in real coordinates instead of brittle selectors. Build robust automation that adapts to layout changes. Get your API key and start testing at https://coasty.ai/developers.
Want to see this in action?
View Case Studies