Most UI automation tools rely on brittle selectors like CSS classes or XPath. When a page layout changes, your script breaks. The Coasty computer use API lets your agent see the screen and understand what it is looking at. The /v1/ground endpoint bridges the gap between visual understanding and precise action. It takes a screenshot and a description of the element you want to interact with, then returns the x,y coordinate to click. This transforms a visual task into a precise sequence of actions that your agent can execute.
How /v1/ground works
You send a POST request to https://coasty.ai/v1/ground with a base64 screenshot and a text description of the UI element. Coasty uses its vision model to identify the element on the screen and returns the bounding box coordinates. The response includes an array of bounding boxes, each with x, y, width, and height. You can then use these coordinates to click or interact with the element directly through your agent. The endpoint costs $0.03 per call.
#!/usr/bin/env python3
import os
import base64
import requests
def ground_ui_element():
# Read API key from environment
api_key = os.getenv("COASTY_API_KEY")
if not api_key:
raise ValueError("COASTY_API_KEY environment variable not set")
# Encode a screenshot as base64
# Replace with the path to your screenshot
with open("screenshot.png", "rb") as f:
img_base64 = base64.b64encode(f.read()).decode("utf-8")
url = "https://coasty.ai/v1/ground"
headers = {
"X-API-Key": api_key,
"Content-Type": "application/json"
}
payload = {
"screenshot": img_base64,
"description": "the blue submit button"
}
response = requests.post(url, json=payload, headers=headers)
response.raise_for_status()
result = response.json()
# Extract the first bounding box
if result.get("boxes"):
box = result["boxes"][0]
print(f"Click at x={box['x']}, y={box['y']}, width={box['width']}, height={box['height']}")
else:
print("No bounding boxes found")
if __name__ == "__main__":
ground_ui_element()Request and response fields
- Screenshot: base64-encoded PNG image of the screen.
- Description: natural language description of the UI element.
- Boxes: array of bounding boxes, each with x, y, width, and height.
- Cost: $0.03 per /v1/ground call.
- Authentication: X-API-Key header or Authorization: Bearer <key>.
Grounding UI elements to coordinates is $0.03 per call.
Where this beats brittle automation
Traditional automation tools depend on stable selectors that break on layout changes. The /v1/ground endpoint lets your agent describe what it sees in plain language and receive precise coordinates in return. This is particularly powerful for web scraping and UI testing where pages are dynamic and selectors are unreliable. Because Coasty’s agent runs on real desktops and browsers, you can ground elements on live interfaces, not mocked versions. You can chain grounding calls with computer use actions like clicking, typing, and scrolling, giving you full control over complex workflows.
The /v1/ground endpoint makes your computer use agent precise and reliable. Use it to turn visual understanding into actionable coordinates. Ready to start? Get your API key at https://coasty.ai/developers and build automated workflows that see and act like a human.
Want to see this in action?
View Case Studies