Desktop automation tools that rely on static selectors or fragile CSS locators fail when UIs change layout. You need your agent to see the screen and understand which part of it refers to a button, input, or menu. The Coasty /v1/ground endpoint solves this. It takes a screenshot and a description of a UI element and returns the exact x,y coordinates for the top-left corner of that element so your agent can click or type with precision.
How /v1/ground works
The endpoint accepts a base64 screenshot and a natural language description of the element you want to target. It returns a JSON body with x and y coordinates in pixels relative to the top-left of the screenshot. Each request costs $0.03. You can call it repeatedly for different elements or reuse the same screenshot to locate multiple things. The coordinates are meant to be passed to pyautogui to perform the final action.
curl -X POST https://coasty.ai/v1/ground \
-H "X-API-Key: $COASTY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"screenshot": "$(base64 -i screenshot.png | tr -d "\n")",
"element_description": "the green submit button in the upper right corner"
}'import base64
import os
import requests
api_key = os.getenv("COASTY_API_KEY")
url = "https://coasty.ai/v1/ground"
with open("screenshot.png", "rb") as f:
screenshot_base64 = base64.b64encode(f.read()).decode("utf-8")
payload = {
"screenshot": screenshot_base64,
"element_description": "the green submit button in the upper right corner"
}
resp = requests.post(url, headers={"X-API-Key": api_key, "Content-Type": "application/json"}, json=payload)
resp.raise_for_status()
data = resp.json()
print("x:", data["x"])
print("y:", data["y"])
# Then use pyautogui to click
import pyautogui
pyautogui.click(data["x"], data["y"])When to use /v1/ground
- You have a screenshot and need an exact top-left coordinate for an element.
- You want to build a computer use agent that clicks dynamically positioned UI items.
- You prefer natural language descriptions instead of maintaining brittle CSS selectors.
- Each request costs $0.03, so use it sparingly for the final click or cursor movement.
Call /v1/ground once per element you need to click, then pass the returned x,y to pyautogui.
Why this beats brittle automation
Traditional scripts rely on fixed IDs, classes, or XPath patterns. When a product manager changes the button class name or shifts a layout, the script breaks. With a computer use agent, you describe what you want in plain language. The agent sees the screen, grounds the description to coordinates, and acts with pixel-perfect precision. This approach adapts to layout changes without you rewriting selectors.
Start building agents that understand UI descriptions and click reliably. Get a free API key and try /v1/ground today at https://coasty.ai/developers.
Want to see this in action?
View Case Studies