You have a screenshot and a description like 'click the save button'. You need the exact x,y coordinates to send to pyautogui. The /v1/ground endpoint does exactly that. It takes a base64 screenshot and a natural language description and returns the screen-relative x,y coordinates of the target element.
How it works
POST /v1/ground takes a base64 screenshot, an element description, and an optional cua_version. It returns an x and y coordinate that you can feed into pyautogui or your own action list. The endpoint is priced at $0.03 per call. This step is separate from the predict loop and gives you precise control over where the agent clicks.
curl https://coasty.ai/v1/ground \ -H "X-API-Key: $COASTY_API_KEY" \ -H "Content-Type: application/json" \ -d '{"screenshot": "$(base64 -i screenshot.png)", "description": "click the save button", "cua_version": "v3"}'import os, base64, requests
api_key = os.getenv("COASTY_API_KEY")
with open("screenshot.png", "rb") as f:
img_b64 = base64.b64encode(f.read()).decode("utf-8")
resp = requests.post(
"https://coasty.ai/v1/ground",
headers={"X-API-Key": api_key, "Content-Type": "application/json"},
json={
"screenshot": img_b64,
"description": "click the save button",
"cua_version": "v3"
}
)
resp.raise_for_status()
result = resp.json()
print("x:", result["x"], "y:", result["y"])When to use /v1/ground
- You have a screenshot and want to know where a specific element is.
- You need reliable coordinates for elements that change layout or text.
- You want to separate element location from action execution.
- You are building a custom computer use agent that mixes vision and control.
After grounding, use the x,y in pyautogui.click(x, y) or include it in your action list.
Where this beats brittle automation
Traditional automation relies on brittle selectors like XPath or CSS classes that break when a UI redesign changes an ID or class name. Grounding uses a natural language description and the full visual context. The model can locate the save button even if the class is renamed or the layout shifts. This makes your computer use agent more resilient to real-world UI changes without maintainer intervention.
Add /v1/ground to your computer use loop to map what you see to where you click. Combine it with stateful sessions and task runs to build agents that operate real desktops and browsers. Get an API key and start grounding at https://coasty.ai/developers .
Want to see this in action?
View Case Studies