Most UI automation tools rely on brittle selectors like XPath or CSS classes that break with layout changes. The Coasty computer use API gives you something better: you describe what you want the agent to see and click, and it returns exact pixel coordinates. That is the job of the /v1/ground endpoint. It costs $0.03 per request and works on any screenshot you provide.
How /v1/ground works
You send a base64 screenshot and a text description of the element you want to target. The endpoint returns the top-left x,y coordinate of the first matching element. This is a stateless call: you control the input and you control the output. You can use the coordinates with any automation library that accepts pixel coordinates, including pyautogui, SELENIUM, or your own robot controller.
curl -X POST https://coasty.ai/v1/ground \
-H "Authorization: Bearer $COASTY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"screenshot": "$(base64 -i screenshot.png | tr -d "\n")",
"description": "the blue submit button in the top right corner"
}'import base64
import requests
import os
api_key = os.getenv("COASTY_API_KEY")
url = "https://coasty.ai/v1/ground"
with open("screenshot.png", "rb") as f:
img_bytes = f.read()
payload = {
"screenshot": base64.b64encode(img_bytes).decode("utf-8"),
"description": "the blue submit button in the top right corner"
}
resp = requests.post(url, headers={
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json"
}, json=payload)
resp.raise_for_status()
print(resp.json()) # {'x': 1234, 'y': 567, 'confidence': 0.94}What you get back
- x and y are the top-left pixel coordinates of the matching element
- confidence is a float between 0 and 1 indicating how certain the model is
- the endpoint is stateless and does not retain any session or trajectory data
- you can call it as many times as you need within your automation flow
One request, one description, one coordinate pair: /v1/ground costs $0.03.
Where this beats brittle automation
Most automation tools require you to maintain a catalog of selectors that must be updated whenever a layout changes. The Coasty computer use API treats the screen as a visual scene. You describe the element in plain language and it finds the pixel location. The model sees the whole UI context, so it can disambiguate overlapping elements and handle dynamic text. You do not need to pin to specific classes or IDs that might change with a single CSS update.
Combine /v1/ground with the /v1/predict endpoint to build a full computer use agent that sees the screen, grounds elements to coordinates, and acts like a human. Get a key at https://coasty.ai/developers and start clicking with language instead of brittle selectors.
Want to see this in action?
View Case Studies