Many automation scripts rely on brittle selectors like XPath or CSS classes. They break on layout changes, theming, or localized text. The computer use API solves this by grounding a description of an element in a screenshot to actual screen coordinates. You send a base64 screenshot, a text description, and get back an x,y pair ready for pyautogui. This costs $0.03 per call and works on any desktop, browser, or app.
How /v1/ground works
The endpoint accepts a screenshot and an element description. It returns the top-left corner of the first matching element as x and y. The request body is a JSON object with three fields. The screenshot must be a base64-encoded PNG, Jpeg, or WebP. The instruction is a short human-readable description like 'the blue button that says sign in'. The cua_version defaults to v3, you can omit it or specify v3 or v4. The response contains x and y coordinates in integers. Use those coordinates directly in pyautogui.click(x, y) or pyautogui.moveTo(x, y).
#!/bin/bash
# Set your API key from the environment
COASTY_API_KEY="${COASTY_API_KEY}"
BASE_URL="https://coasty.ai/v1"
# Base64 encode a screenshot (adjust to your actual file)
SCREENSHOT=$(base64 -w0 screenshot.png)
# Build the request body
PAYLOAD=$(cat <<EOF
{
"screenshot": "$SCREENSHOT",
"instruction": "the blue button that says sign in",
"cua_version": "v3"
}
EOF
)
# Call grounding endpoint
curl -s -X POST "${BASE_URL}/ground" \
-H "x-api-key: ${COASTY_API_KEY}" \
-H "Content-Type: application/json" \
-d "$PAYLOAD"Grounding vs brittle selectors
- No fragile selectors: XPath and CSS classes break on layout changes, theming, or localized text.
- Works on any UI: browsers, native apps, or custom dashboards without custom selectors.
- Human-like understanding: you describe what the element looks like in natural language.
- Cost-effective: $0.03 per grounding call, far cheaper than maintaining complex selector maps.
- Simple integration: use the returned x,y with pyautogui for immediate action.
Use /v1/ground to turn a screenshot and description into exact x,y coordinates, then feed that directly to pyautogui.
Where this beats brittle automation
Traditional automation tools require you to write and maintain a library of selectors for every element. Every UI change forces you to update those selectors, often breaking your entire workflow. The computer use agent approach uses vision to locate elements based on what they look like, not how you describe them in code. You can describe a button by its color, text, and surrounding context, and the model returns a precise coordinate. This makes your scripts resilient to redesigns, theming, and localized copy changes. It also eliminates the need to reverse-engineer proprietary UI structures or build custom selectors for each application.
Start grounding UI elements with /v1/ground. Build automation that adapts to changes instead of breaking. Get your API key at https://coasty.ai/developers.
Want to see this in action?
View Case Studies