PyAutoGUI works great for static scripts, but adaptability is a pain. When UI elements shift, screenshots change, or windows move, your code breaks. The Coasty /v1/parse endpoint turns existing PyAutoGUI scripts into structured action sequences. You ship a screenshot and instruction, get validated actions, and use them with the rest of the computer use API. No selectors, no brittle coordinates.
How the Parse Endpoint Works
The endpoint accepts a base64 screenshot, a natural language instruction, and the version of the computer use agent you want to use. It returns a structured action list that the vision model produces. You can then feed those actions into a session or run them directly. The parse call itself is free. This lets you validate a batch of PyAutoGUI snippets before committing them to a paid vision step.
#!/bin/bash
# Example: Parse a PyAutoGUI snippet using curl
# Replace COASTY_API_KEY with your actual key
COASTY_API_KEY="${COASTY_API_KEY}"
# Replace with your base64-encoded screenshot
SCREENSHOT="$(base64 -i screenshot.png | tr -d '\n')"
INSTRUCTION="Find the button with text 'Submit' and click it."
RESPONSE=$(curl -s -X POST https://coasty.ai/v1/parse \
-H "X-API-Key: $COASTY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"screenshot": "'"$SCREENSHOT"'",
"instruction": "'"$INSTRUCTION"'",
"cua_version": "v3"
}')
echo "$RESPONSE"
What You Get Back
- A JSON array of structured actions such as click, type, scroll, and wait.
- Each action includes a type, coordinates (x, y), and optional parameters.
- The actions are aligned with the cua_version you requested so they match the vision model's output.
- You can save these actions to a file and replay them in a session or workflow step.
- Because the parse step is free, you can test multiple variations before incurring vision costs.
Use /v1/parse to validate and normalize PyAutoGUI snippets before running them in a paid vision loop.
Where This Beats Brittle Automation
Traditional automation relies on exact coordinates, hardcoded element selectors, or fragile image matching. When windows shift or UI changes, you must rewrite your scripts. The computer use approach reads the live screen, understands intent, and performs actions that adapt to layout changes. By converting existing PyAutoGUI snippets into structured actions first, you keep the logic you already have, but gain robustness against visual drift. You then feed those actions into vision-based sessions or workflows for even more adaptability.
Start by converting a few key PyAutoGUI scripts with the free parse endpoint. Use the resulting structured actions in a session or workflow to build a resilient computer use agent. Ready to get started? Get your key at https://coasty.ai/developers .
Want to see this in action?
View Case Studies