Turn PyAutoGUI Into Structured Actions With the Free Parse Endpoint
Most desktop automation starts with PyAutoGUI, click, type, find a control by coordinates. But those coordinates drift when windows resize, and finding text in a UI without a robust selector engine is slow and fragile. The Coasty computer use API gives you a visual layer that reads a screenshot and acts like a human. But you still need structured actions to orchestrate them. The free /v1/parse endpoint solves this by taking PyAutoGUI code and turning it into a structured action DSL you can run directly against the computer use agent.
How it works
The /v1/parse endpoint accepts a raw PyAutoGUI script and returns a JSON array of structured actions. The request body is a single string field called code. The response is a JSON object with a single field called actions. Each action is an object with type (for example, click, type, move), target (a description of what was matched), and coordinates (x, y). This lets you reuse the same logic across different runs, browsers, or desktop environments without hardcoding screen positions.
curl https://coasty.ai/v1/parse \n -H 'Authorization: Bearer $COASTY_API_KEY' \n -H 'Content-Type: application/json' \n -d '{
"code": "pyautogui.click(500, 300)\npyautogui.write("hello world")"
}'What you get back
- ●actions: JSON array of structured objects
- ●Each action has a type (click, type, move, etc.)
- ●target describes what the action operated on
- ●coordinates are included when relevant
- ●You can serialize this array and replay it with the Coasty computer use agent
POST https://coasty.ai/v1/parse with a code string, read the actions array, and use it to drive the computer use agent.
Why this beats brittle automation
PyAutoGUI scripts that rely on absolute coordinates break when windows move, resize, or change layout. UI frameworks ship different selectors, XPath, CSS, IDs, that can become stale or invalid. The Coasty computer use API reads the actual screen, matches an element description, and moves the cursor exactly where the human would. By turning PyAutoGUI into structured actions first, you decouple the logic from pixel-level details. You can then run those actions on any machine, any browser, or any desktop environment the agent can access.
What you can build next
- ●Replay the parsed actions with POST /v1/sessions/{id}/predict to run them on a real desktop
- ●Combine parsed workflows with the task-run API for multi-step automation
- ●Use the parse endpoint as a bridge to build MCP tools that expose your automation to Cursor or Claude Desktop
The free /v1/parse endpoint gives you a clean, structured representation of your automation logic without extra cost. Start with a simple PyAutoGUI script, parse it, and run it with the computer use API. Get your API key at https://coasty.ai/developers and start building agents that see and act like a human.