Turn PyAutoGUI into Structured Actions with the Free Parse Endpoint
You write scripts with pyautogui.click, pyautogui.moveTo, pyautogui.typewrite, and maybe some mouse.dragTo. These scripts work today but break when UIs change, windows move, or coordinates drift. You need a way to turn that imperative code into a reusable, declarative representation that a computer use agent can follow. The free /v1/parse endpoint does exactly that. It takes a pyautogui script and outputs structured actions you can inject directly into Coasty workflows or task runs.
How the Parse Endpoint Works
The Coasty computer use API offers a /v1/parse endpoint that is free. You POST a JSON payload containing the pyautogui code as a string. The server parses the code and returns a list of structured actions with fields like type, x, y, text, and duration. These actions represent what the script would do step by step. You can then use them as input to task steps in workflows or pass them directly to the stateful /v1/sessions/{id}/predict endpoint. No credits are consumed for this parse call.
#!/bin/bash
# Turn a simple PyAutoGUI script into structured actions using the free parse endpoint
COASTY_API_KEY="${COASTY_API_KEY}"
PYAUTOGUI_CODE='''import pyautogui
import time
pyautogui.moveTo(500, 300, duration=0.2)
time.sleep(0.5)
pyautogui.click()
time.sleep(0.3)
pyautogui.typewrite("Hello Coasty", interval=0.05)'''
curl -s https://coasty.ai/v1/parse \
-H "Authorization: Bearer $COASTY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"pyautogui_code": "'"$PYAUTOGUI_CODE""",
"cua_version": "v3"
}' | jq '.'What You Get Back
- ●A JSON array of structured actions representing each pyautogui call.
- ●Each action has a type field like mouse_move, click, type, and duration.
- ●The x and y coordinates are extracted from the script so the agent knows where to move the cursor.
- ●Text input is captured so the agent can type the same string.
- ●The response is ready to be used as the actions payload for /v1/sessions/{id}/predict or as the actions array in a workflow task step.
POST /v1/parse is free and returns structured actions you can reuse in Coasty workflows and sessions.
Where This Beats Brittle Automation
Traditional automation often relies on hardcoded coordinates, DOM selectors, or brittle API wrappers. When a UI changes, those scripts break. Coasty computer use agents instead see the screen and follow structured actions derived from your pyautogui scripts. This means your automation stays robust even if windows move, buttons change labels, or elements shift. You keep the original intent of your script but gain the resilience of computer use. You can also mix parse actions with vision-based actions from /v1/predict or /v1/ground to create hybrid workflows that adapt to real UI changes.
Start converting your existing pyautogui scripts into structured actions with the free /v1/parse endpoint. Use those actions in Coasty workflows, task runs, and sessions to build resilient computer use agents that can handle real desktops. Get your API key at https://coasty.ai/developers.