Turn PyAutoGUI Code Into Structured Actions With the Free /v1/parse Endpoint
Most desktop automation starts with PyAutoGUI, click, drag, type, press. But those low‑level commands break as soon as a window moves, resizes, or changes text. A selector‑based library like Playwright or Puppeteer can help, but it only works for web apps. You need something that understands what is on the screen and can act like a human, not a list of coordinates. The Coasty Computer Use API solves this with a vision‑driven agent. Before you ask the agent to run a script, you can turn any PyAutoGUI snippet into a structured, semantic action list using the free /v1/parse endpoint.
How it works
The /v1/parse endpoint takes a PyAutoGUI script as input and returns a structured array of actions. You POST the raw PyAutoGUI source to https://coasty.ai/v1/parse with an X-API‑Key header. The server parses the script, extracts mouse and keyboard operations, and maps them to high‑level actions like click_element, type_text, press_key, and scroll. The response body is a JSON object containing an actions array, where each action has a type field, a target field (such as screen, element, or keyboard), and parameters like x, y, text, button, and key. Because this endpoint is free, you can experiment with large scripts without paying per call.
curl -X POST https://coasty.ai/v1/parse \
-H "Authorization: Bearer $COASTY_API_KEY" \
-H "Content-Type: application/json" \
-d @- <<'PYSCRIPT'
pyautogui.moveTo(100, 200)
pyautogui.click()
pyautogui.dragTo(300, 400, duration=0.5)
pyautogui.typewrite("hello world", interval=0.1)
pyautogui.press('enter')
PYSCRIPTWhat parse returns
- ●actions array in the response body
- ●type field: click_element, type_text, press_key, scroll, drag_element
- ●target field: screen, element, keyboard, etc.
- ●parameters: x, y, button, text, key and any other step‑specific fields
POST /v1/parse is free and turns any PyAutoGUI script into a structured action list you can feed to a computer use agent.
Where this beats brittle automation
Traditional automation relies on fixed coordinates and brittle selectors. If a UI element shifts or changes text, the script breaks. A computer use agent, powered by vision, can see the current state of the screen and adapt. By first converting PyAutoGUI code to structured actions with /v1/parse, you get a portable, semantic plan that the agent can execute against real desktops, browsers, and terminals. The agent can handle window moves, dynamic text, and layout changes without you rewriting every click coordinate. This approach is especially valuable when you cannot inject selectors into the application or when you need cross‑platform automation that works on Windows, macOS, and Linux.
Start by posting any PyAutoGUI script to the free /v1/parse endpoint. Use the resulting actions list with a computer use agent to run your automation on real machines. Sign up for a key at https://coasty.ai/developers and try it yourself.