Turn PyAutoGUI Code Into Structured Actions With the Free Parse Endpoint
You have a PyAutoGUI script that clicks, drags, and types into a web app. It works on your laptop but fails as soon as the button text changes or the layout shifts. You could chase selectors and flaky XPath strings, or you could ship that script to the Coasty Computer Use API and let the model see the screen and act like a human. First, you need to convert your PyAutoGUI code into the structured actions the API consumes. The free /v1/parse endpoint does exactly that.
How it works
Send your Python script as a POST body to https://coasty.ai/v1/parse. The endpoint parses PyAutoGUI calls and returns a JSON array of structured actions. Each action encodes a screen-reading task followed by a click, type, scroll, or other low-level operation. The response does not require a paid credit, making it ideal for prototyping and integration. You can feed the resulting actions into a stateful session or a full task run. The parse endpoint operates at zero cost, unlike the model-based /v1/sessions and /v1/runs endpoints which bill per step.
import os
import json
import urllib.request
API_KEY = os.getenv("COASTY_API_KEY")
URL = "https://coasty.ai/v1/parse"
pyautogui_script = '''
import pyautogui
pyautogui.moveTo(100, 100)
pyautogui.click()
pyautogui.typewrite("hello")
pyautogui.hotkey("ctrl", "f")
pyautogui.moveTo(200, 300)
pyautogui.dragTo(250, 350, button="left")
'''
req = urllib.request.Request(
URL,
data=pyautogui_script.encode("utf-8"),
headers={
"Content-Type": "text/plain",
"X-API-Key": API_KEY,
},
)
with urllib.request.urlopen(req) as response:
structured = json.load(response)
print(json.dumps(structured, indent=2))
What you get back
- ●An array of actions, each with a type (e.g., click, type, scroll, drag).
- ●Coordinates relative to the top-left of the screen for move and drag.
- ●Typing actions include the text and optional modifiers (e.g., ctrl).
- ●The response is a plain JSON list you can store, version, and compose into workflows.
The /v1/parse endpoint is free and requires no credits, so you can iterate on your structured actions before moving to paid model-based steps.
Where this beats brittle automation
Selector-based tools rely on exact class names, IDs, and XPath strings. A single UI change breaks those selectors and forces you to patch your code. Structured actions generated from parse are semantic and closer to what a human does: they point to regions of the screen and rely on the model to interpret what is visible. When you combine parse actions with the computer use API, the model can see the live screen, decide whether the click is still appropriate, and fall back to a more robust plan if needed. This combination gives you reusable state machines that adapt to changes instead of failing outright.
Start by converting a small PyAutoGUI routine to structured actions with /v1/parse. Integrate those actions into a stateful session or a task run to let a computer use agent perform complex workflows across browsers and desktops. Ready to build? Get your API key at https://coasty.ai/developers and see how easy it is to go from PyAutoGUI scripts to production computer use agents.