Convert PyAutoGUI Code to Structured Actions with the Free Coasty Parse Endpoint
You have a working PyAutoGUI script that clicks buttons, types text, and drags windows. But as apps evolve, your hard‑coded pixel offsets break. You end up with fragile selectors and endless maintenance. The Coasty parse endpoint solves this by turning your PyAutoGUI code into a structured, version‑controlled action graph. It costs nothing and works with any screenshot you provide.
How the parse endpoint works
Send a PNG screenshot and the Python code you want to convert as a JSON payload to POST /v1/parse. The server parses the code, maps each pyautogui call to a structured action (click, type, scroll, etc), and returns a canonical action list. This lets you store the result, reuse it across runs, and version it like a regular source file. The endpoint is free and has no per‑action billing.
import os
import base64
import requests
from pathlib import Path
def parse_pyautogui_to_actions(
screenshot_path: str,
pyautogui_code: str,
api_key: str,
base_url: str = "https://coasty.ai/v1"
) -> dict:
"""Convert PyAutoGUI code into a structured action graph.
Args:
screenshot_path: Path to the PNG screenshot.
pyautogui_code: Python code using pyautogui functions.
api_key: Your Coasty API key from COASTY_API_KEY env var.
base_url: Coasty API base URL.
Returns:
JSON response with actions and metadata.
"""
# Read the screenshot and encode it as base64.
with open(screenshot_path, "rb") as f:
screenshot_bytes = f.read()
base64_screenshot = base64.b64encode(screenshot_bytes).decode("utf-8")
# Build the request payload.
payload = {
"screenshot": base64_screenshot,
"code": pyautogui_code
}
headers = {
"X-API-Key": api_key,
"Content-Type": "application/json"
}
resp = requests.post(
f"{base_url}/parse",
headers=headers,
json=payload
)
resp.raise_for_status()
return resp.json()
if __name__ == "__main__":
api_key = os.getenv("COASTY_API_KEY")
if not api_key:
raise RuntimeError("Set COASTY_API_KEY before running the example.")
# Example screenshot and code.
screenshot_path = Path("screenshot.png")
pyautogui_code = (
"import pyautogui\n"
"pyautogui.moveTo(500, 300)\n"
"pyautogui.click()\n"
"pyautogui.write('Hello world!')\n"
"pyautogui.moveTo(800, 400)\n"
"pyautogui.doubleClick()"
)
result = parse_pyautogui_to_actions(
screenshot_path=str(screenshot_path),
pyautogui_code=pyautogui_code,
api_key=api_key
)
print(result)What you get back
- ●A list of actions where each has a type (click, type, scroll, etc) and coordinates, plus optional parameters like text or duration.
- ●A canonical representation that you can serialize to JSON, store in a Git repository, or reference in later runs via /v1/sessions/{id}/predict.
- ●No per‑action billing; the endpoint is free and you pay for the prediction steps when you run the actions on a machine.
POST /v1/parse is free and turns PyAutoGUI code into a reusable action graph.
Why this beats brittle automation
Selector‑based tools rely on IDs, classes, or XPath that can change with UI updates. The parse endpoint starts from the actual screenshot and the code you run, producing actions that are grounded in pixels and user intent. You can then feed those actions into a computer use agent that sees the screen, validates results, and recovers from layout drift. This makes your pipelines more resilient and easier to maintain across app versions.
Stop rewriting selectors every time the UI changes. Use the free /v1/parse endpoint to turn your PyAutoGUI scripts into structured actions, then run them with a computer use agent on real desktops and browsers. Get your API key at https://coasty.ai/developers and start building robust automation that sees and acts like a human.