Turn PyAutoGUI Code Into Structured Actions with the Free Parse Endpoint
You have a working pyautogui script. It clicks, drags, and types. It works on your machine, but it breaks when the UI changes or the layout shifts. You need an automation that adapts to visual changes instead of brittle coordinates. The /v1/parse endpoint solves this by turning your pyautogui code into a structured, computer use API workflow. It is free, easy to call, and returns actions you can feed into Coasty task runs or workflows.
How it works
The /v1/parse endpoint takes a string of raw pyautogui code and returns a JSON array of structured actions. It expects a base64-encoded screenshot (png or jpeg) and the instruction text that describes what you want to do. The endpoint maps pyautogui functions (click, moveTo, dragTo, press, etc.) into canonical Coasty actions without charging per call. The response contains an actions array with objects like { type: 'move_to', x, y } and { type: 'click', x, y }. Once you have this output, you can run it directly in a Coasty workflow or use it as the initial step of a task run.
import os
import base64
import requests
API_URL = "https://coasty.ai/v1"
api_key = os.getenv("COASTY_API_KEY")
# Base64 encode a screenshot (replace with your actual image path)
with open("screenshot.png", "rb") as f:
screenshot_b64 = base64.b64encode(f.read()).decode("utf-8")
instruction = "Click the login button in the top right corner."
response = requests.post(
f"{API_URL}/parse",
headers={
"X-API-Key": api_key,
"Content-Type": "application/json"
},
json={
"screenshot": screenshot_b64,
"instruction": instruction
}
)
response.raise_for_status()
structured_actions = response.json()["actions"]
print(structured_actions)What you get back
- ●actions: an array of structured action objects
- ●Each action has a type field such as 'move_to', 'click', 'drag_to', 'press', 'type', 'screenshot', 'wait'
- ●Coordinates are relative to the top-left of the screenshot
- ●The response is pure JSON, no extra markup
- ●No charge is applied for parsing (it is free)
Parse your pyautogui code once, then reuse the structured actions in workflows or task runs.
Where this beats brittle automation
PyAutoGUI relies on pixel-level coordinates that drift when windows resize or UI changes. The parse endpoint turns that code into a set of visual actions that the computer use agent can recompute at runtime using a fresh screenshot. This enables agents like Coasty to handle layout shifts, responsive designs, and UI updates without rewriting scripts. You can combine parse output with /v1/ground to resolve element descriptions to coordinates, or feed actions into /v1/sessions for stateful trajectories. The result is a robust automation that behaves like a human user, not a brittle script.
Start building a computer use agent that reads and acts on your pyautogui scripts. Use the free parse endpoint to transform your existing codebase into structured Coasty actions. Then integrate those actions into workflows or task runs for self-driving automation. Get your API key at https://coasty.ai/developers and begin turning your scripts into living agents.