Most desktop and browser automation scripts rely on PyAutoGUI. You find a button with click(x, y) and hope the layout stays the same. That breaks fast. The Coasty Computer Use API gives you a free parse endpoint that turns raw pyautogui code into a structured, versioned action DSL. You can feed those actions into task runs or workflows and build agents that see the screen and act like a human. No brittle selectors, no manual coordinate maintenance.
How the Parse Endpoint Works
The POST /v1/parse endpoint accepts a PyAutoGUI function call string and returns a structured, versioned action object. You send Python syntax that can be evaluated with ast.literal_eval. The response includes an action_id, a type field (click, drag, scroll, type, etc.), and typed parameters: x, y, button, keys, duration. The endpoint is free, so you can parse many scripts during development without consuming credits. The result is a schema you can use directly in task steps or workflow DSL steps.
import os
import base64
import json
import subprocess
import time
from pathlib import Path
COASTY_API_KEY = os.getenv('COASTY_API_KEY')
BASE_URL = 'https://coasty.ai/v1'
# Example PyAutoGUI code snippet you want to parse.
# In practice, you could read files or run pyautogui scripts and capture output.
PYAUTOGUI_SOURCE = '''pyautogui.click(540, 360)
pyautogui.moveTo(200, 200)
pyautogui.dragTo(400, 400, duration=0.5)
pyautogui.typewrite("Hello, Coasty", interval=0.01)
pyautogui.scroll(5)'''
HEADERS = {
'Content-Type': 'application/json',
'X-API-Key': COASTY_API_KEY,
}
# Small screenshot to make the parse endpoint happy (not used for parsing).
# You can use a blank 640x480 PNG.
EMPTY_SCREENSHOT = base64.b64encode(open('/dev/null', 'rb').read()).decode('utf-8')
def call_parse(pyautogui_code: str) -> dict:
url = f'{BASE_URL}/parse'
payload = {
'pyautogui': pyautogui_code,
'screenshot': EMPTY_SCREENSHOT,
'cua_version': 'v3'
}
resp = requests.post(url, headers=HEADERS, json=payload)
resp.raise_for_status()
return resp.json()
if __name__ == '__main__':
import requests
parsed = call_parse(PYAUTOGUI_SOURCE)
print(json.dumps(parsed, indent=2))
# You can now use actions directly in workflows or task steps.
# Example: if parsed['actions'] is a list, iterate and build a task list.
What the Response Looks Like
- The top-level key is actions, an array of action objects.
- Each action has an action_id, type, and typed parameters.
- Common types include click, drag, scroll, type, and others.
- Parameters are typed: x and y are numbers, keys is a string, button is a string.
- The DSL is versioned, so you can pin a schema version for reproducibility.
Parse your PyAutoGUI script once at development time, reuse the structured actions across workflows and task runs.
Where This Beats Brittle Automation
Traditional automation looks for buttons or inputs by ID or XPath. When a UI updates, the selector breaks and the script fails. With the parse endpoint, you start from human-like coordinate code, then enrich it into structured actions that the agent can see and reason about. The agent still uses pixels but you provide a versioned schema that can evolve with your UI. You can also validate actions before creating workflows, catching errors early. This bridges the gap between rule-based scripts and intelligent computer use agents.
Start converting your PyAutoGUI scripts into structured, reusable actions with the free parse endpoint. Integrate those actions into Coasty workflows and task runs to build robust computer use agents that see the screen and act like a human. Get your API key at https://coasty.ai/developers and begin building at scale.
Want to see this in action?
View Case Studies