How to Automate Any Desktop App with the Coasty Computer Use API
Every desktop application has a UI, buttons, inputs, menus, scrollbars. Traditional automation tries to find those elements with selectors or by guessing coordinates. When the UI changes, your script breaks. The Coasty computer use API skips the guessing game. You send a screenshot and a natural language task, the API returns a list of human-like actions, and you can replay them with pyautogui or integrate them into any workflow. There are no brittle class names or IDs to maintain.
How the computer use API works
The core of the computer use API is the /v1/predict endpoint. You upload a base64-encoded screenshot, provide an instruction in natural language, and specify the computer use agent version. The API returns a JSON payload containing actions and a status. The status starts as 'processing' and eventually becomes 'done'. You capture another screenshot, call /v1/predict again until status is 'done'. Each predict call costs $0.05. This loop lets the agent see the screen, decide what to do next, and act.
#!/usr/bin/env python3
import os
import base64
import requests
import json
import time
def encode_image(path):
with open(path, 'rb') as f:
return base64.b64encode(f.read()).decode('utf-8')
def predict_task(api_key, image_b64, instruction):
url = 'https://coasty.ai/v1/predict'
headers = {'X-API-Key': api_key}
payload = {
'image': image_b64,
'instruction': instruction,
'cua_version': 'v3'
}
resp = requests.post(url, headers=headers, json=payload)
resp.raise_for_status()
return resp.json()
def run_agent(api_key, screenshot_path, task):
image_b64 = encode_image(screenshot_path)
while True:
result = predict_task(api_key, image_b64, task)
actions = result.get('actions', [])
status = result.get('status', '')
print('Actions:', json.dumps(actions, indent=2))
print('Status:', status)
if status == 'done':
break
time.sleep(1)
if __name__ == '__main__':
api_key = os.getenv('COASTY_API_KEY')
screenshot = 'screen.png'
task = 'Click the login button that says "Sign in" and type my email address into the email field.'
run_agent(api_key, screenshot, task)
Key endpoint details
- ●Base URL: https://coasty.ai/v1
- ●Auth header: X-API-Key or Authorization: Bearer <key>
- ●POST /v1/predict costs $0.05 per call
- ●Request body fields: image (base64), instruction (text), cua_version (default 'v3')
- ●Response fields: actions (list of structured actions), status (processing/done)
- ●Loop: capture screenshot → predict → act → repeat until status is 'done'
Call /v1/predict with a screenshot and instruction, then repeat until status is 'done'. Each predict costs $0.05.
Why computer use beats brittle automation
With selectors you must know the class name, ID, or XPath of every element you interact with. When the UI changes, the selector fails and you must hunt for the new one. The computer use agent does not rely on selectors. It looks at the screenshot, understands the visual layout, and decides what to click or type. It works with any desktop app, browsers, native tools, legacy apps, without you having to maintain a mapping. You only need to provide the current screenshot and a natural language description of the goal.
What to build next
Try automating a simple workflow like filling a form, navigating a web portal, or launching and using a third-party desktop tool. Use the code example above as a starting point, then extend it with pyautogui to replay actions on your machine. For longer tasks, you can use the workflow DSL or the task runs API to orchestrate multiple steps and handle retries and human approval. Get your API key at https://coasty.ai/developers and start automating any desktop app with the Coasty computer use API.