Traditional UI test frameworks depend on brittle selectors and mock APIs, leading to flaky tests and maintenance overhead. The computer use API lets you drive real desktops, browsers, and terminals by seeing the screen and acting like a human. This enables autonomous end-to-end QA that can handle dynamic UI, third-party auth flows, and app behaviors that mock-only tests cannot reach.
How a self-running QA bot works
A QA bot built with the computer use API follows this cycle: capture a screenshot, send it with the next validation step to the model, receive a list of actions (click, type, scroll), and apply those actions to the target machine. The API keeps a stateful session in memory so the agent can remember context across steps. When the test concludes, the status is reported as succeeded, failed, or cancelled. You can stream live events for progress updates and cancel or resume runs at any time.
import os
import requests
import base64
import json
API_KEY = os.environ.get('COASTY_API_KEY')
BASE_URL = 'https://coasty.ai/v1'
HEADERS = {'X-API-Key': API_KEY}
# Capture screenshot (example) with PIL
from PIL import ImageGrab
img = ImageGrab.grab(bbox=(0, 0, 1920, 1080))
img_bytes = img.tobytes('raw', 'RGB')
screenshot_b64 = base64.b64encode(img_bytes).decode('utf-8')
# Start a stateful session
resp = requests.post(f'{BASE_URL}/sessions', headers=HEADERS, json={})
session = resp.json()
session_id = session['id']
# Validation step: check for a button on the current screen
instruction = 'Verify that the login button is visible and click it if present.'
resp = requests.post(f'{BASE_URL}/sessions/{session_id}/predict',
headers=HEADERS,
json={'screenshot': screenshot_b64, 'instruction': instruction, 'cua_version': 'v3'})
predict = resp.json()
print('Actions:', predict['actions'])
print('Status:', predict['status'])
# Cleanup
requests.delete(f'{BASE_URL}/sessions/{session_id}', headers=HEADERS)Task runs for autonomous QA cycles
- POST /v1/runs provisions a run with task, max_steps, deadline_seconds, and an optional instructions field that appends to the base prompt. The agent drives the machine to completion, and each step is billed at $0.05.
- GET /v1/runs returns a list of recent runs, and GET /v1/runs/{id} gives detailed status, events, and result. You can cancel or resume a run via POST /v1/runs/{id}/cancel and POST /v1/runs/{id}/resume.
- The run can specify on_awaiting_human as pause, fail, or cancel to decide how to handle human approval steps. The cua_version parameter lets you use v3 for controlled steps or v4 for autonomous runs with a pass/fail verifier.
Each agent step costs $0.05, and runs with v4 are autonomous with pass/fail verification.
Where this beats brittle automation
Selecting elements by class or text can break when a UI library reorders classes, injects dynamic IDs, or changes wording. By seeing the screen, the computer use agent understands context and adapts to changes. It can handle third-party auth flows, dynamic modals, and drag-and-drop interactions that selector-based tools cannot reliably automate. This reduces false positives and the need for constant selector maintenance.
Next steps
Start by provisioning a test machine and defining a task that validates core user flows. Build a workflow to orchestrate multiple runs, assertions, and retries. Then, integrate the events stream to get real-time feedback from each test. For more details on endpoints, billing, and workflows, visit the official docs at https://coasty.ai/developers and get your API key.
Want to see this in action?
View Case Studies