QA automation often breaks because UI changes expose brittle selectors. You can write XPath or CSS selectors for a login form today and have them break tomorrow. A computer use agent does not need selectors. It sees the screen, understands the instruction, and clicks the button that matches your description. This guide shows how to build a self-running QA bot that drives a browser using the Coasty Computer Use API. You will POST to /v1/predict, capture screenshots, and loop until the task succeeds or fails. Each agent step costs $0.05.
How the QA Bot Works
The bot runs as a loop that captures a screenshot, sends it to the vision API, receives an action, and executes the action. The vision endpoint is POST /v1/predict. It takes a base64 screenshot, an instruction, and the cua_version. The response contains the next action and a status. The status is either done or something else. When the status is done, the agent stops. If it is not done, you capture another screenshot and call predict again. This loop continues until the task completes or you hit the configured max_steps or deadline_seconds.
curl https://coasty.ai/v1/predict \
-H 'X-API-Key: $COASTY_API_KEY' \
-H 'Content-Type: application/json' \
-d '{
"image": "$(base64 -i screenshot.png -b)",
"instruction": "Click the button that says Continue",
"cua_version": "v3"
}'QA Bot in Python
- Install pyautogui to move the mouse and click.
- Loop: capture screenshot, POST to /v1/predict, execute action.
- Each predict call costs $0.05 when billed per agent step.
- You can also use POST /v1/runs to let the server drive the agent to completion for $0.05 per step.
import base64
import os
import requests
import pyautogui
API_KEY = os.environ.get("COASTY_API_KEY")
BASE_URL = "https://coasty.ai/v1"
def capture_and_predict(instruction, cua_version="v3"):
screenshot = pyautogui.screenshot()
with open("screenshot.png", "wb") as f:
screenshot.save(f)
with open("screenshot.png", "rb") as f:
image_b64 = base64.b64encode(f.read()).decode("utf-8")
resp = requests.post(
f"{BASE_URL}/predict",
headers={"X-API-Key": API_KEY},
json={"image": image_b64, "instruction": instruction, "cua_version": cua_version},
)
resp.raise_for_status()
return resp.json()
for step in range(10):
result = capture_and_predict("Click the button that says Continue")
print("Next action:", result)
if result.get("status") == "done":
print("Task complete.")
break
# Execute the action returned by the API
# For example: pyautogui.moveTo(result["x"], result["y"])
# pyautogui.click(result["x"], result["y"])
break # Placeholder for real execution loopEach predict call costs $0.05. Use POST /v1/runs to let the server run the full task for the same rate per step.
Where This Beats Brittle Automation
Traditional test tools rely on static selectors like IDs, classes, or XPaths. If the UI layout changes or the text is slightly different, the test fails. A computer use agent does not know IDs or classes. It sees pixels and text. It understands natural language instructions like 'click the button that says Continue'. It can also handle dynamic forms, popovers, and layout shifts. This makes your QA bot more resilient to changes in the product you test.
You now know how to build a self-running QA bot using the Coasty Computer Use API. Start by capturing a screenshot, posting to /v1/predict, and executing the returned action. Each agent step is $0.05. For longer tests, consider using POST /v1/runs and POST /v1/runs/{id}/resume to manage state and timeouts. Get your API key and start automating like a human at https://coasty.ai/developers.
Want to see this in action?
View Case Studies