Design guide
Computer-use environments.
A computer-use environment lets an agent observe and operate software under a defined task contract. Desktop, browser and mobile environments differ in what the agent can see and do. Reliable training and evaluation require controlled starting states, explicit action boundaries and checks of the actual result.
What is a computer-use environment?
A computer-use environment combines software, an interaction interface and the state needed to execute a task. The agent receives observations, issues actions and observes their consequences. For research, the environment also needs task initialization, recorded attempts and an outcome check.
An automation API alone does not define a benchmark. A screenshot-and-click interface alone does not define a reinforcement learning task. Those become useful research tools when paired with task rules, repeatable setup and meaningful feedback. The RL environment guide explains how rewards and episode endings complete that contract.
Choose desktop, browser or mobile deliberately
| Environment | Typical task surface | Choices to document |
|---|---|---|
| Desktop | Windows, files and workflows across applications. | Window focus, display scale, clipboard and terminal access. |
| Browser | Web pages, tabs, forms and web application records. | Screenshots, DOM or accessibility-tree access; allowed domains. |
| Mobile | Touch interfaces, app switching and device state. | Gestures, keyboard behavior, permissions and emulator configuration. |
These categories overlap: a browser can run inside a desktop task. Name the actual boundary. A browser agent with direct DOM access and a screenshot-only desktop agent may solve the same business task with different information. Report that difference before comparing results.
Specify observations and actions
List every observation channel: screenshots, accessibility trees, URLs, files or structured application state. State its timing, resolution and missing-data behavior. A screenshot taken before a delayed save completes can show a different state from the next observation.
Define actions at the level the agent actually controls. Coordinate clicks and keyboard input test different capabilities from element selectors or direct database writes. Record whether actions are synchronous, what a failed action returns and how the agent waits for a UI change. Privileged evaluator access should remain separate from the agent’s permitted controls.
Worked example: update a customer record
Consider a hypothetical task: change a customer’s delivery address in a desktop application, then save a confirmation file. Several customers have similar names. This example is a proposed research task, not a Coasty product specification.
- Initial state
- A versioned application database, a known customer identifier, an empty output folder and a documented application version.
- Instruction
- The target customer, the new address and the required confirmation format.
- Observation
- The application screenshot after each completed action; no access to privileged reference artifacts or verifier internals.
- Actions
- Mouse and keyboard controls, including app switching, with a declared action and time budget.
- Outcome
- The target record contains the new address, unrelated customer fields are unchanged, and the confirmation file agrees with the saved record.
- Reset
- Restore the database and output folder, clear pending dialogs, and verify that the application is ready.
The outcome check should reject a confirmation file that claims success while the database still contains the old address. It should also reject editing the wrong customer. Vary customer names and addresses independently from the required behavior, then retain the exact variation with the run.
Reset more than the visible screen
Software state includes files, database rows, sessions, open dialogs and queued jobs. Write a reset checklist and verify its postconditions before launching an agent. A clean window can conceal a previous run’s changed record.
For live external services, identify dependencies you cannot restore or freeze. Record relevant changes and separate infrastructure failures from agent errors. A fixed random seed controls only the randomness the environment actually owns; it cannot guarantee that an external website returns identical content.
Learn from published environment designs
OSWorld connects real computer tasks with initial-state setup and execution-based evaluators. BrowserGym provides a shared framework for web-agent tasks and benchmarks. AndroidWorld uses Android applications with task initialization, success checks and cleanup. These are useful references for different environment boundaries, not interchangeable measures of ability.
When choosing a reference, inspect its task versions, dependencies, observation channels and grading rules. A benchmark’s published success rate applies to its own protocol. It does not transfer automatically to your application, agent harness or task distribution.
Define the work before choosing the infrastructure
Start with one representative workflow and a manually inspectable result. Add task variation, concurrent runs and longer horizons after setup and grading are reliable. Keep a trajectory containing observations, actions, timestamps and outcome evidence so failures can be examined and replay attempted where the recorded environment and dependencies permit it.
For Coasty, explore the platform around the workflows you want frontier models to learn. The public computer-use API cookbook provides interaction examples. Its API examples should not be read as a guarantee of compatibility with a particular RL training framework.