OSWorld Style Synthetic Benchmarks for Computer Use Agents

Why OSWorld-style benchmarks matter

OSWorld is a recent benchmark designed to test agents that can use a computer interface. It uses a set of real-world tasks like file management, web navigation, and system configuration. The benchmark publishes scores based on how well an agent completes these tasks. This approach has become a reference point for evaluating autonomous agents. But OSWorld has constraints: it relies on a fixed set of live sessions and a limited number of environments. Scaling it further requires new data sources that are diverse, reproducible, and safe to use in training.

What synthetic data brings to benchmarks

Synthetic data lets you generate task scenarios that mirror real workflows. Instead of waiting for a human operator to complete a task, a model or agent can execute it in a simulated environment. This creates a feedback loop: you can generate variations of the same task, from simple clicks to multi-step workflows, without hitting any real-world limits. A typical synthetic benchmark pipeline might look like this: 1) Define a library of tasks (e.g., open settings, edit a CSV, install a package). 2) Generate a set of initial states and user instructions. 3) Run an agent or model to perform the tasks. 4) Collect the full trajectory, clicks, text inputs, screen observations, and evaluate success. This process can run thousands of times, producing a dataset that is both large and consistent. Synthetic data also removes the risk of exposing real credentials or production systems, which is critical for enterprise workloads. Teams can create benchmarks that cover edge cases and rare workflows that rarely appear in real sessions.

Tradeoffs to watch

• ●Simulation fidelity: synthetic environments must behave enough like real systems to train agents effectively. Poor fidelity leads to agents that fail when they encounter the real world.
• ●Coverage vs. realism: you can generate many tasks, but they may not cover the full range of real-world usage patterns. Balancing breadth with depth is key.
• ●Data quality: synthetic datasets need human validation or expert review to ensure tasks are meaningful and not trivial.
• ●Evaluation alignment: a synthetic benchmark must map closely to real evaluation criteria so that scores reflect true agent capability.

How Coasty fits

Coasty runs computer use agents on real desktops and browsers. This lets the team capture realistic interaction data, including screen states, clicks, text inputs, and system events. That data can be used to create custom synthetic datasets and trajectories for training and evaluating agents and models. Coasty's offering is a custom service led by direct contact, not a self-serve product with fixed packages or published pricing. If you need synthetic data that reflects actual user workflows and real-world system behavior, you can discuss your requirements with the Coasty data team.