Why Synthetic Data Is the Real Bottleneck for Computer Use Agents

The gap between AI hype and reality

Most public benchmarks for computer use agents report success rates around 40 to 60 percent on controlled tasks. That sounds like progress, but the gap between benchmark performance and production reliability is wide. A 2024 Stanford paper on AI agents showed that training on synthetic trajectories could lift success rates by 10 to 15 percentage points, but only when the synthetic data closely matched the distribution of real user environments. When synthetic data was too clean or too narrow, gains were minimal. The bottleneck isn't model capacity. It's the availability of diverse, grounded interaction scenarios at scale.

Real costs of real data

Collecting interaction data for computer use agents is expensive. Each hour of logged activity requires a live environment, monitoring tools, and manual or automated cleanup. One enterprise team estimated that generating a dataset of 10,000 high-fidelity user sessions cost over $150,000 in infrastructure, labor, and compliance checks. Adding privacy restrictions, multi-step workflows, and real-world edge cases can increase costs by 2x or more. Moreover, reusing data is limited by privacy regulations and the risk of exposing proprietary workflows. Real data is valuable, but its cost and sensitivity make it unsuitable for rapid iteration and large-scale training.

Why synthetic data is different

Synthetic data lets you generate interactions that are identical in structure to real ones but created on demand. You can simulate edge cases, rare workflows, or security-sensitive tasks without touching production systems. A recent study from OpenAI showed that synthetic trajectories could reduce the cost of training computer use agents by up to 70 percent while maintaining comparable performance on standard benchmarks. The key is fidelity: the actions, error states, and environmental variations must match what a real user would encounter. Low-fidelity synthetic data, such as random clicks or simple text matching, doesn't improve agents and can even degrade them by teaching incorrect patterns.

Key tradeoffs and techniques

• Fidelity vs. volume: High-fidelity synthetic data requires more effort upfront but yields better long-term performance.
• Domain adaptation: Synthetic scenarios must be mapped to the specific software stack your agents will use.
• Error modeling: Agents need to see realistic failures, crashes, network blips, UI glitches, to recover gracefully.
• Regulatory compliance: Synthetic data must still respect privacy and legal constraints, especially for sensitive workflows.
• Evaluation alignment: Synthetic datasets should reflect the same task distribution you care about in production to avoid overfitting to benchmarks.

How Coasty fits

Coasty runs computer use agents on real desktops and browsers, capturing realistic interaction data in live environments. This approach produces synthetic datasets and trajectories that reflect actual user behavior, including common errors, UI variations, and edge cases. Coasty's synthetic data service is custom and contact-led: you discuss your specific use cases, target environments, and success metrics with the Coasty team, and they build a tailored data generation pipeline that aligns with your workflows.