Building agents that can use real desktop software and web applications demands high-quality interaction data. Real-world clicks, keyboard sequences, and window states are expensive to collect, risky to share, and hard to scale. Synthetic trajectories aim to fix that by generating realistic interaction histories that models can learn from and evaluate against.
Why real data alone is not enough
Quality desktop and browser data is rare. Companies with niche tools often have no public logs. Even when logs exist, they may contain PII, internal workflows, or sensitive operations. Sharing raw logs can expose proprietary processes or violate compliance rules. Adding labels, such as intent or success, requires manual annotation, which is slow and expensive.
Estimating the cost of manual labeling
A typical enterprise application can require dozens of distinct user flows. Labeling each flow to answer questions like 'Is this task completed successfully?' or 'What is the user’s goal?' can take hours per instance. For a dataset of 10,000 examples, that workload balloons into thousands of hours of human labor. Synthetic data can dramatically reduce that cost by providing labeled trajectories upfront.
The secret sauce: realistic interaction variety
Synthetic trajectories must behave like humans. They should handle common error states, navigate menus, and recover from unexpected interruptions. Realistically modeling the order of operations, timing, and error handling is what separates noisy simulations from useful training data. A trajectory that never encounters a missing button or a network error is not representative of real usage.
Key tradeoffs to know
- Realism vs. control: Synthetic environments give you full control over scenarios but require careful design to match real-world variability.
- Labeling overhead: Synthetic data can reduce labeling effort, but the initial design and generation pipeline still needs engineering.
- Privacy and compliance: Synthetic trajectories can leave PII and sensitive details behind, simplifying data sharing and compliance.
- Domain coverage: Synthetic data excels at covering edge cases and rare workflows that are hard to capture in production logs.
- Evaluation alignment: The distribution of synthetic trajectories should match the target deployment environment to avoid performance drops.
The takeaway: synthetic desktop and browser trajectories are a practical way to scale agent training and evaluation, but only when they capture realistic interaction patterns and are carefully aligned with production workflows.
How Coasty fits
Coasty runs computer use agents on real desktops and browsers to capture realistic interaction data. This allows teams to produce custom synthetic datasets and trajectories for training and evaluating agents. The offering is a custom, contact-led service, so you discuss your specific use cases and requirements with the Coasty data team to design the right solution.
If you need realistic synthetic desktop and browser trajectories for agent training or evaluation, the next step is to talk to the Coasty data team. Book a data call at https://cal.com/coasty/coasty-data-call to explore how Coasty can help you build the datasets you need.
Want to see this in action?
View Case Studies