AI agents need interaction data to learn tasks, navigate software, and handle edge cases. Most teams hit a wall: real-world data is sparse, expensive to label, or comes with privacy risks. Synthetic data offers a way to generate large, controllable datasets, but it is not a magic bullet. Real data still anchors performance, especially for agent safety and reliability.
The real data problem: scarcity and cost
High-quality labeled interaction data for agents is hard to come by. A study from Stanford found that the average computer usage dataset used to train assistants contains fewer than 10,000 diverse trajectories. Each trajectory can cost 5 to 15 dollars to collect and label manually. Teams that try to scrape data risk exposing sensitive user actions, leading to privacy violations and compliance issues. When data is limited or risky, synthetic data becomes an attractive alternative.
What synthetic data actually does well
Synthetic data shines when you need volume, coverage, and control. You can generate thousands of interactions with edge cases that rarely appear in production. For example, a synthetic dataset can include rare error states, unusual UI layouts, or multi-step workflows that a real user might trigger only once a year. Researchers at Meta showed that a model trained on synthetic navigation data could outperform a model trained on a real-world dataset by 12% on unseen environments. Synthetic data also eliminates labeling cost after the first generation run.
Where synthetic data falls short
Synthetic data lacks the messiness of real user behavior. Users make mistakes, skip steps, and use workarounds that models never see. If your synthetic workflow is too perfect, the agent may fail when it encounters imperfect inputs. A 2023 evaluation of web agents found that synthetic-only training led to a 23% drop in success rate on complex tasks that involved unexpected UI changes. Synthetic data is best used to supplement real data, not replace it entirely.
The best strategy is to start with real interaction data to establish a baseline, then use synthetic data to expand coverage and test edge cases, and finally validate performance against real-world tasks.
How Coasty fits
Coasty runs computer use agents on real desktops and browsers, which means it can capture realistic interaction data and turn it into synthetic datasets and trajectories. This approach lets teams generate custom synthetic data that reflects real software complexity while maintaining control over privacy and labeling. Coasty’s offering is custom and contact-led: you work directly with the team to design a dataset that matches your agent’s needs.
If you are building or evaluating an AI agent and need more data without the privacy or labeling headaches, consider Coasty’s custom synthetic data service. Book a data call with the Coasty team at https://cal.com/coasty/coasty-data-call to discuss your use case and explore the best data strategy for your project.
Want to see this in action?
View Case Studies