The Data Flywheel: Synthetic Data for Self-Improving Agents
Building agents that can actually use tools, navigate interfaces, and reason through tasks is hard. You need data, but real-world interaction data is sparse, expensive, or just hard to get. You also need data to evaluate those agents, and that data often leaks into production. Synthetic data can solve both problems.
Why agents hit a data wall
Agents that act in software environments need sequences of actions and outcomes. Real data is great, but generating it at scale is a bottleneck. A study of 20 large language model code agents found that 94 percent of their performance came from a single dataset, and adding more real-world examples only helped a little. The gains flatten out. You need new types of data, not just more of the same.
The data flywheel in practice
- ●Start with a small real dataset to bootstrap the agent.
- ●Use the agent to generate synthetic trajectories in the target environment.
- ●Curate and label those synthetic examples (ground truth labels are easier when you generate the data yourself).
- ●Retrain the agent on the mixed real+synthetic set.
- ●Repeat, each cycle producing higher-quality synthetic examples that reflect the agent's newer capabilities.
Key tradeoffs to watch
- ●Quality over quantity: synthetic trajectories need careful filtering and validation.
- ●Realism vs. safety: simulated environments must resemble production interfaces enough to transfer, but stay safe from exploits.
- ●Label drift: synthetic data can drift away from real-world distributions over time. Periodically re-synthesize with updated agents or environments.
- ●Computational cost: generating realistic interaction data at scale requires compute, not just time.
The most effective agent fleets keep a data flywheel in motion: real data to bootstrap, synthetic data to iterate, and continuous curation to maintain quality.
How Coasty fits in
Coasty runs computer use agents on real desktops and browsers to capture realistic interaction data. This lets teams describe the exact workflows they care about and receive synthetic datasets and trajectories tailored to their environments. The service is custom and contact-led, meaning you talk to the Coasty data team to scope your needs and receive a solution built around your use case.
If you're building agents that need more data and better evaluation, start the flywheel moving. Book a data call with the Coasty data team to explore how synthetic data can close the loop at https://cal.com/coasty/coasty-data-call .