Internet archives for agents.

An archive preserves a view of the web

An internet archive contains captured web resources and information about those captures. For an agent researcher, that can provide a stable reference for studying what information was available at a particular time. The value depends on knowing what was captured and what was left out.

A crawl is not automatically a complete copy of an interactive application. A saved page may depend on resources, authentication or server behavior that the capture does not include. Keep the distinction between reading preserved evidence and executing a repeatable task explicit from the start.

Keep provenance with the content

The WARC format defines records that can hold captured resources and associated metadata. It supports preserving protocol information and linking related records. WARC is a storage convention; the quality and completeness of a capture still depend on the collection process.

Common Crawl is a public example of an archive corpus with raw web data, metadata and text extracts. Its published crawl collections and URL index illustrate why a resource’s source and capture identity should travel with derived material.

For a research collection, retain the source URL, capture time, response status, content digest and collection version. Record transformations such as text extraction or filtering. If an answer cites an extracted paragraph, a reviewer should be able to trace it back to the captured resource and distinguish the extraction from the original.

A concrete example: research as of a date

Consider a hypothetical agent task: compare the public features of three software products as they appeared on a specified date. Build a collection of relevant captures, record gaps, and give the agent the task’s time boundary. The reference answer should point to the exact captures supporting each comparison.

A current pricing page would not establish what a product offered months earlier. A capture from the right month may still be incomplete or geographically different. The evaluation should allow the agent to identify those limits instead of rewarding an unsupported confident answer.

One useful test varies the evidence: a complete feature page, a missing asset, or two inconsistent captures. Check whether the agent follows the time boundary, cites the right version and distinguishes missing evidence from a negative finding. This is an evidence-reasoning task; it does not require claiming that an entire historical website can execute.

Decide what can be replayed

If archived resources become part of a test environment, specify which behaviors are reconstructed and which are preserved. A local form handler created for an experiment is a new implementation. Labeling it clearly makes later results easier to interpret.

Build a defensible research collection

Define the collection’s scope and permitted uses before assembling a dataset. Preserve rights and access information with each source. Use a documented process for sensitive material, removals and collection updates. Public availability alone should not be treated as a blanket permission for every downstream use.

Separate training and evaluation material deliberately. Similar pages, later copies and repeated templates can cross a naive split. Keep a versioned manifest and document the checks used to detect overlap. For historical tasks, enforce the cutoff both in the supplied evidence and in the grading process.

How this connects to Coasty

Preserved evidence can inform the context and evaluation of agents acting in software. Coasty’s platform brings environments, data and evaluation together. This guide describes research design considerations; it does not announce a Coasty internet archive, a downloadable corpus or a complete web-replay service.