Web archiving

Web archiving is the practice of collecting portions of the World Wide Web, storing them with metadata, and providing access tools so future users can view how sites appeared at particular times. National libraries, universities, and non-profits run crawls; the best-known public service is the Internet Archive's Wayback Machine.

Methods

• Broad crawls with tools such as Heritrix
• Event-based or End of Term campaigns
• On-demand saving (Save Page Now)
• Transactional archiving inside institutions

Stored objects are often packaged as WARC files with capture timestamps and HTTP headers. Interoperability frameworks such as Memento help clients find captures across multiple archives.

Challenges

Robots.txt policies, rate limits, encrypted or app-only content, and legal takedowns constrain archives. Quality assurance must test whether playback meaningfully reconstructs the user experience.

See also

• Digital preservation
• Link rot
• Archive.today
• Memento (HTTP)

Sources

• International Internet Preservation Consortium (IIPC) overviews.
• Internet Archive technical and blog documentation.
• WADL workshop literature on archives vs live-web ephemerality.