[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Archiving Websites



An announcement of this thesis was just posted to the web-archive mailing list:

Timo Burkard, Herodotus: A Peer-to-Peer Web Archival System, Master's
Thesis, June, 2002.

http://www.pdos.lcs.mit.edu/papers/chord:tburkard-meng.pdf

"In this thesis, we present the design and implementation of Herodotus, a peer-to-peer web archival
system. Like the Wayback Machine, a website that currently offers a web archive, Herodotus
periodically crawls the world wide web and stores copies of all downloaded web content. Unlike
theWayback Machine, Herodotus does not rely on a centralized server farm. Instead, many individual
nodes spread out across the Internet collaboratively perform the task of crawling and storing the
content. This allows a large group of people to contribute idle computer resources to jointly achieve
the goal of creating an Internet archive.  Herodotus uses replication to ensure the persistence of
data as nodes join and leave. Herodotus is implemented on top of Chord, a distributed peer-to-peer
lookup service.  It is written in C++ on FreeBSD.  Our analysis based on an estimated size of the
World Wide Web shows that a set of 20,000 nodes would be required to archive the entire web, assuming
that each node has a typical home broadband Internet connection and contributes 100 GB of storage."

=========================================================
 Cal Lee
 University of Michigan             Phone: 734-647-0505
 School of Information
 http://www-personal.si.umich.edu/~calz/
   "To the extent that archival organizations are concerned
    with current record practices and the content of future
    holdings, they cannot long delay plans for appraising,
    accessioning, and servicing this new [computer] media
    of communication."
                                  - Meyer Fishbein, 1970

A posting from the Archives & Archivists LISTSERV List!

To subscribe or unsubscribe, send e-mail to listserv@listserv.muohio.edu
      In body of message:  SUB ARCHIVES firstname lastname
                    *or*:  UNSUB ARCHIVES
To post a message, send e-mail to archives@listserv.muohio.edu

Or to do *anything* (and enjoy doing it!), use the web interface at
     http://listserv.muohio.edu/archives/archives.html

Problems?  Send e-mail to Robert F Schmidt <rschmidt@lib.muohio.edu>