Computer-Mediated Communication Magazine / Volume 1, Number 6 / October 1, 1994 / Page 8

Challenges for Web Information Providers

by John December (john@december.com)

[This article is a chapter in the forthcoming book, The World Wide Web Unleashed (Sams Publishing, 1994).]

Developers are producing information on the Web, and users are accessing it at a breakneck rate. Web traffic over the National Science Foundation (NSFNET) Backbone (just one conduit for exchanging Internet packets) increased from a monthly transfer total of 78 megabytes in December 1992 to 1,056,081 megabytes (over 1 terabyte) in July 1994. It would be cliche to call this an "information explosion." However, this rapid growth in Web use begs the question--what is the meaning of all this "information" transmitted so feverishly? What are the considerations for making sense of it and improving information quality? While there certainly has been an explosion of activity, traffic, servers, and data or "stuff" presented on the Web, an explosion of information leading to knowledge and wisdom, to significance, is still in its nascent stages.

Without tools and methodologies for gathering, evaluating, managing, and presenting information, the Web's potential as a universe of knowledge could be lost.

In an increasingly thin soup of redundant, poor quality, or incorrect information, even the smartest Web spiders won't be very effective. A flood of information unfiltered by the critical and noise-reducing influences of collaboration and peer review can overwhelm users and obscure the value of the Web itself. The Web certainly needs solutions in information discovery and retrieval--indeed, developing intelligent spiders, worms, robots, and ants is crucial to making sense of the Web. The Web will also need new protocols, tools, browsers, hypermedia interfaces, and software. But along with these tools for information discovery and delivery, we need to develop information shaping capabilities--skills to select and present information on the Web.

These information shaping abilities cannot be based on machine intelligence alone. Human wisdom, judgment, and aesthetics must play a part in improving the quality of Web information. In this chapter, I first explore the growth in Web activity in terms of increased Web servers, traffic, and information. I then survey some indicators of a growing diversity and expanding extent of Web communication. These growth indicators dramatize the need to increase information quality. Next, I describe lessons I've learned about providing information through my experiences in gathering, managing, and presenting Internet information. Finally, I discuss what all this adds up to for a notion for information quality for Web information providers.

Web Growth

An extensive user community has developed on the Web since its public introduction in 1991. In the early 1990s, the developers at CERN spread word of the Web's capabilities to scientific audiences worldwide. By September 1993, the share of Web traffic traversing the NSFNET (National Science Foundation Network) Internet backbone reached 1 percent. On December 8, 1993, John Markoff reported on Mosaic on the front page of the business section of The New York Times. At the same time, Mecklermedia's Internet World '93 conference and exposition in New York City featured colorful views of the Web through NCSA's Mosaic, released (working versions for X, Windows, and Mac) just a few months before. By January 1994, the Web comprised 2.6 percent of NSFNET backbone traffic. Web growth, however, includes not just traffic increases. Web growth can also be seen in terms of the number of servers, amount of traffic, and increasingly various kinds of information offered.

Growth in the Number of Web Servers

Matthew Gray, writing in his web page "Growth of the World Wide Web" reports a dramatic increase in the number of servers. Using his "World Wide Web Wanderer" (W4) program, Gray found the following results:

Date     Number of identified Web servers
Jun 1993       130
Sep 1993       204
Oct 1993       228
Nov 1993       272
Dec 1993       623
Mar 1994      1265
Jun 1994      3184

Gray notes that this W4 survey does have limitations, and these figures should be looked on as representative of what W4 could find. However, these figures give a good snapshot of the rapid growth in the number of Web servers--one part of the "information explosion" on the Web. Each server represents the work of one or more administrators (webmasters) and as well as a group of people who are working to provide information in large or small local webs on each server. Each of these servers, then, are potentially large sources of even more information, and are analogous to television stations ready to broadcast to the masses--and the June 1994 figure is like a world with more than 3,000 channels. Moreover, by August 1994, the SG-Scout robot had located over 7,000 Web servers.

Growth in Web Traffic

Along with an increasing number of Web servers, of course, there have been dramatic increases in Web traffic. According to NSFNET backbone statistics, during the first several months of 1994, the Web's share of NSFNET backbone traffic increased from 2.6 percent in January to 6.1 percent in June, surpassing Gopher in terms of bytes transferred in March. (Note, however, that this is only a limited measurement of traffic and only over the NSFNET backbone; other estimates place the Web still behind Gopher in terms of traffic; see also Georgia Tech's NSFNET statistic's page). This growth is significant because Gopher, just a year before, was ahead of the Web--in January 1993, Gopher's share of the backbone was 0.8 percent (the Web's was 0.002 percent), and by June 1993, Gopher was at 1.61 percent (Web 0.515 percent). The Web thus has overtaken Gopher---an information system that had reached wide popularity and had a large base of deployed information in place previous to the Web's widespread use. This "byte ratings" race on the NSFNET backbone dramatizes the pull the Web has for information and traffic.

WWW TRAFFIC OVER NSFNET BACKBONE - IN MEGABYTES/MONTH

MONTH/YEAR       WWW      GOPHER 
Dec 92            78      34,247
Jan 93           122      43,238
Feb 93           512      60,897
Mar 93         3,613      79,024
Apr 93         8,116      89,074
May 93        17,298     103,870
Jun 93        35,701     111,881
Jul 93        48,728     139,006
Aug 93        50,779     148,795
Sep 93        75,401     198,096
Oct 93       122,174     250,785
Nov 93       172,340     291,133
Dec 93       225,443     309,691
Jan 94       269,129     374,681
Feb 94       347,503     396,066
Mar 94       518,084     480,690
Apr 94       671,950     517,625
May 94       799,163     555,708
Jun 94       946,539     567,479
Jul 94     1,056,081     555,089

The above statistics show a dramatic increase in Web traffic. The daily average number of Web bytes exchanged in July 1994 (34,067 megabytes) exceeds the monthly total for May 1993. Also, notice that there was a slight decrease in the number of Gopher bytes transferred from June to July 1994.

Continued on page 9

This Issue / Index /