Dissertation Information for Bella Hass Weinberg NAME: - Bella Hass Weinberg
DEGREE:
- Ph.D.
DISCIPLINE:
- Library and Information Science
SCHOOL:
- Columbia University (USA) (1980)
ADVISORS: - Jane Anne Therese Hannigan
COMMITTEE MEMBERS: - Michael Edward Davison Koenig - Jane Stevens
MPACT Status: Fully Complete
Title: WORD FREQUENCY AND AUTOMATIC INDEXING
Abstract: The discovery by Estoup and Zipf that
the frequency of words in natural
language yields a predictable graph
stimulated research in many fields.
Several IinguisLS have stated that the
phenomenon is irrelevant to the
problem of grammatical or content
analysis; however, information
scientists have independently
experimented with word frequency data
for the purpose of automatically
extracting content indicators or index
terms from texts. The major methods of statistically-
based automatic indexing include: a
focus on a given level of -frequency
in a text, e.g., high or medium, as
the carrier of the content indicators; the assignment of high weight to words
that occur in the fewest documents'"In
a section; and the computation of
deviation from expected frequency,
which give's high value terms which
occur more frequently. in a single
document than is to be expected from
their frequency in a document
collectton or in general English. The purpose of this study is to test
whether human sets of content
indicators can be characterized by any
or a combination of the above text and
document frequency-based algolithms
The method chosen for the testing of
these problem involved the examination
of the level of frequency of humanly-
assigned index terms in natural
language texts and in a cumulative sort of those texts.
Sixty-five journal articles and their
associated abstracts from the
Proceedings of the American Society of
Civil Engineers and four sets of human
indexing (including author indexing)
for those texis constituted the raw
data set. The major findings were: 1.44% of the
index terms had low frequency in
abstracts (occurred only once) I whtch
leads one to questjon the validity of
increasing the weight of words which
occur mUltiplte times in abstracts" 2. 23% of all index terms and 21% of
major terms as ldentified by indexers
did not occur at all in abstracts, but
did occur in full text, indicating
the9'importance of the latter docĀ·
ument form for indexing research. 3. Index terms_ were spread throughout
all the frequency levels of articles
(28% low; 38% medium; 34% high). About 15% of
terms were found in each of the
extreme intervals very high and very
low relative frequency. 4. 34% of index terms were unique to
their documents in the abstract
collection, but a higher percentage
(39%) were commonly listributed in the
article collection. 5. 25% of index terms were found t~have
highly skewed frequency distributions
in the abstract collection, but in the
article collection, a concentration of
terms was noted at the other end of
the distribution -44.6% of index terms
were commonly distributed in the
collection, indicating that
discrimination techniques which may
work for abstracts do not serve to
characterize the distribution of
humanly assigned index terms in full
text. Linguistic phenomena which account for
these findings Were then examined.
Synonymy played a rather insignificant
role -if a term was not found in a
text, its cross reference was not
likely to be either. Anaphoric and
deictic (referring) mechanisms, as
well as styIistic phenomena, accounted
for most of the suppression of
repetition of content indicators. The implication of the study is: as
tests on multiple sets of human
content in,dicators, including titles,
have demonstrated that their frequency
distribution is not characterized by any of the theories mentioned above,
the utility of statistically-based
automatic indexing 'algorithms for
extracting meaningful content
indicators must be questioned. The main recommendation for further
research is a study of the
interactions of various statistical
phenomena in indexing -including
posting frequency and searching
frequency with'the frequency of words
in natural language texts.
| |
MPACT Scores for Bella Hass Weinberg A = 0
C = 0
A+C = 0
T = 0
G = 0
W = 0
TD = 0
TA = 0
calculated 2008-01-31 06:01:55
Advisors and Advisees Graph
generating graph, please reload |