Login

Publications  •  Project Statistics

Glossary  •  Schools  •  Disciplines
People Search: 
   
Title/Abstract Search: 

Dissertation Information for Bella Hass Weinberg

NAME:
- Bella Hass Weinberg

DEGREE:
- Ph.D.

DISCIPLINE:
- Library and Information Science

SCHOOL:
- Columbia University (USA) (1980)

ADVISORS:
- Jane Anne Therese Hannigan

COMMITTEE MEMBERS:
- Michael Edward Davison Koenig
- Jane Stevens

MPACT Status: Fully Complete

Title: WORD FREQUENCY AND AUTOMATIC INDEXING

Abstract:
The discovery by Estoup and Zipf that

the frequency of words in natural

language yields a predictable graph

stimulated research in many fields.

Several IinguisLS have stated that the

phenomenon is irrelevant to the

problem of grammatical or content

analysis; however, information

scientists have independently

experimented with word frequency data

for the purpose of automatically

extracting content indicators or index

terms from texts.
The major methods of statistically-

based automatic indexing include: a

focus on a given level of -frequency

in a text, e.g., high or medium, as

the carrier of the content indicators;
the assignment of high weight to words

that occur in the fewest documents'"In

a section; and the computation of

deviation from expected frequency,

which give's high value terms which

occur more frequently. in a single

document than is to be expected from

their frequency in a document

collectton or in general English.
The purpose of this study is to test

whether human sets of content

indicators can be characterized by any

or a combination of the above text and

document frequency-based algolithms

The method chosen for the testing of

these problem involved the examination

of
the level of frequency of humanly-

assigned index terms in natural

language texts and in a
cumulative sort of those texts.

Sixty-five journal articles and their

associated abstracts from the

Proceedings of the American Society of

Civil Engineers and four sets of human

indexing (including author indexing)

for those texis constituted the raw

data set.
The major findings were: 1.44% of the

index terms had low frequency in

abstracts (occurred only once) I whtch

leads one to questjon the validity of

increasing the weight of words which

occur mUltiplte
times in abstracts"
2. 23% of all index terms and 21% of

major terms as ldentified by indexers

did not occur at all in abstracts, but

did occur in full text, indicating

the9'importance of the latter docĀ·

ument form for indexing research.
3. Index terms_ were spread throughout

all the frequency levels of articles

(28% low;
38% medium; 34% high). About 15% of

terms were found in each of the

extreme intervals very high and very

low relative frequency.
4.
34% of index terms were unique to

their documents in the abstract

collection, but a higher percentage

(39%) were commonly listributed in the

article collection.
5.
25% of index terms were found t~have

highly skewed frequency distributions

in the abstract collection, but in the

article collection, a concentration of

terms was noted at the other end of

the distribution -44.6% of index terms

were commonly distributed in the

collection, indicating that

discrimination techniques which may

work for abstracts do not serve to

characterize the distribution of

humanly assigned index terms in full

text.
Linguistic phenomena which account for

these findings Were then examined.

Synonymy played a rather insignificant

role -if a term was not found in a

text, its cross reference was not

likely to be either. Anaphoric and

deictic (referring) mechanisms, as

well as styIistic phenomena, accounted

for most of the suppression of

repetition of content indicators.
The implication of the study is: as

tests on multiple sets of human

content in,dicators, including titles,

have demonstrated that their frequency

distribution is not characterized by
any of the theories mentioned above,

the utility of statistically-based

automatic indexing 'algorithms for

extracting meaningful content

indicators must be questioned.
The main recommendation for further

research is a study of the

interactions of various statistical

phenomena in indexing -including

posting frequency and searching

frequency with'the frequency of words

in natural language texts.

MPACT Scores for Bella Hass Weinberg

A = 0
C = 0
A+C = 0
T = 0
G = 0
W = 0
TD = 0
TA = 0
calculated 2008-01-31 06:01:55

Advisors and Advisees Graph

generating graph, please reload

Students under Bella Hass Weinberg

ADVISEES:
- None

COMMITTEESHIPS:
- None