Subject: All these strange charcters...

From: Lars-Erik Fredriksson <[email-masked]>
Date: Sun, 18 Feb 90 13:17:35 +0100

Well I just thought it was better to amend just one little bit more on the
topic of "so many charchters so little ink" (i.e. code space)

I have had some very interesting pieces of information from the Unicode
designers, most notably mr Joseph Becker but there are several tidbits from
different sources.

...................

It is worth adding only that one of the members of the CCCII group in
Taiwan, which has amassed the largest database of something like 75,000 Han
glyphs, told us succinctly:

    > "Of these 75,000, only 5,000 are real characters; the remaining
70,000 are variant forms of them."

If you pause to reflect on how these variant glyph forms evolved (mistakes
or innovations by scribes, etc.), and on how many different characters the
average person can be expected to actually read (about 5,000), you will see
that this comment is necessarily true.

Chinese characters have a tendency to induce hallucinations in us
Westerners (especially us Midwesterners ...).   There's even a book on the
subject, DeFrancis's "The Chinese Language: Fact and Fantasy".  Listening
to computer scientists telling each other scare stories about Chinese
characters is a lot like listening to kids camping out in a pup tent
telling each other ghost stories on a stormy night ...  Hey, I heard
there's this really big Chinese dictionary that has 200,000 characters!  Oh
yeah, well I heard there's a really REALLY big one that has 300,000!!

Although "Night of the Living JIS Kanji, Part III" is admittedly a fun
scare, please do not actually believe all the fictional nonsense you hear.
The true fax are more along the following lines:

    > In Chinese, 2,000 characters account for about 99.5% of modern text.
The remainder is a tail of very low-frequency usage, adding about 6,000
more characters for a total of around 8,000 used in generic modern
publishing (e.g. newspapers and magazines).  Unfortunately, a few agencies
such as the Taiwan phone company have encouraged creative
micro-distinctions in name characters, nominally adding about 15,000 forms
that have no use in general text communication.  An educated person's
passive reading vocabulary might be in the range of 5,000 characters,
although the active-use vocabulary would still be much closer to 2,000.  So
few characters can still generate zillions of words, since most Chinese
words are two-character compounds.  The linguist Yuan Ren Chao isolated a
set of about 2,100 basic characters, from which he found all others to be
derived as relatively late micro-differentiations.  The Chinese classics
themselves have a small vocabulary (again 2,000 ballpark) because they
predate the elaborations that later took place in character forms.

    > In Japanese, a study determined that around 2,500 characters are used
in modern publishing, EXCEPT for womens' magazines, whose purple prose has
a vocabulary of about 3,500!

    > Korean is often written with no ideographs at all, so usage levels
are similar to but less than in Japanese.

    > The Research Libraries Group, which maintains a database of
publications in Chinese, Japanese, and Korean, uses the (R)EACC code, whose
14,000-odd characters are sufficient to that purpose.  This the same
magnitude as a corpus collected at UC Berkeley with the specific goal of
enumerating all Chinese characters in modern use.

Scholarly compendia of obscure and obsolete characters run in the range of
50,000-70,000.  These are names of long-vanished people and places, terms
for species of bugs and flowers, and an endless number of micro-variations
and writing errors generated by 4,000 years of creative transcription.  Are
these 50,000-odd characters ever used?  No.  Can anyone read them?  No.
But ah, this form turned up in a Tang manuscript, and no-one knows what it
meant, so we'd better stick it in a dictionary so it won't get lost ...

There is no such thing as "the current Chinese computer standard", but
rather many competing collections.  Here are some of the most important
ones:

Taiwan CCCII     4,808  -- Common
                17,032  -- Next most frequent
                11,517  -- Variant forms of 6,699 of the preceding
                ------
                33,357

Taiwan "Big 5"   5,401  -- Common
                 7,652  -- Next most frequent
                ------
                13,053

China GB2312-80  3,755  -- Level 1
                 3,008  -- Level 2
                ------
                 6,763  -- the XinHua news agency adds 694 characters, for a
 total of
7,457 in modern news use

GB simplified sets "2", "4" and traditional sets "1", "3", "5" forthcoming

NISO Z39.64 EACC 9,876  -- Layer 1 (baseforms)
                 3,605  -- Layers 2-12 (variants)
                ------
                13,481

JIS X0208-1983   2,965  -- Level 1
                 3,388  -- Level 2
                ------
                 6,353

                 5,843  -- Level 3 forthcoming

KS C5601-1987    4,888  -- this includes about 250 pairs of duplicates!

We are currently building a table to cross-reference most of the above
enumerations, in order to evaluate the size of their union, which would
define a collection of all CJK characters that are really in general use.
As usual, it seems that the answer will be in the 14,000 ballpark.

------------------------------------------------

As it turned out, the union of JIS Levels 1 & 2, GB2312 Levels 1 & 2, Big 5
Levels 1 & 2, and KS C5601 came out to about 16,300 characters.

It is worth looking at the ones that were UNIQUE to one of those standards:

    JIS-unique:     700  -- around 100 "kokuji", 500 simplified forms, 100
others

    GB-unique:     2160  -- around 2000 simplified forms, 100 others

    Big5-unique:   6260  -- many micro-variants, rarer characters

    KS-unique:      110

Note that "Big 5" extends much farther into rare characters than the other
standards (the ROC national standard CNS 11643 is virtually identical).  If
you subtract off the excess Big 5 forms that are entering the "rare" zone,
and the also the national simplified variants in the other standards, then
the number of true "common-use" Han characters sinks to about 7,500.

By the way, the union of just "Level 1" of each of these standards is about
7,600 before subtracting anything, and at least 2,500 of those are merely
variant forms, so the actual number of "basic" Han characters is, ahem,
about 5,000.

-------------------

So can we get back to some more constructive discussion, like describing
perhaps good searching facilities in chinese text databases (anyone care to
define multicharacter-words, compounds) and the level of smartness in
various inputsystems or related topics.

mrfung