Subject: Too many characters?

From: Lars-Erik Fredriksson <[email-masked]>
Date: Sun, 18 Feb 90 04:33:08 +0100

So many characters so little codespace?

Well I realize now that some of you are more than tired of
hearing people telling you that 16-bit still will not be enough!
But to tell you the truth this is the most common objection
from sinologues, researchers of Naxi (moso) Tangut, and
manichaean manuscripts from the Tang and what have you. I
myself have a still gnawing feeling that I need to be able to
forcefully say that the codespace in UNICODE is enough for
a standard, and I think what finally made it easier for me was
some lines by Lee Collins to me ( I hope he doesn't mind me
quoting him here! and I am sorry if everyone else in this group
are too fed up with this topic! but the quest of UNICODE
is worth a rehash of things some of you have hopefully
long ago dealt with)

Lee Collins says:
=>Unicode does provide a potential 50K (including User space)
=>for Han. A quick glance at the Dai Kanwa Jiten should convince
=>you that many of the CCCII characters are so rare (they appear
=>exactly once in some ancient dictionary, without reading or
=>meaning) or obsolete that they should never burden a standard
=>for information interchange. How do you type them. They will
=>simply make most Chinese input methods unuseable.
=>
=>If you need to do work in jiaguwen or zhuanwen, then it
=>seems reasonable to use a font mapped to the more
=>common modern forms. If I wanted to put the Chinese
=>classics into machine readable form, I would use the more
=>common forms, not the rare variants of CCCII. If really
=>needed to record a specific form, then I would just store the
=>entire text as a picture. Also, on the Macintosh we are
=>now developing a mechanism for selecting swash and other
=>variants in Roman fonts that could be applied to Chinese.
=>
=>I admire the work done by CCCII, but frankly, I just
=>don't see the need for encoding every squiggle ever made
=>by semi-literate scribes.

Well for me, that is a reasonable, brief summing up that should
be quite enough for most of us but obviously it will do more
harm than good if we when we argument for Unicode with
people bringing up this argument so here is the ultimate posting
on this topic and I won,t do it again!

If the argumentation we have is that there is not enough space
in the Unicode standard lets look at the real number of characters
someone suggested a humongous figure but really there are not
many more than the supposedly slightly over 49,946 charcters
in the Morohashi : Dai Kan-Wa jiten. (The last character in
the supplement is numbered 49,964, but a small, undetermined
number of characters have been assigned prime rather than
discrete numbers - e.g., 366' follows 366, 13359' follows
13359 - making the exact total uncertain.) The zhonghua da
zidian has approx 42,800 single character entries, and some
of the explanation for the larger figure in Morohashi is of
course the large amount of Japanese Kokuji and an
extraordinary number of unusual and a abbreviated forms.

        someone said that there are *very many* (No I won't repeat
        the amazing figure because it is wrong!) charcters in Kangxi
        zidian but ...

The Kanxi zidian lists 49.030 characters and of these 1995
are variant forms, leaving a total of 47,035 different
characters.

Well if this is not enough to convince you here is some more
quotations this time by mr Joseph Becker

=>*Sigh*, OK, here goes.  Nearly all of the designers of Unicode are
=>themselves Han character junkies, but we also have a responsibility to
=>make Unicode usable in practice.  There is no value whatsoever in going
=>crazy counting characters in Han dictionaries.  Let me explain:
=>
=>The objects encoded in Unicode are the characters in "modern text
=>communication use".  Roughly, this might be approximated as the union
=>of the characters (not glyphs) used in printing the text of all major
=>newspapers and magazines in the world this year.
=>
=>More relevantly, if an object is to be used as an encoded character in
=>text communication, then when you send it to someone else's computer,
=>their computer must have a corresponding picture in its font resources,
=>otherwise they won't see it!  This means that you can divide the world
=>roughly into two kinds of characters:
=>
=>    > Public-use characters:  Characters that will get their pictures into a
=>lot of commonly-purchased fonts, so that thousands or millions of
=>people can use them for communication
=>
=>    > Private-use characters:  Characters that may be used within a very
=>small community, perhaps one or two installations, or even individuals,
=>but will not be worthwhile for font vendors to include in general-market
=>fonts
=>
=>There are MANY types of things that fall into the category of
=>"Private-use characters":  rare Han characters, characters of rare
=>languages, private writing systems (e.g. some professor's personal
=>phonetic symbols for a langauge he/she is working on), technical
=>symbols used by specialists in various fields, corporate logos, a million
=>random dingbats, "graphic" cell fragments of 2-dimensional drawings,
=>personal creations (e.g. my signature), creative treatments of text (e.g.
=>whole English words considered as units), and indeed, in general,
=>ANYTHING.
=>
=>Unicode has no desire at all to forbid any of these objects from being
=>communicated between consenting computers.  The Unicode position is
=>simply to insist that a sensible dividing line be placed between Public-use
=>and Private-use objects.  Public-use characters will be represented as
=>text character codes in the normal way, and be communicable in the
=>normal way because font vendors will sell fonts containing their glyphs
=>and lots of people will buy those fonts.  Private-use characters can be
=>handled by a whole variety of DIFFERENT computer mechanisms, e.g.:
=>
=>    > User Space:  Unicode provides 4096 cells that may be assigned
=>character codes by consenting systems, which presumably will share
=>private fonts for rendering them.
=>
=>    > Composition (character sequences):  All major scripts contain
=>extension mechanisms for creating new characters; e.g. for the
=>Greek/Latin/Cyrillic family the mechanism is diacritical marking.  It is
=>absurd to attempt to pre-encode all possible combinations of characters
=>with diacritical marks (a fact that some folks have taken as a challenge
=>...).  What is useful is to make sure that text-handling facilities
=>themselves contain parallel generative mechanisms.  So, Unicode
=>specifies that diacritically marked characters are to be represented as a
=>SEQUENCE of codes, e.g. the base character followed by the mark
=>character.  It is presumed that software will dynamically parse such
=>sequences and do the proper things when rendering, letter-counting,
=>etc.  Since it is appropriate to represent such objects by sequences of
=>characters, the number of such composites representable in Unicode
=>goes far beyond 65,000 (didn't think of that, ha?).  Now, the so-called
=>"65,000+" Han characters are just extensions built up according to
=>strict compositional principles. At some point, it will make far greater
=>sense to represent the rare ones as a cobbled-together composition of
=>fragments, which they are, rather than as unique general-use characters,
=>which status they do not deserve.
=>
=>    > Gaiji:  "Gaiji" by definition means characters that are outside the
=>set that you have defined at the moment.  Since there will always be
=>gaiji, it makes sense to respond to this fact by providing mechanisms that
=>let a user define, draw, and transmit personally-created characters (e.g.
=>I might make one for my signature).  Notice that this sort of mechanism
=>exists in computer systems, but does NOT treat the new objects via the
=>same data structures as ordinary characters.
=>
=>    > Data structures:  People are free to communicate ANYTHING via
=>computer, but that does not mean that anything has to receive a text
=>character code.  There are other data structures in computing.  Suppose
=>your life's work revolves around bridge girder joints.  Suppose there
=>are 1,024 different types of bridge girder joints that you talk about all
=>the time.  Does this mean that they should receive character codes in
=>Unicode?? No, the bridge girder joint people should get together and
=>design and promulgate their own data structures, software, and systems
=>to enable them to deal with these objects.  Precisely the same is true for
=>people who deal with rare Han characters, which for all I know form a
=>smaller community than the bridge girder joint people (no mail on this,
=>please).  Anyhow, the point is that when it comes to specialized
=>applications of interest to a limited community, including rare Han
=>character fanatics, the answer is simply to define and use their own
=>computer mechanisms to handle their data.
=>
=>
=>
=>Summary:  Unicode addresses only "Public-use characters" which meet
=>the criterion that font vendors will sell fonts containing glyphs for them
=>and the general public will buy those fonts.  There are millions of other
=>objects that can and should be communicated via computer, of which
=>rare Han characters are just one example, but those objects are best
=>handled by mechanisms other than assigning character codes to them.

all the best from mrfung

So many characters so little time!

PS. I can get back to this topic if needed but please think carefully
about this before posting! I will quote some more figures of usage
and frequency as I have built up a input-method for chinese I feel
qualified to discuss this topic and I would rather move this discussion
into talking about algorithms for syntactic analyzis of modern chinese
and vocabulary "cihui" and so on...

BTW some of you may find it amusing that one of the originators
of the Unicode-standard proposal mr  Joseph Becker has suggested
a title to a soon to be a major new USDA pamphlet:  "Chemical
Removal of People Who Tell You There are More Than 65,000
Chinese Characters" :-)