Subject: Han unification in Unicode/ISO-10646

From: Ross Paterson <[email-masked]>
Date: Tue, 10 Aug 1993 15:51:00 BST

Some people have recently been complaining that the above-mentioned
Han unification doesn't go far enough.  I also would have liked it to
go further, but we have to remember that the UniHan range is not just
for Chinese, but also Korean and Japanese (as well as for Vietnamese
Chu Nom).  The primary objective of unification was to cover these
writing systems without blowing out the 16-bit coding space.  It was not
to undo 1200 years of divergence, or even the modern simplifications.
(I'm not connected with either organization; I just scan the relevant
mailing lists.)

Unification of 4 writing systems is not trivial, maybe even impossible.
If you impose a Chinese idea of what characters are identical, you'll mess
things up for the Koreans and Japanese (and if you do that, you've lost
the 16-bit code).  So they went for a less ambitious approach, "they"
being the CJK Joint Research Group of ISO, composed of representatives
of the various governments and of Unicode.  My impression is that the
Unicoders were keener on unification than the various national bodies.

The first principle was the "source separation rule": distinct codepoints
in any reference standard are mapped to distinct codepoints in the
new set.  The reference standards here are GB 2312-80, CNS 11643-1986
(levels 1 and 2: approximately BIG-5), JIS X0208-1990 and X0212-1990
and KSC 5601-1987.  The importance of this is that if you translate text
encoded in a national character set into Unicode and back again, you get
the same text you started with.  This already means, for example, that
the simplified and traditional forms of guo2 (country) need separate
codes, as they are separately encoded in JIS X0208-1990.

The next principle is quoted (via the ISO10646 mailing list) from the
"Explanatory notes for the Unified Ideographic CJK Characters Repertoire
and Ordering [URO], V.1.0," dated November 29, 1991:

   "In this model, three atributes of a written form of a character
    (named X, Y, and Z) are considered, and we suppose a 3-dimensional
    space with these three attributes as its axes.  The X-axis represents
    classes of characters with their meanings or ideographic characters...

    The Y-axis is for variant forms or abstract shape of one ideographic
    character...

    The Z-axis is for type face differences (the actual shape used in
    printing the form) of each variant form....

    Unification is done on the Y-axis, ignoring X-differences (meanings)
    and Z-differences (type faces)."

That is, meanings and fonts don't count, only abstract shapes.  But for
all this Cartesian rationalism, the distinction between a font difference
and a different shape is somewhat subjective.  The application of
this principle has meant that not only do simplified radicals (with a
few exceptions) make for different characters, but such pairs as the
following get different codepoints (514C and 5151 respectively), even
though no source set distinguishes them:

      ;;'   '''';'                        ';,      ;;'
     ;;         ';,                         ;;,   ;'
   ,;'            ';,,                   ,   '  ,'   ,,
,,'',,,,,,,,,,,,,,, ';;'                 ;;'''''''''';;'
    ;;           ;;                      ;;          ;;
    ;;           ;;                      ;;          ;;
    ;;,,,,,,,,,,,;;                      ;''';;'';;'';'
        ;;  ;;   ''                         ,;'  ;;      ,
       ;;   ;;        ,                     ;;   ;;      ;
     ,;'    ;;        ;                   ,;'    ;;      ;,
   ,;'      ;;,,,,,,,,;;               ,,''      ';;;;;;;;'
'''          ''''''''''              ''

Of course the whole process (and not just the Han unification) was
warped by politics.  The result is not perfect, but it is reasonably
comprehensive and consistent, and no other new standard has much
chance now.  It's not nearly bad enough to justify the effort of another
standard -- we may as well make the best of it.

As for the waste of space, this limited unification has probably cost
about 3000 codepoints in comparison with a full Chinese unification (which
would surely have been unacceptable to the Japanese and Koreans) or about
1300 in comparison with a unification of simplified and traditional forms
of radicals.  There is worse waste elsewhere in Unicode/ISO-10646, but
that's a different story.
--
Ross Paterson <[email-masked]>
Department of Computing, Imperial College, London SW7