Subject: Re: Unicode

From: Ross Paterson <[email-masked]>
Date: Tue, 26 Apr 1994 12:53:00 BST

Stephen G Simpson <[email-masked]> hits the nail on the head:
> The question is, how can internationalized software provide
> mixed-language capability?  Unicode and ISO-2022 are two good answers
> to that question, which GB and Big5 don't address at all.

And the next question is, which of these should we prefer?  I used to be
sure it was Unicode, and still have an instinctive bias that way, but it's
difficult to nominate definite reasons.

For a bit of background, let me describe the Unicode-based environment
I'm using to type this message (I'm only using the ASCII subset, of
course, since anything more wouldn't be readable in most places).
I use an X Windows port of some programs from the Plan 9 experimental
Unix system; see

        ftp://research.att.com/dist/plan9man/09utf.ps.Z

for a fuller description.  Unicode, or the Basic Multilingual Plane
of ISO-10646 with which it has been unified, gives you a single 16-bit
character set that covers just about any language you could wish to write.
For compatability with existing ASCII-based software, in Plan 9 the
non-ASCII characters are externally represented using 2 or 3 bytes with
the high bit set, but this is invisible: they are just characters as
far as the interface is concerned.

In Plan 9, a Unicode font is built out of subfonts (= X fonts in the
X version) covering subranges of the characters.  That way, a subrange
that is not heavily used at some installation may have a single subfont
shared by many fonts, while more locally important subranges may have
a variety of subfonts.

With everything set up that way, one can edit text, name files, etc just
as with ASCII, but with many more characters available.  The system is
rather sparse, however, with poor support in many areas (in particular
input).  There is no inherent reason for this; it's just a matter of
investing time -- think of it as a prototype.

Now for mail and news, I can imagine programs that
(a) take my outgoing messages and recode them as MIME messages with
    charset US-ASCII, ISO-8859-1 or UNICODE-1-1-UTF-7 as appropriate
    (the former two are simple subsets of Unicode),
(b) convert incoming MIME messages in various charsets to my local
    encoding before I see them, since all of those sets are subsets
    of Unicode.
so that the outside world will appear in Unicode.

Now much the same kind of environment could be achieved using ISO-2022
-- take Mule and Mosaic-l10n as prototypes.  For internal processing,
one could use a 32-bit code combining an identifier of a character
set (ASCII, GB, JIS etc) with a code point in that character set.
Incidentally, that's approximately what was initially proposed for ISO
10646 before Unicode derailed it.

John Delacour <[email-masked]> writes:
> It is quite absurd to suggest that GB and Big5 should coexist with two
> identical characters having different addresses in thousands of cases.

(This is what will happen in an ISO-2022 encoding, with Big5 replaced
by CNS.)  I would instinctively agree, but it's hard to point at definite
advantages, except that 16 bits is smaller than 32.  Clearly unification
of the various ISO Latin sets is a good thing -- one can write German
with either Latin-1 or Latin-2 (depending on which end of Europe
you're in) and the same text would have different ISO-2022 encodings.
Similarly having lots of different versions of Latin, Greek, Cyrillic
and kana would be a nuisance.

With hanzi, it's not so clear.  Han unification gives some of these
advantages: one can search for the pair of characters U+5317 U+4EAC
("Bei3jing1") and find references to the city in mainland, Taiwanese,
Korean and Japanese documents.  But the unification reflects a compromise
-- many simplified characters have distinct code points, as do some
national variants -- so that this won't always work, and you'll need a
character-synonym table (a bit like searching for color/colour in English
documents).  That's a great pity, but I think this state of affairs is
unavoidable: it's not possible to cover all the languages adequately and
provide the differing unifications required by different groups of users.
Still, maybe some unification is better than none.