Subject: End-of-line problem and transmission format

From: [email-masked]
Date: Tue, 24 Dec 1991 05:28:28 EST

Thanks to CND-CM (HXWZ) and CHPOEM lists and the people who make
them reality to provide wonderful services for Chinese netters.
Chinese computation is also benefited because these two lists
have put many ideas into real test, and provided many experiences
as well as new-found problems.

We had hot discussions on "popular GB + uuencode" vs "pure GB +
escape sequences" early this year.  Both CND-CM and CHPOEM chose
uuencode format.  CND-CM has operated for more than 8 months,
having issued nearly 50 articles, and CHPOEM is also functioned
for over one months.  Now, we can have some retrospect on this
issue.

1. It is pointed out that there is an "end-of-line" problem in
   transferring Chinese text.  I think it is related to the external
   representation (or transmission format) of Chinese text file.

   The end-of-line marker becomes a problem because we transfer
   text file in "binary mode" (uuencoded).  Had we sent the Chinese
   articles as text files (assuming the network was 8-bit clean),
   we wouldn't have this problem.  Do you see people complaining
   that the ENGLISH TEXT received by mail/news has incompatible
   end-of-line markers?  Not likely.  It is because the mail/news
   systems treat each file as a text file and do the necessary
   conversion.  This is part of the job for the software (mail, news,
   kermit, etc) that transfer text file between different architectures.
   The end-users, you and me, posters and readers of CHPOEM-L,
   shouldn't have to worry this.

   Chinese article is line-oriented and should be put in "text mode".
   But uuencode format treats it as pure binary code.

2. Uuencode is more "costly" than escape sequences's approach (e.g zW, HZ).
   I have done some research today and here is the result.

   I chose a large sample of Chinese text files, convert them to
   some common used formats, and compared their sizes.

   Sample:  All CND-CM (HXWZ) articles issued so far (total 48 issues,
		from cm9104a.gb to cm9112c.gb)

   a) Total size:	1211795 bytes.
      Total 14395 line of texts (average 84.2 bytes per line).
      There is a total of 586400 Chinese characters (2 bytes for each
      Chinese character), i.e. 96.78% of the text are Chinese.

   b) The total size after "uuencode": 1670731 bytes.
      The overhead is 37.8% !
      (This is exactly how CND-CM is distributed now.)

      "btoa" is not much better, with total size = 1536579,
      overhead = 26.8%.

   c) The total size in HZ format is 1263055.
      The overhead is only 4.2% !
	(HZ uses "~{", "~}" as escape sequences to quote pure GB code)

   d) The total size in ISO format is 1288685, overhead only 6.3%.
	(ISO format uses "<ESC>$A" to mark the beginning of pure GB
	code, and "<ESC>(B" to mark the beginning of ASCII code)

   e) The total size in HZ+zW format is 1247167.
      The overhead is as little as  2.9% !!
	(zW puts "zW" at the beginning of a line of pure GB codes;
	HZ+zW uses both HZ and zW format in one file and optimizes
	line by line: if one line consist of only GB code and no
	ASCII, "zW" is used, otherwise HZ format is used)

   The conclusion I got is that the extra cost of HZ+zW format
   is about 1/13 of that of the current used uuencode.  The
   conversion between HZ+zW, HZ, zW, GB-code is rather simply.
   (On the other hand, uuencode/uudecode is a lot more complex.)

Then why use "popular GB + uuencode" transmission, when there is
alternative way which 1) has no "end-of-line" problem, 2) produces
smaller file for transmission and saves network bandwidth (plus
many advantages we've gone over in CCNET early this year) ?

Happy Xmas and enjoy the holidays!

Yongguang Zhang
[email-masked]