Han Character Font Sharing Across Incompatible Bitmap File Formats

Man-Chi Pong & Fung Fung Lee & Ricky Yeung & Yongguang Zhang *

Technical Report HKUST-CS94-5

March 1994

(HTML version)



Abstract

We identify the problem of sharing Han character font across incompatible bitmap file formats and discuss the related issues. The ideal solution should meet the requirements including independence of character code schemes, such as GB2312-80, Big5, and Unicode, no conversion of existing bitmap font files, storage efficiency, flexibility, extensibility, platform independence, as well as simplicity. Basic principles of how to achieve the above requirements are described. As a realization of the ideas, a new Han character bitmap font specification format, called HBF, has been developed and is presented here. In fact, the HBF approach allows the sharing of fixed-pitch fonts (hanzi bitmap fonts in particular) described in HBF format easily.

1. Introduction

Users of Chinese software may have frequently encountered the following situations. A software package often comes with its own set of Han character (hanzi 1) font files. Since each software system is likely to have its own font format, sharing font files among different vendors' systems or different applications is difficult (even for the experienced users or programmers). As a result, even though a font file's contents may be the same or only be slightly different from others, a user often needs to keep multiple versions of the same font, each in a different format. This not only wastes a lot of storage space, but also creates great confusion for the user and the system administrator, for example, to decide whether a certain set of bitmap files should be removed when a certain software is upgraded or becomes obsolete.

It is desirable to have a unified scheme for sharing the hanzi bitmap fonts among different applications or operating systems. We believe the sharing scheme should have the following features:

Functionality considerations:

Efficiency considerations:

We will point out the inadequacies of the existing systems in Section 2. Principles of sharing hanzi bitmap fonts across incompatible formats are discussed in Section 3. In Section 4, we describe a particular hanzi bitmap font specification format, called HBF, as a realization of the ideas. Section 5 is the conclusion and the Appendix presents some historical notes on HBF.

2. Inadequacies of Existing Systems

Let us consider the existing systems based on the criterion set forth in the previous section.

In commercial systems like ETen [3], Kuo-Chiao [4], and CCDOS [5], bitmaps are stored as raw binary data, usually as contiguous sequences of bytes in the encoding order of the glyphs. The advantages are that the bitmaps are packed efficiently and can be accessed efficiently as well. However, these systems suffer one or more of the following drawbacks:

These varieties of font file formats make reuse of a font difficult. A bitmap font designed and built according to the format in one system can only be used in another one after conversion.

In the Unix/X Window community, there is an industrial open standard format, called BDF (Bitmap Distribution Format) [6], for bitmap font distribution. BDF has many desirable features. Font attributes are expressed in textual form for easy customization. It is also extensible to add new attributes. The format is not hard-wired into any character-code scheme. Bitmaps are encoded in hexadecimal digits. Since BDF is designed for distribution purposes, it is required to compile a BDF file into some more efficient propriety format for actual use.

Although BDF can be used for hanzi bitmap fonts as well, it has several drawbacks. First, extending the font requires specifying the new glyphs in BDF format and then converting the whole new BDF file to the specific propriety format. The overhead could be large, considering the number of characters per hanzi font. More importantly, BDF incurs a huge file size penalty for hanzi bitmap fonts because of the hexadecimal encoding and the unnecessary redundancy of information like per-glyph bounding box. Worse yet, after compiling into the propriety format, the huge BDF file still needs to be kept if there is no "lossless" conversion utility from the propriety format back to BDF.

From the above discussion, we can see that it is difficult to use the existing schemes to share each other's bitmap fonts, and even more difficult to share user defined fonts across platforms.

3. Principles of Sharing Hanzi Bitmap Fonts

We observe that the basic format of most hanzi bitmap font files are almost identical except for some minor differences. Major similarities are:

Minor differences:

Our principle of sharing is to combine the best of both worlds: the rich functionalities and flexibility of some systems (like BDF) and efficiency of others (such as ETen, CCDOS).

The code-order preserving and equal bitmap size characteristics provide compact storage representation and efficient access. What we need is a level of flexibility. Since we would like to achieve sharing without "intrusion," that is, no need to convert existing font data files to other format, a two-level structure is needed. Thus, a separate file is used to describe the attributes of the font and information of segments of glyphs.

Each segment contains the code range, a pointer (e.g. file name) to the actual bitmap file, an offset from the start of the bitmap file to locate the bitmap data, and optionally a per-glyph byte offset increment value to accommodate inter-character padding space. Further information for the segment can also be added. For example, it could accommodate non-upright scanning order or compressed bitmap files in the future.

The separate information file could also provide facilities to exploit the characteristics of the common character code standards, like GB, Big5, JISX0208-1990 [7] and KSC5601-1987 [8], that they only make use of a certain subrange of the second byte of a two-byte character code. For examples, in GB codes, the valid subrange of the second byte is 0x21-0x7E (which becomes 0xA1-0xFE when its high-bit 4, is set to '1'), and in Big5 codes, 0x40-0x7E and 0xA1-0xFE. With the specification of the "second byte code range" in the separate file, the user can simply state the range of one consecutive sequences of planes of codes 5 for each segment.

With segments and second byte code range, it is possible to write applications independent of the character code schemes. For example, the same text display program would work for GB code files and fonts, as well as Big5 ones.

The same ideas can also be applied to a font in three-byte character code scheme, such as CCCII or EACC. We could specify "third byte code range" and "second byte code range" information to describe such a font in a concise way.

For flexibility and easy customization, the separate file should be in plain ASCII text. For instance, sharing a set of symbols could be simply done by editing the textual file. For extensibility, a font attribute should be expressed as a pair of strings of property name and property value.

As an optimization for hanzi, common attributes, such as font bounding box information, should be factored out and put into the header file as well.

4. Realization of the Approach -- HBF

We have put together the ideas described in the previous section and developed a Han character bitmap font specification format (HBF) through a collaborative effort (see historical notes in Appendix A).

4.1 Highlights

An HBF file is a text file with syntax similar to the BDF format. In fact, most of the HBF properties are those of BDF. A property specific to the HBF approach are introduced by a keyword with prefix "HBF_". For details of BDF properties, the reader is referred to the BDF document.

Figure 1 is an example HBF file which specifies a Song style 24x24 bitmap font in GB codes. Note that the common per-glyph property "bitmap bounding box" is factored out into an HBF_BITMAP_BOUNDING_BOX line in an HBF file. This greatly reduces duplicated information for all the glyphs.

The concepts of second byte code range and segment are realized as HBF_BYTE2_RANGE and HBF_CODE_RANGE 6 specification lines, respectively. The HBF_CODE_RANGE line specifies the following values: the code range, the bitmap file name, and the offset of the first bitmap in the bitmap file.

The overhead of the HBF file size comparing to the bitmap file (for cclib simplified characters) is negligible (less than 1%), as shown in Table 1. The table also shows the saving of using HBF file 7 and bitmap file comparing with using BDF or PCF file for the same font. Thus, the HBF approach realizes the goal of low overhead and compact representation.

For details of HBF, the reader is referred to the standard document (as Appendix B) [9].


HBF_START_FONT 1.0
HBF_CODE_SCHEME GB2312-1980
FONT Song24
HBF_BITMAP_BOUNDING_BOX 24 24 0 -2
FONTBOUNDINGBOX 24 24 0 -2
STARTPROPERTIES 5
FAMILY_NAME "song"
ADD_STYLE_NAME "jiantizi"
DEFAULT_CHAR 0xA1A1
COPYRIGHT "This HBF specification file is in the public domain."
NOTICE    "The bitmap files is cclibb.24"
ENDPROPERTIES
CHARS 7614
COMMENT
HBF_START_BYTE_2_RANGES 1
HBF_BYTE_2_RANGE 0xA1-0xFE
HBF_END_BYTE_2_RANGES
COMMENT
HBF_START_CODE_RANGES 2
HBF_CODE_RANGE 0xA1A1-0xA9FE cclib.24  0
HBF_CODE_RANGE 0xB0A1-0xF7FE cclib.24  60912
HBF_END_CODE_RANGES
HBF_END_FONT
Figure 1: An HBF file for a font of characters in GB codes.


font nameb, bitmap file sizehbf file sizebdf file sizepcf file sizehbf/bbdf/(b+hbf)pcf/(b+hbf)
fangsong 24x24548208164219356569614840.00173.521.75
hei 24x24548208164019355309614840.00173.521.75
kai 24x24548208164019355309614840.00173.521.75
song 24x24548208165119354559614840.00173.521.75
song 16x16243648176412658087178600.00255.162.93

Table 1: Comparison of file size for HBF, BDF, and X11R5 PCF. (File sizes are in bytes.)

4.2 More examples

Figure 2 shows the HBF file for the Big5 character font using the bitmap files from the ETen System v3.1. We can see that even though the code ranges for the frequently used and less frequently used Chinese characters are not consecutive and their bitmaps are stored in the same file "stdfont.24k", it is easy to specify them in the HBF file by using two HBF_CODE_RANGE lines with different offset values for the first bitmaps in the file.

If one would like to exclude vendor-specific characters, the last HBF_CODE_RANGE line in Figure 2 can simply be replaced by the line:

        HBF_CODE_RANGE 0xC940-0xF9D5 stdfont.24k 388872

HBF_START_FONT 1.0
HBF_CODE_SCHEME Big5 ETen v3.1
FONT etk24
HBF_BITMAP_BOUNDING_BOX 24 24 0 -2
FONTBOUNDINGBOX 24 24 0 -2
COMMENT An equivalent full X11R5 XLFD conforming font name is
COMMENT -ETen-Kai-medium-r-normal-fantizi-24-24-75-75-c-240-BIG5.1
STARTPROPERTIES 5
FAMILY_NAME "Kai"
ADD_STYLE_NAME "fantizi"
DEFAULT_CHAR 0xA140
COPYRIGHT "This HBF specification file is in the public domain."
NOTICE    "The bitmap files are spcfont.24, stdfont.24k, spcfsupp.24"
ENDPROPERTIES
CHARS 13867
COMMENT
HBF_START_BYTE_2_RANGES 2
HBF_BYTE_2_RANGE 0x40-0x7E
HBF_BYTE_2_RANGE 0xA1-0xFE
HBF_END_BYTE_2_RANGES
COMMENT
HBF_START_CODE_RANGES 4
HBF_CODE_RANGE 0xA140-0xA3BF spcfont.24  0
HBF_CODE_RANGE 0xA440-0xC67E stdfont.24k 0
HBF_CODE_RANGE 0xC6A1-0xC8D3 spcfsupp.24 0
HBF_CODE_RANGE 0xC940-0xF9FE stdfont.24k 388872
HBF_END_CODE_RANGES
COMMENT 0xA140-0xA3BF =>  408 punctuation, numerals, Roman alphabets, etc.
COMMENT 0xA440-0xC67E => 5401 frequently used Big5 Chinese characters.
COMMENT 0xC6A1-0xC8D3 =>  365 ETen specific numerals, Russian alphabets, etc.
COMMENT 0xC940-0xF9D5 => 7652 less frequently used Big5 Chinese characters
COMMENT 0xF9D6-0xF9FE =>   41 other ETen specific characters.
HBF_END_FONT
Figure 2: An HBF file for a font of characters in Big5 codes.

Another advantage of specifying the code ranges separately from the bitmap files can be seen from Figure 3, which shows the "code range section" of the HBF file for the Ming style bitmap font distributed with the PC ETen System v3.1. Comparing Figure 2 and Figure 3, we can see that the two fonts share the same bitmap file "spcfont.24" for the special non-hanzi symbols. The only difference from the HBF file for the Kai (and other) style bitmap font is in the properties FONT, FAMILY_NAME, and the bitmap file name "stdfont.24m", which holds the Ming style Chinese glyphs.


...	
FAMILY_NAME "Ming"	
...	
HBF_CODE_RANGE 0xA140-0xA3BF spcfont.24  0	
HBF_CODE_RANGE 0xA440-0xC67E stdfont.24m 0	
HBF_CODE_RANGE 0xC6A1-0xC8D3 spcfsupp.24 0	
HBF_CODE_RANGE 0xC940-0xF9FE stdfont.24m 388872	
...	

Figure 3: HBF file fragment specifying the Ming style bitmap font of the ETen System

Figure 4 shows the HBF_BYTE_2_RANGE and HBF_CODE_RANGE lines in an HBF file for a bitmap font in Unicode. The HBF_BYTE_2_RANGE is from 0x00 to 0xFF, meaning that the whole coding space representable by the second byte is valid. The HBF_CODE_RANGE lines specify the use of bitmap files for 128 ASCII characters, 128 Latin-1 characters, Japanese hiragana and katakana, and the Han characters defined in Unicode. We can see the flexibility of using various bitmap files to create a font in Unicode. The only requirement is that the glyphs in the various bitmap files have the same size.


...
HBF_START_BYTE_2_RANGES 1
HBF_BYTE_2_RANGE 0x00-0xFF
HBF_END_BYTE_2_RANGES
COMMENT
HBF_START_CODE_RANGES 5
HBF_CODE_RANGE 0x0000-0x007F ascii.24      0
HBF_CODE_RANGE 0x0080-0x00FF latin1.24     0
HBF_CODE_RANGE 0x3040-0x309F hiragana.24   0
HBF_CODE_RANGE 0x30A0-0x30FF katakana.24   0
HBF_CODE_RANGE 0x4E00-0x9FA5 unicodehan.24 0
HBF_END_CODE_RANGES
...
Figure 4: An HBF file fragment for a bitmap font in Unicode.

4.3 HBF in practice

Although it is not part of the HBF standard, a sample API to access HBF fonts has been defined for the developers' convenience. The API and related utility programs have been implemented by Ross Paterson and Yidao Cai. For a hanzi application which needs to retrieve hanzi bitmap, it can easily take advantage of HBF through the sample API.

Accessing HBF data is simple. First, the application opens a font specified by an HBF file and make an API call to retrieve information like the bitmap bounding box, and possibly other calls to get other properties of the font, such as the font bounding box if mixed font display is done. Then the application retrieves each desired bitmap of a character by passing the character code to an API routine. This frees the programmer from worrying about the font format, and even which character code scheme the font is in. Since HBF file is designed to be easily parsed, programmers can write their own library, if deemed necessary, to access the HBF file and bitmap data.

Users of systems like X11 window system or Microsoft Windows could access the HBF specified fonts through some conversion utilities. For example, X11 users can use Paterson's hbf2bdf utility program (see next section) to obtain the BDF file from an HBF file and the bitmap files. After compiling the BDF file into the file format used by X11, such as PCF for X11 Release 5, the BDF file can be deleted. In fact, if we have a modified font manager for X11 which directly supports HBF, no conversion to BDF or PCF would be necessary in the first place.

5. Current Status of HBF

Many people have contributed to the creation of the HBF standard version 1.0 described in this paper. The following publicly and freely available material has been produced 8.

6. Conclusion

The problem of sharing Han character font across incompatible bitmap file formats has been addressed. By taking advantage of the fact that the basic format of most hanzi bitmap fonts are almost identical except for some minor differences, we have outlined the principles of sharing by using a two-level structure. The structure decouples the properties from the bitmaps of a font. This scheme enables a rich set of functionalities and provides an efficient solution. A new and simple hanzi font specification file format, called HBF, was devised as a concrete embodiment of the sharing principles.

We have described the essential features and the status of HBF. HBF (version 1.0) seems adequate to accommodate many of the existing bitmap font formats. Additional attributes, such as bitmap orientation, may have to be included in the future to accommodate other less uncommon cases. This can be easily achieved since extensibility is there by conscious design. In the meantime, simplicity is preferred.

As multilingual computing is getting more popular, sharing font files across systems becomes more important. We believe the design principles as embodied in HBF will provide a useful and constructive reference point for an open standard.

In fact, the HBF approach can be employed to represent any fixed-pitch fonts. A simple extension is to add the line "HBF_BYTES_PER_CODE numberOfBytesPerCharacterCode" in an HBF file to specify the number of bytes in a character code, e.g., 1 for Latin-1 code (and the HBF_BYTE2_RANGE line is not applicable in this case). For a font of character code making up of n bytes, where n < 2, the specification line(s) HBF_BYTEm_RANGE, where 2 <= m <= n, could be added. This direction has been successful in an experiment to optimize X window system to support all kinds of fixed-pitch fonts [10].

7. Acknowledgement

Besides the authors, the following people have contributed actively to the discussion and the development of the HBF standard version 1.0 (the latest version): Yidao Cai, Yuan Jiang, Werner Lemberg, Hongbo Ni, Ross A. Paterson, Stephen G. Simpson, Luns Tee, Ya-Gui Wei, Bo Yang, and Roxin Zhang. Ross Paterson and Yidao Cai have contributed sample HBF API implementations.


Footnotes

* Man-Chi Pong's research was partially supported by the Hong Kong UPGC Research Infrastructure Grant No. RI92/93.EG08 administrated by the Hong Kong University of Science and Technology.

1 Han character is referred to as hanzi in Chinese, kanji in Japanese, and hanja in Korean.

2 Big5 is a de facto standard adopted by many vendors in Taiwan.

3 The term glyph means the bitmap of a character.

4 The "high-bit" is the most significant bit of a byte.

5 A plane of two-byte character codes is identified by the value of the first byte of the code.

6 This is named so for historical reason. A better name might be HBF_SEGMENT.

7 The HBF files also contain other font attributes not shown in the examples.

8 All these can be retrieved by ftp from the computer ifcss.org [129.107.1.155] under the directory /software/fonts/{HBF,d}/.


References

  1. "Codes of Chinese Graphic Characters for Information Interchange, Primary Set (GB2312-80)." National Standards Bureau of China, Beijing, 1980.
  2. Unicode Consortium, The Unicode Standard: Worldwide Character Encoding, vol. 1 and 2. Addison-Wesley, 1991.
  3. "ETen Chinese operating system v3.1." ETen Computer Co., Taipei, Taiwan, 1993.
  4. "Kuo-Chiao Chinese operating system v6.20." Kuo-Chiao Computer Co., Taipei, Taiwan, 1993.
  5. "CCDOS Chinese operating system v4." Institute of Systems Engineering, Ministry of Electronic Industries of China, Beijing, 1988.
  6. "X11R5 Bitmap Distribution Format (BDF) version 2.1." In X window system version 11 release 5 distribution, 1991. MIT X Consortium Standard.
  7. "Standard Codes of Japanese Characters for Information Interchange, Japanese Industrial Standard JIS X 0208." Japan, 1990.
  8. "Standard Codes of Korean Characters for Information Interchange, KSC 5601." Korea, 1987.
  9. M. Pong et al. , "Hanzi Bitmap Font (HBF) File Format Standard Version 1.0." Archived in ftp://ftp.ifcss.org/pub/software/fonts/hbf-discussion/, 1993.
  10. M. Pong, Y. Zhang, and P. Lai, "Optimize X window system to support fixed-pitch fonts using the HBF approach." submitted for publication, Dec. 1993.

Appendix A: Historical Notes

The discussion leading to the HBF standard were conducted through electronic mail across North America, Asia, Europe and Australia. The first round of discussions about HBF started in October 1992 with the active participation of some Chinese software developers, including Fung Fung Lee, Man-Chi Pong, Bo Yang, Ricky Yeung and Yongguang Zhang. Within two weeks of frequent and productive mail exchanges, the major characteristics of HBF were worked out. Independently in January 1993, Yidao Cai, Yuchao Lu, and Xiaofei Wang had mail discussions on standardizing hanzi bitmap file formats, mostly around existing (GB-based) "CCLIB" fonts. As a system administrator of ifcss.org (a major archive site of Chinese software), Xiaofei Wang quickly put together an electronic mailing list, called soft-authors@ifcss.org, for further discussion. The mailing list includes developers of software archived in ifcss.org for ftp distribution, and other interested parties. The total number of people on the list grew from about 20 to 40 as the discussion went on in the following months.

Since the formation of the mailing list on January 30, 1993, the HBF drafting activity regained its momentum. The first draft, version 0.1, was finished on February 17 by Man-Chi Pong, followed by 54 mail messages of responses. Four more revisions were drafted by Pong in the next four months: version 0.2 on February 28, version 0.3 on March 15, version 0.4 on April 30, and version 0.5 on May 28. The final draft, a very minor revision of version 0.5, was posted to ifcss.org on June 4, 1993 as the HBF Standard v1.0. From the day the mailing list started to the day the standard was posted, 157 mail messages consuming a total of 587 KB was posted.

Appendix B: HBF File Format Standard Version 1.0



About the Authors

Address at time of first publication, March 1994

Man-Chi Pong
Department of Computer Science
The Hong Kong University of Science and Technology
Clear Water Bay, Kowloon, Hong Kong
mcpong@cs.ust.hk

Fung Fung Lee
Computer Systems Laboratory
Stanford University
Stanford, CA 94305, U.S.A.
lee@csl.stanford.edu

Ricky Yeung
Sun Microsystems, Inc.
2250 Garcia Avenue
Mountain View, CA 94043, U.S.A.
Ricky.Yeung@eng.sun.com

Yongguang Zhang
Department of Computer Sciences
Purdue University
West Lafayette, IN 47907, U.S.A.
ygz@cs.purdue.edu

Current Address

Man-Chi Pong
Computer Centre
University of Hong Kong
Pokfulam Road, Hong Kong
mcpong@hkusub.hku.hk

Fung Fung Lee
Computer Systems Laboratory
Stanford University
Stanford, CA 94305, U.S.A.
lee@csl.stanford.edu

Ricky Yeung
Microsoft Corporation
One Microsoft Way
Redmond, WA 98052-6399, U.S.A.
ryeung@microsoft.com

Yongguang Zhang
Hughes Research Laboratories
3011 Malibu Canyon Road
Malibu, CA 90265, U.S.A.
ygz@isl.hrl.hac.com


See Also


First converted to HTML by Fung Fung Lee and Ricky Yeung on March 18, 1995.

Formatting last modified on November 25, 1995.