NYS Open Records Discussion Must Recognize Technical Requirements

Jon Bosak, Ithaca, NY
Last updated 26 June 2008

The normally somnolent world of computer data format standards has been roiled over the last year by a clash between the biggest names in the computer business, a struggle that has spread to include not just industry giants but also national governments and large sectors of the programming community as well. Everyone agrees that it's time to move beyond the dominant, proprietary Microsoft Office formats (the familiar .doc, .xls, and .ppt files) and into the new world of open, accessible XML-based formats. The question is, which one?

On one side: ODF (Open Document Format), backed by a group of companies that includes IBM, Google, and Sun Microsystems (my employer), plus most of the “open-source” software community. ODF was approved as an International Standard (ISO 26300) in 2006.

On the other side: OOXML (Office Open XML), backed by Microsoft, its industry partners, and a vast army of Microsoft developers. OOXML was rushed into standardization in order to preserve Microsoft's historic domination of office productivity formats. It has been tentatively approved as ISO 29500 pending the resolution of appeals lodged by several national standards organizations.

The struggle to establish one or the other of these competing standards as the single format for office productivity software (a generic term that means Microsoft Office and its competitors, most notably the free open-source OpenOffice suite) has gone beyond the technical questions to raise larger issues ranging from European antitrust policy to the validity of the standards process to the ability of governments to provide universal data access for their citizens.

In the U.S., no fewer than seven states have introduced legislation seeking to define public policy in this area. The most notable activity has taken place in Massachusetts, which mandated “open standards” and found itself in the end supporting both ODF and OOXML as overlapping formats.

The state of New York has not been lagging in its own efforts to resolve the issue. A 2007 statute directed the NYS Chief Information Officer and Director of the Office For Technology to “study how electronic documents and the mechanisms and processes for obtaining access to and reading electronic data can be created, maintained, exchanged, and preserved by the state in a manner that encourages appropriate government control, access, choice, interoperability, and vendor neutrality” and to report findings back to the governor and the legislature. To carry out this mandate, the CIO/OFT solicited comments from the public and convened a workgroup consisting of representatives of several state agencies to review the input received and produce a report containing draft recommendations. That report, “A Strategy for Openness: Enhancing E-Records Access in New York State,” was released in mid-May.

While the workgroup failed to decide between “choice” (Microsoft's mantra) and “openness” (the ODF mantra), predictably punting this question to a new Electronic Records Committee, it did issue a number of interesting findings, the most important of which reads as follows:

In the office suite format debate, there currently is no compelling solution for the State’s openness needs. The State needs open standards and formats. Simultaneously, the State needs electronic records to be preserved in their original formats whenever possible. Many Request for Public Comments commenters, particularly in response to the e-discovery questions, stated preserving a record in the same format as it was created results in a more faithful record and diminishes the possibility of expensive e-discovery disputes. This is important to ensure future generations of New Yorkers can access the permanently valuable electronic records being created today. Moreover, State Archives emphasizes creating records in open formats makes it easier to preserve their essential characteristics and demonstrates they are authentic (i.e., they were created in the course of State government business and have not been altered without proper authorization).

I imagine that the workgroup must have found some level of solace in arriving at the one conclusion that all the experts seem to agree on: that electronic documents should be published using the same format in which they are created. If this principle held true for state documents, it would reduce the job of the new Electronic Records Committee to deciding between three alternatives: (1) require all state agencies to create and publish their documents in OOXML, (2) require all state agencies to create and publish their documents in ODF, or (3) allow each agency to decide which of these formats, OOXML or ODF, they will use in creating and publishing their documents. Unfortunately, this central assumption is incorrect, and adopting it as a basis for further deliberation will lead to a deeply defective state open records policy. As a New York State taxpayer, I find this prospect disturbing.

Leaving aside spreadsheets (like Microsoft Excel files) and presentations (like Microsoft PowerPoint files), which contain programmatic constructs that make them act something like software, and also excluding dynamically formatted hypertext web pages, which pose their own set of challenges, the workgroup's finding as applied to ordinary documents — the kind that would be produced using Microsoft Word, for example — is simply wrong. When it comes to state documents, using the same format for both document creation and document publication is a very bad idea. The reasons are somewhat technical, but it is the obligation of everyone involved in this decision to understand them.

The key distinction that needs to be grasped here is between editable formats (authoring formats) and final formats (publishing formats). This distinction has nothing to do with ODF vs. OOXML.

Editable document formats are designed to allow quick display and easy modification of content. ODF and OOXML are both editable document formats. Publishing formats, on the other hand, are designed to precisely express the location of text and graphics on the finished page. Adobe's PDF (Portable Document Format) is an example of a publishing format. Editable formats like ODF and OOXML are inappropriate for the publication of state documents because they cannot guarantee line and page fidelity.

Line fidelity is the ability of a format to preserve line breaks, and page fidelity is the ability of a format to preserve page breaks. Line and page fidelity are important for several reasons, most obviously in the publication of formal documents such as laws and contracts, which frequently make reference to line and page numbers.

Neither ODF nor OOXML guarantees line and page fidelity. It is impossible to know whether an ODF or OOXML document displayed or printed using one software product will exhibit the same line and page breaks as the identical document displayed or printed using a different software product. You can't get presentational uniformity even between a version of Microsoft Word for the PC and the same product running on a Macintosh, much less anything like identical treatment of the same ODF or OOXML file when processed by products from different manufacturers.

It is important to note that this limitation is not due to any defect in the products themselves; the inability of ODF and OOXML to guarantee line and page fidelity is a result of the way these formats are designed. The final layout of an ODF or OOXML document depends on details of the proprietary formatting algorithms for hyphenation, justification, and footnote placement that are unique to each particular program. By contrast, a publishing format like PDF does guarantee line and page fidelity but is almost impossible to edit interactively.

The practical result of these design considerations is that any transfer of an ODF or OOXML file between systems or applications may break any or all of the page or line references that might be contained in it, including not just explicit references to line numbers and page numbers in the text but also the page references in indexes and tables of contents. This reason alone eliminates ODF and OOXML from serious consideration as vehicles for the published form of state documents. But the problem extends further than the preservation of line and page references.

Anyone who has tried to create something as complicated as a four-page newsletter knows how sensitive the layout of such a piece can be, especially if it contains artwork. The smallest change — sometimes no more than the introduction of a single character — can create an extra line and push a large graphic element onto a new page, turning a four-page leaflet into a five-page mess. I invite anyone who doubts this to try a simple experiment: take any well-formatted and reasonably complex brochure or newsletter, select all the body type, and increase the font size by 0.5 points. The difference in any one paragraph will be nearly invisible, but it's a rare document of any size whose layout will not be radically altered by the change. The same is true when an ODF or OOXML file is rendered by a program different from the one that created it, or when a different font must be substituted for the original due to its absence from the particular computer on which the document is displayed.

In a publishing format like PDF, on the other hand, the location of individual elements is specified to a level that allows documents to be proofed electronically with confidence that nothing critical will change farther down the line. Many commercial print shops now have “direct-to-plate” equipment that creates printing plates directly from PDF files sent in from remote locations. There are even printing presses that allow PDF files to be sent directly to the press itself. Anyone who knows how much a printing error can cost in materials and time will readily appreciate the level of reliability and consistency required to proof a four-color catalogue on a computer and then send that file halfway across the country and directly to the press. One way to sum up the key limitations of editable formats like ODF and OOXML is that they are incapable of supporting this kind of workflow.

The belief that a document creator's intent is best preserved by using an editable format for final publication rests on the assumption that exactly the same software will be used to consume the document that was used to produce it. This may be true in a monopoly environment where everyone uses the same program running under the same operating system on the same hardware, but as we begin to open that environment to competition, this assumption no longer holds. If the idea is to preserve the original intent of the author in a heterogeneous environment, then publishing in an editable format is exactly the wrong way to do it.

Publishing state documents using ODF or OOXML has further drawbacks resulting simply from their ability to be edited. Note again the prospect raised by State Archives in the passage quoted above:

State Archives emphasizes creating records in open formats makes it easier to preserve their essential characteristics and demonstrates they are authentic (i.e., they were created in the course of State government business and have not been altered without proper authorization).

This is an entirely valid concern, but it is scarcely addressed by office formats like ODF and OOXML that are designed to make such alteration as easy as possible. PDF, on the other hand, is designed precisely to “preserve the essential characteristics” of a document. PDF also makes it significantly harder to pull out the graphic elements from an official document and use them to create fraudulent artifacts, as is so common in the “phishing” expeditions that infest our email.

A publishing format like PDF does not actually prevent ad hoc editing by a determined programmer — probably no text-based commercial format could do that — but at least it does not put large-scale free-form modification of state documents at the skill level of the average sixth grader.

The idea that document creators can go straight from the editable form to the final product has a certain rough appeal on the basis of efficiency, but on closer examination it becomes clear that the advantage of publishing the editable file not only assumes that everyone uses the same software and operating system but also that the documents are created outside of a formal workflow, like someone knocking out an internal memo or printing up a recipe for chicken divan. In publicly issued state work there are no ad hoc documents; every document issues from the state itself acting through one or more of its employees. To put it more familiarly, there is no state document that is not approved by someone and (presumably) proofread by someone, and that person bears as much responsibility for its final layout as for its final text. Once the piece is published, the last thing to be desired is its modification by members of the public. The only exceptions are documents like tax forms that are intended to be modified by filling out data for submission back to the state.

In workflow terms, the thing that gets reviewed by the person who is going to bear formal responsibility for its issuance has to be the PDF version of the document, because it is only at this stage that every aspect of the document is locked down into the form that will be released into the larger world. The proper application of ODF and OOXML is in the stages leading up to final approval. In a document workflow that recognizes these considerations, the document is created in ODF or OOXML, edited as necessary on a variety of ad hoc systems, and then when ready is rendered in PDF for final approval before release. All the current leading word processing programs make this final step as easy as pressing a button; but procedurally, it has to be a separate operation to preserve organizational accountability for the final product.

Some of the state initiatives to define an official publishing format exhibit a mistaken belief that since ODF and OOXML are based on the XML metalanguage, they must be more “open” and capable than a non-XML (but text-based) page description language like PDF. It is true that an XML format that was semantically rich enough could in theory support the addition of unlimited amounts of metadata that would greatly enhance our ability to search and navigate the document space. But in reality there are at least two factors that render this theoretical benefit irrelevant. First, enabling this functionality across all state documents would require a level of metadata standardization that lies decades in the future, if indeed it can ever be attained; second, and more to the point, the human work needed at the current stage of technology to add even the smallest amount of semantic enrichment to any meaningful number of documents is far beyond what any state agency has the resources to accomplish. And even if we were somehow able to overcome these enormous barriers, the fact is that neither ODF nor OOXML is natively equipped to support semantic enrichment any better than PDF is. In other words, in reality, a PDF file is just as searchable as an ODF file or an OOXML file. And of course, the PDF file specifies unambiguously what the piece was intended to look like, which is something that the editable formats cannot do.

If by “open” the proponents of XML-based formats simply mean that the contents of ODF and OOXML documents are easier to manipulate, they are correct — but as noted above, this capability is the opposite of what we want in electronic documents published by the state.

Given the very clear advantages of publishing state documents using a format designed for the purpose, it would be unfortunate if we faced the kind of standards battle over publishing formats that we have seen in the case of the editable formats. Luckily, the subset of PDF that would be appropriate for this use, PDF/A (the “A” stands for “archival”) is already an International Standard, ISO 19005-1:2005. The PDF/A standard was specifically designed to preserve documents over a long period of time in a way that allows them to be retrieved and rendered with a consistent and predictable result in the future. Generic PDF software that can reliably display any PDF/A document is already freely available and widely distributed.

There is currently no competition for PDF/A in the world of international standards — it is the only document format recognized by ISO for its intended purpose. And PDF itself is the de facto standard of the entire global printing industry, supported by fifteen years of commercial tool development. It is entirely possible that Microsoft will attempt to create competition for ISO PDF/A, just as it did by introducing OOXML as competition to the already existing ISO ODF format, but the enormous installed base of PDF tools and the greatly enhanced consciousness of the damage this kind of disruption would cause makes it unlikely that such an effort will succeed.

Which editable format to adopt for document creation remains an open question. I believe that there are strong reasons for standardizing on ODF as the document creation format across all state agencies, but this is an issue separate from which format to use for the electronic publication of state documents that are not intended to be filled out and sent back. For the publication of the ordinary run of state documents there is only one sensible choice — PDF/A. As a New York State resident, I call upon the Legislature to recognize this technical reality before mandating a broken policy that we will have to live with long into the future.


Sun Microsystems Distinguished Engineer Jon Bosak organized and led the working group that created the XML specification. He is a long-time member of ISO/IEC JTC1/SC34, the body responsible for the international standardization of both ODF and OOXML. His views do not necessarily represent those of his employer.