<!DOCTYPE pldoc SYSTEM "pldoc.dtd">

<pldoc>
<head>
<version>1.0.14
<release-date>March 2001
<package>sgml2pl
</head>

<titlepage>
<title>SWI-Prolog SGML/XML parser</title>
<author>
  <name>Jan Wielemaker</name>
  <address>
    SWI, <br>
    University of Amsterdam <br>
    The Netherlands
  </address>
  <email>jan@swi.psy.uva.nl</email>
</author>

<abstract>
Markup languages are an increasingly important method for 
data-representation and exchange. This article documents the package
<ref pkg/sgml2pl/, a foreign library for SWI-Prolog to parse SGML and XML 
documents, returning information on both the document and the document's
DTD. The parser is designed to be small, fast and flexible. 
</abstract>
</titlepage>

<s1>Introduction</title>

<p>
Markup languages have recently regained popularity for two reasons. One 
is document exchange, which is largely based on HTML, an instance of 
SGML, and the other is for data exchange between programs, which is often
based on XML, which can be considered a simplified and rationalised 
version of SGML.

<p>
James Clark's SP parser is a flexible SGML and XML parser. Unfortunately
it has some drawbacks. It is very big, not very fast, cannot work under 
event-driven input and is generally hard to program beyond the scope of 
the well designed generic interface. The generic interface however does 
not provide access to the DTD, does not allow for flexible handling of 
input or parsing the DTD independently of a document instance.

<p>
The parser described in this document is small (less than 50 kbytes
executable on a Pentium or 80 kbytes on a SPARC), fast (between 2 and 5
times faster than SP), provides access to the DTD, and provides flexible
input handling.

<p>
The document output is equal to the output produced by <jargon/xml2pl/, 
an SP interface to SWI-Prolog written by Anjo Anjewierden.


<s1>Bluffer's Guide</title>

<p>
This package allows you to parse SGML, XML and HTML data into a Prolog 
data structure. The high-level interface defined in <ref lib/sgml/ 
provides access at the file-level, while the low-level interface defined
in the foreign module works with Prolog streams. Please use the source 
of <ref file/sgml.pl/ as a starting point for dealing with data from 
other sources than files, such as SWI-Prolog resources, network-sockets,
character strings, <em>etc.</em> The first example below loads an HTML file.

<code>
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">

<html>
<head>
<title>Demo</title>
</head>
<body>

<h1 align=center>This is a demo</title>

<p>Paragraphs in HTML need not be closed.

<p>This is called `omitted-tag' handling.
</body>
</html>
</code>

<code>
?- load_html_file('test.html', Term),
   pretty_print(Term).

[ element(html,
          [],
          [ element(head,
                    [],
                    [ element(title,
                              [],
                              [ 'Demo'
                              ])
                    ]),
            element(body,
                    [],
                    [ '\n',
                      element(h1,
                              [ align = center
                              ],
                              [ 'This is a demo'
                              ]),
                      '\n\n',
                      element(p,
                              [],
                              [ 'Paragraphs in HTML need not be closed.\n'
                              ]),
                      element(p,
                              [],
                              [ 'This is called `omitted-tag\' handling.'
                              ])
                    ])
          ])
].
</code>

<p>
The document is represented as a list, each element being an atom to 
represent <c/CDATA/ or a term <term>element(Name, Attributes, Content)</term>.
Entities (e.g. <verb>&lt;</verb>) are returned as part of <c/CDATA/, 
unless they cannot be represented. See <pref>load_sgml_file/2</pref> for
details.

<s2>`Goodies' Predicates</title>

<p>
These predicates are for basic use of the library, converting entire and
self-contained files in SGML, HTML, or XML into a structured term. They
are based on <pref>load_structure/3</pref>.

<definitions>
    <pred>load_sgml_file(+File, -ListOfContent)
    <summary>Parse SGML file into Prolog term 
Same as <term>load_structure(File, ListOfContent, [dialect(sgml)])</term>.
    <pred>load_xml_file(+File, -ListOfContent)
    <summary>Parse XML file into Prolog term
Same as <term>load_structure(File, ListOfContent, [dialect(xml)])</term>.
    <pred>load_html_file(+File, -Content)
    <summary>Parse HTML file into Prolog term
Load <aref/File/ and parse as HTML. Implemented as:

<code>
load_html_file(File, Term) :-
        dtd(html, DTD),
        load_structure(File, Term,
                       [ dtd(DTD),
                         dialect(sgml)
                       ]).
</code>
</definitions>


<s1>Predicate Reference</title>

<s2>Loading Structured Documents</title>

<p>
SGML or XML files are loaded through the common predicate
<pref>load_structure/3</pref>. This is a predicate with many options. For 
simplicity a number of commonly used shorthands are provided: <pref>
load_sgml_file/2</pref>, <pref>load_xml_file/2</pref>, and
<pref>load_html_file/2</pref>.

<definitions>
    <pred>load_structure(+File, -ListOfContent, +Options)
    <summary>Parse XML/SGML/HTML file into Prolog term
Load the XML file <aref/File/ and return the resulting structure in <aref/
ListOfContent/. <aref/Options/ is a list of options controlling the 
conversion process.

<p>
A proper XML document contains only a single toplevel element whose name
matches the document type. Nevertheless, a list is returned for 
consistency with the representation of element content. The <aref/
ListOfContent/ consists of the following types:
<argdef>
    <termitem><var/Atom/ Atoms are used to represent <c/CDATA/. Note 
this is possible in SWI-Prolog, as there is no length-limit on atoms and
atom garbage collection is provided.
    <termitem>element(Name, ListAttributes, ListOfContent)
<aref/Name/ is the name of the element. Using SGML, which is 
case-insensitive, all element names are returned as lowercase atoms.

<p>
<aref/ListOfAttributes/ is a list of <aref/Name/=<aref/Value/ pairs for
attributes that appeared in the source. No information is returned on
<!--****OUCH****.  So that's what was going wrong.
    How hard would it be to change this rather devastating limitation?
    The essence of attribute processing is that an application should
    have no reason to care, let alone know, which attribute values
    came from what source.  The HyTime "SGML Extended Facilities", which are
    supposed to become part of the SGML standard, even provide a way
    for a document to provide its own default values (<dvlist>).
-->other attributes, such as <c/fixed/ or <c/default/ attributes. See
<pref>dtd_property/2</pref> for accessing the DTD for this information.
Attributes of type <c/CDATA/ are returned literal. Multi-valued
attributes (<c/NAMES/, <em>etc.</em>) are returned as a list of atoms.
Handling attributes of the types <c/NUMBER/ and <c/NUMBERS/ depends on
the setting of the <term>number(+NumberMode)</term> attribute through
<pref>set_sgml_parser/2</pref> or <pref>load_structure/3</pref>.  By
default they are returned as atoms, but automatic conversion to Prolog
integers is supported. <aref/ListOfContent/ defines the content for the
element.
    <termitem>entity(Code)
If a character-entity (e.g. <verb>&#913;</verb>) is encoutered that 
cannot be represented in the Prolog character set, this term is 
returned, representing the referred character code.
    <termitem>entity(Name)
If an entity refers to a character-entity holding a single character, 
but this character cannot be represented in the Prolog character set, 
this term is returned. For example, the HTML input text
<verb>&Alpha; &lt; &Beta;</verb>
is returned as below. Please note that entity names are case sensitive
in both SGML and XML.

<code>
[ entity('Alpha'), ' < ', entity('Beta') ]
</code>

<p>
This is a special case of <term>entity(Code)</term>, intended to handle 
special symbols by their name rather than character code.
    <termitem>sdata(Text)
If an entity with declared content-type <c/SDATA/ is encountered, this 
term is returned holding the data in <aref/Text/.
    <termitem>ndata(Text)
If an entity with declared content-type <c/NDATA/ is encountered, this 
term is returned holding the data in <aref/Text/.
    <termitem>pi(Text)
If a processing instruction is encountered (<verb><?...?></verb>), <aref/
Text/ holds the text of the processing instruction. Please note that the
<verb><?xml ...?></verb> instruction is handled internally. 
<!--This interface does not match what the XML spec requires nor
    what the Infoset calls for.  Given <?target data...data?>, XML
    expects something like pi('target','data...data').
    XML also requires (DOM, Infoset) that comments be provided
    if you want them (and even if you don't, *sigh*).
    comment(Text), provided if comment(yes) appeared in the Options
    would be good.  Similarly, cdata(Text), for the content of
    CDATA marked sections, provided if cdata(yes) appear in the Options
    would seem to be required by XML InfoSet.
--></argdef>


<p>
The <aref/Options/ list controls the conversion process. Currently 
defined options are:

<argdef>
    <termitem>dtd(?DTD)
Reference to a DTD object. If specified, the <verb><!DOCTYPE ...></verb>
declaration is ignored and the document is parsed and validated against 
the provided DTD. If provided as a variable, the created DTD is
returned. See <ref sec/implicitdtd/.
    <termitem>dialect(+Dialect)
Specify the parsing dialect. Supported are <c/sgml/ (default), <c/xml/ 
and <c/xmlns/. See <ref sec/xml/ for details on the differences.
    <termitem>shorttag(+Bool)
Define whether SHORTTAG abbreviation is accepted.  The default is true
for SGML mode and false for the XML modes.  Without SHORTTAG, a <c>/</c>
is accepted with warning as part of an unquoted attribute-value, though
<c>/></c> still closes the element-tag in XML mode.  It may be set to
false for parsing HTML documents to allow for unquoted URLs containing
<c>/</c>.
    <termitem>space(+SpaceMode)
Sets the `space-handling-mode' for the initial environment. This mode is
inherited by the other environments, which can override the inherited
value using the XML reserved attribute <elem/xml:space/. See <ref sec/space/.
    <termitem>number(+NumberMode)
Determines how attributes of type <c/NUMBER/ and <c/NUMBERS/ are handled.
If <c/token/ (default) they are passed as an atom.  If <c/integer/ the
parser attempts to convert the value to an integer.  If successful, the
attribute is passed as a Prolog integer.  Otherwise it is still passed
as an atom.  Note that SGML defines a numeric attribute to be a sequence
of digits.  The <c/-/ sign is not allowed and <xmp/1/ is different from
<xmp/01/.  For this reason the default is to handle numeric attributes as
tokens.  If conversion to integer is enabled, negative values are silently
accepted.
    <termitem>defaults(+Bool)
Determines how default and fixed values from the DTD are used.  By
default, defaults are included in the output if they do not appear in
the source.  If <c/false/, only the attributes occurring in the source
are emitted.
    <termitem>file(+Name)
Sets the name of the file on which errors are reported. Sets the 
linenumber to 1.
    <termitem>line(+Line)
Sets the starting line-number for reporting errors.
    <termitem>max_errors(+Max)
Sets the maximum number of errors. If this number is reached, an 
exception of the format below is raised. The default is 50.

<quote>
<term>error(limit_exceeded(max_errors, Max), _)</term> 
</quote>
</argdef>
</definitions>

<s2 name="space">Handling white-space</title>

<p>
SGPL2PL has four modes for handling white-space. The initial mode can be
switched using the <term>space(SpaceMode)</term> option to
<pref>load_structure/3</pref> and <pref>set_sgml_parser/2</pref>. In XML
mode, the mode is further controlled by the <elem/xml:space/ attribute,
which may be specified both in the DTD and in the document. The defined
modes are:

<valdef>
    <termitem>space(sgml)
In SGML, newlines at the start and end of an element are removed.<fn>In
addition, newlines at the end of lines containing only markup should be
deleted. This is not yet implemented.</fn> This is the default mode for 
the SGML dialect. 

    <termitem>space(preserve)
White space is passed literally to the application. This mode leaves all
white space handling to the application. This is the default mode for
the XML dialect. 

    <termitem>space(default)
In addition to <c/sgml/ space-mode, all consequtive white-space is 
reduced to a single space-character. This mode canonises all white 
space. 

    <termitem>space(remove)
In addition to <c/default/, all leading and trailing white-space is 
removed from <c/CDATA/ objects. If, as a result, the <c/CDATA/ 
becomes empty, nothing is passed to the application. This mode is 
especially handy for processing `data-oriented' documents, such as RDF. 
It is not suitable for normal text documents. Consider the HTML 
fragment below. When processed in this mode, the spaces between the 
three modified words are lost.  This mode is not part of any standard;
XML 1.0 allows only <c/default/ and <c/preserve/.

<code>
Consider adjacent <b>bold</b> <ul>and</ul> <it>italic</it> words.
</code>
</valdef>

<s2 name="xml">XML documents</title>

<p>
The parser can operate in two modes: <c/sgml/ mode and <c/xml/ mode, as 
defined by the <term>dialect(Dialect)</term> option. Regardless of this 
option, if the first line of the document reads as below, the parser is 
switched automatically into XML mode.

<code>
<?xml ... ?>
</code>

<p>
Currently switching to XML mode implies:

<ul>
   <tag>XML empty elements
The construct <verb><element [attribute...] /></verb> is recognised as 
an empty element. 
   <tag>Predefined entities
The following entitities are predefined: <c/lt/ (<verb><</verb>), <c/gt/
(<verb>></verb>), <c/amp/ (<verb>&</verb>), <c/apos/ (<verb>'</verb>) 
and <c/quot/ (<verb>"</verb>). 
   <tag>Case sensitivity
In XML mode, names are treated case-sensitive, except for the DTD 
reserved names (i.e. <xmp/ELEMENT/, <em>etc.</em>). 
   <tag>Character classes
In XML mode, underscores (<verb>_</verb>) and colon (<verb>:</verb>) are
allowed in names. 
   <tag>White-space handling
White space mode is set to <c/preserve/. In addition to setting
white-space handling at the toplevel the XML reserved attribute
<elem/xml:space/ is honoured. It may appear both in the document and the
DTD. The <c/remove/ extension is honoured as <elem/xml:space/ value. For
example, the DTD statement below ensures that the <elem/pre/ element
preserves space, regardless of the default processing mode.

<code>
<!ATTLIST pre xml:space nmtoken #fixed preserve>
</code>
</ul>


<s3 name="xmlns">XML Namespaces</title>

<p>
Using the <jargon/dialect/ <c/xmlns/, the parser will interpret XML 
namespaces. In this case, the names of elements are returned as a term 
of the format
<!--I am a bit puzzled here.  It is clear from the XML InfoSet and
    from the Document Object Model and also, I thought, quite clear
    from XML Namespaces itself, that under namespaces, the name
    of an element is a TRIPLE:  qualified name, local name, uri
    An element always retains the name it was originally written
    with, not least because that is exactly what the <!ELEMENT>
    declarations constrain, and under Namespace processing acquires
    two ADDITIONAL pieces of information: the local name and the uri
    The same goes for attribute names; I must say that the approach
    taken in sgml2pl seems excessively complicated.
    Why not take a step forward and use 
      element(Name,Addl,Atts,Content) 
      attribute(Name,Addl,Value) 
    where Addl is an abstract data type, leaving you room to plug 
    source location in there later (which would be useful in 'sgml'
    and 'xml' without namespaces modes) plus other stuff like 
    namespace, xbase, and so on.  You'd need a predicate 
      xml_namespace(Addl, LocalName, URI) 
    to get at the information, but it would work.
-->

<quote>
<aref/URL/<c/:/<aref/LocalName/ 
</quote>

<p>
If an identifier has no namespace and there is no default namespace it 
is returned as a simple atom. If an identifier has a namespace but this 
namespace is undeclared, the namespace name rather than the related URL 
is returned.

<p>
Attributes declaring namespaces (<xmp>xmlns:<aref/ns/=<aref/url/</xmp>)
are reported as if <c/xmlns/ were not a defined resource.
<!--Unless I've misunderstood, xml*:* attributes are not supposed
    to be returned with other attributes.
-->
<p>
In many cases, getting attribute-names as <xmp><aref/url/:<aref/name/</xmp>
is not desirable. Such terms are hard to unify and sometimes multiple 
URLs may be mapped to the same identifier. This may happen due to poor 
version management, poor standardisation or because the the application 
doesn't care too much about versions. This package defines two 
call-backs that can be set using <pref>set_sgml_parser/2</pref> to deal 
with this problem.

<p>
The call-back <c/xmlns/ is called as XML namespaces are noticed.
It can be used to extend a canonical mapping for later use
by the <c/urlns/ call-back. The following illustrates this behaviour.
Any namespace containing <c/rdf-syntax/ in its URL or that is used as
<c/rdf/ namespace is canonised to <c/rdf/. This implies that any
attribute and element name from the RDF namespace appears as
<xmp>rdf:<aref/name/</xmp>.

<code>
:- dynamic
        xmlns/3.

on_xmlns(rdf, URL, _Parser) :- !,
        asserta(xmlns(URL, rdf, _)).
on_xmlns(_, URL, _Parser) :-
        sub_atom(URL, _, _, _, 'rdf-syntax'), !,
        asserta(xmlns(URL, rdf, _)).

load_rdf_xml(File, Term) :-
        load_structure(File, Term,
                       [ dialect(xmlns),
                         call(xmlns, on_xmlns),
                         call(urlns, xmlns)
                       ]).
</code>


<s2>DTD-Handling</title>

<p>
The DTD (<b>D</b>ocument <b>T</b>ype <b>D</b>efinition) is a separate 
entity in sgml2pl, that can be created, freed, defined and inspected. 
Like the parser itself, it is filled by opening it as a Prolog output
stream and sending data to it. This section summarises the predicates 
for handling the DTD.
<definitions>
    <pred>new_dtd(+DocType, -DTD)
    <summary>Create a DTD object
Creates an empty DTD for the named <aref/DocType/. The returned 
DTD-reference is an opaque term that can be used in the other predicates
of this package.

    <pred>free_dtd(+DTD)
    <summary>Free a DTD object
Deallocate all resources associated to the DTD. Further use of <aref/DTD/
is invalid.

    <pred>load_dtd(+DTD, +File)
    <summary>Read DTD information from a file
Define the DTD by loading the SGML-DTD file <aref/File/. This predicate 
is defined using the low-level <pref>open_dtd/3</pref> predicate:

<code>
load_dtd(DTD, DtdFile) :-
        open_dtd(DTD, [], DtdOut),
        open(DtdFile, read, DtdIn),
        copy_stream_data(DtdIn, DtdOut),
        close(DtdIn),
        close(DtdOut).
</code>

    <pred>open_dtd(+DTD, +Options, -OutStream)
    <summary>Open a DTD object as an output stream
Open a DTD as an output stream. The option-list is currently 
empty. See <pref>load_dtd/2</pref> for an example.

    <pred>dtd(+DocType, -DTD)
    <summary>Find or build a DTD for a document type
Find the DTD representing the indicated <jargon/doctype/. This predicate
uses a cache of DTD objects. If a doctype has no associated dtd, it 
searches for a file using the file search path <xmp/dtd/ using the call:

<code>
...,
absolute_file_name(dtd(Type),
                   [ extensions([dtd]),
                     access(read)
                   ], DtdFile),
...
</code>

    <pred>dtd_property(+DTD, ?Property)
    <summary>Query elements, entities and attributes in a DTD
This predicate is used to examine the content of a DTD. Property is one 
of:
<argdef>
    <termitem>doctype(DocType)
An atom representing the document-type defined by this DTD. 
    <termitem>elements(ListOfElements)
A list of atoms representing the names of the elements in this DTD. 
    <termitem>element(Name, Omit, Content)
The DTD contains an element with the given name. <aref/Omit/ is a term 
of the format <term>omit(OmitOpen, OmitClose)</term>, where both 
arguments are booleans (<c/true/ or <c/false/ representing whether the 
open- or close-tag may be omitted. <aref/Content/ is the content-model 
of the element represented as a Prolog term. This term takes the 
following form:
<!--You say nothing about how exceptions +(...) -(...) are
    represented.  Exceptions are used in HTML, e.g. 
     <!ELEMENT A - - (%inline)* -(A)> 
    so what DOES sgml2pl do with them? 
--><argdef>
    <termitem><c/empty/
The element has no content. 

    <termitem><c/cdata/
The element contains non-parsed character data.  All data up to the
matching end-tag is included in the data (<jargon/declared content/).

    <termitem><c/rcdata/
As <c/cdata/, but entity-references are expanded.

    <termitem><c/any/
The element may contain any number of any element from the DTD in
any order.

    <termitem><c/#pcdata/
The element contains parsed character data .

    <termitem><var/element/ An element with this name. 

    <termitem>*(SubModel)
0 or more appearances. 

    <termitem>?(SubModel)
0 or one appearance. 

    <termitem>+(SubModel)
1 or more appearances. 

    <termitem>,(SubModel1, SubModel2)
<aref/SubModel1/ followed by <aref/SubModel2/. 

    <termitem>&amp;(SubModel1, SubModel2)
<aref/SubModel1/ and <aref/SubModel2/ in any order. 

    <termitem>|(SubModel1, SubModel2)
<aref/SubModel1/ or <aref/SubModel2/. 
</argdef>

    <termitem>attributes(Element, ListOfAttributes)
<aref/ListOfAttributes/ is a list of atoms representing the attributes 
of the element <aref/Element/. 

    <termitem>attribute(Element, Attribute, Type, Default)
Query an element. <aref/Type/ is one of <c/cdata/, <c/entity/, <c/id/,
<c/idref/, <c/name/, <c/nmtoken/, <c/notation/, <c/number/ or
<c/nutoken/. For DTD types that allow for a list, the notation
<term>list(Type)</term> is used. Finally, the DTD construct
<verb>(a|b|...)</verb> is mapped to the term
<term>nameof(ListOfValues)</term>. 

<p>
<aref/Default/ describes the sgml default. It is one <c/required/,
<c/current/, <c/conref/ or <c/implied/. If a real default is present, it
is one of <term>default(Value)</term> or <term>fixed(Value)</term>. 

    <termitem>entities(ListOfEntities)
<aref/ListOfEntities/ is a list of atoms representing the names of the 
defined entities. 
    <termitem>entity(Name, Value)
<aref/Name/ is the name of an entity with given value. Value is one of 
<argdef>
    <termitem><var/Atom/ If the value is atomic, it represents the 
literal value of the entity. 
    <termitem>system(Url)
<aref/Url/ is the URL of the system external entity. 
    <termitem>public(Id, Url)
For external public entities, <aref/Id/ is the identifier. If an URL is 
provided this is returned in <aref/Url/. Otherwise this argument is 
unbound. 
</argdef>
    <termitem>notations(ListOfNotations)
Returns a list holding the names of all <c/NOTATION/ declarations. 
    <termitem>notation(Name, Decl)
Unify <aref/Decl/ with a list if <term>system(+File)</term> and/or
<term>public(+PublicId)</term>.
</argdef>
</definitions>

<s3>The DOCTYPE declaration</title>

<p>
As this parser allows for processing partial documents and process the 
DTD separately, the DOCTYPE declaration plays a special role.

<p>
If a document has no DOCTYPE declaraction, the parser returns a list 
holding all elements and CDATA found. If the document has a DOCTYPE 
declaraction, the parser will open the element defined in the DOCTYPE as
soon as the first real data is encountered.

<s2 name="implicitdtd">Extracting a DTD</title>

<p>
<!--There's a PhD thesis from Helsinki about inferring a DTD.
    There's also a recent paper about inferring XML DTDs.
-->Some documents have no DTD. One of the neat facilities of this library 
is that it builds a DTD while parsing a document with an <jargon/
implicit/ DTD. The resulting DTD contains all elements encountered in 
the document. For each element the content model is a disjunction of 
elements and possibly <verb>#PCDATA</verb> that can be repeated. Thus, 
if we found element <elem/y/ and CDATA in element <elem/x/, the model 
is:

<code>
<!ELEMENT x - - (y|#PCDATA)*>
</code>

<p>
Any encountered attribute is added to the attribute list with the type
<c/CDATA/ and default <c/#IMPLIED/.

<p>
The example below extracts the elements used in an unknown XML document.

<code>
elements_in_xml_document(File, Elements) :-
        load_structure(File, _,
                       [ dialect(xml),
                         dtd(DTD)
                       ]),
        dtd_property(DTD, elements(Elements)),
        free_dtd(DTD).
</code>


<s2>Parsing Primitives</title>

<definitions>
    <pred>new_sgml_parser(-Parser, +Options)
    <summary>Create a new parser
Creates a new parser. A parser can be used one or multiple times for 
parsing documents or parts thereof. It may be bound to a DTD or the DTD 
may be left implicit, in which case it is created from the document 
prologue or parsing is performed without a DTD. Options:
<argdef>
    <termitem>dtd(?DTD)
If specified with an initialised DTD, this DTD is used for parsing the 
document, regardless of the document prologue. If specified using as a 
variable, a reference to the created DTD is returned. This DTD may be 
created from the document prologue or build implicitely from the 
document's content. 
</argdef>

    <pred>free_sgml_parser(+Parser)
    <summary>Destroy a parser
Destroy all resources related to the parser. This does not destroy the 
DTD if the parser was created using the <term>dtd(DTD)</term> option.

    <pred>set_sgml_parser(+Parser, +Option)
    <summary>Set parser options (dialect, source, <em>etc.</em>)
Sets attributes to the parser. Currently defined attributes: 

<argdef>
    <termitem>file(File)
Sets the file for reporting errors and warnings. Sets the line to 1. 
    <termitem>line(Line)
Sets the current line. Useful if the stream is not at the start of the 
(file) object for generating proper line-numbers. 
    <termitem>charpos(Offset)
Sets the current character location.  See also the <term>file(File)</term>
option.
    <termitem>dialect(Dialect)
Set the markup dialect. Known dialects: 
<argdef>
    <termitem><c/sgml/
The default dialect is to process as SGML. This implies markup is 
case-insensitive and standard SGML abbreviation is allowed (abreviated 
attributes and omitted tags). 
    <termitem><c/xml/
This dialect is selected automatically if the processing instruction
<verb><?xml ...></verb> is encountered. See <ref sec/xml/ for details. 
    <termitem><c/xmlns/
Process file as XML file with namespace support. See <ref sec/xmlns/ for
details. 
</argdef>
    <termitem>space(SpaceMode)
Define the initial handling of white-space in PCDATA.  This attribute is
described in <ref sec/space/.
    <termitem>number(NumberMode)
If <c/token/ (default), attributes of type number are passed as a Prolog atom.
If <c/integer/, such attributes are translated into Prolog integers.  If
the conversion fails (e.g. due to overflow) a warning is issued and the
value is passed as an atom.
    <termitem>doctype(Element)
Defines the toplevel element expected. If a <verb><!DOCTYPE</verb>
declaration has been parsed, the default is the defined doctype. The
parser can be instructed to accept the first element encountered as the
toplevel using <xmp>doctype(_)</xmp>. This feature is especially
useful when parsing part of a document (see the <c/parse/ option to
<pref>sgml_parse/2</pref>.
</argdef>

    <pred>get_sgml_parser(+Parser, -Option)
    <summary>Get parser options
Retrieve infomation on the current status of the parser. Notably useful 
if the parser is used in the call-back mode. Currently defined options: 

<argdef>
    <termitem>file(-File)
Current file-name. Note that this may be different from the provided 
file if an external entity is being loaded. 

    <termitem>charpos(-CharPos)
Offset from where the parser started its processing in the file-object. 
See <ref sec/indexaccess/. 

    <termitem>charpos(-Start, -End)
Character offsets of the start and end of the source processed causing the
current call-back.  Used in <ref exe/PceEmacs/ to for colouring text in
SGML and XML modes.

    <termitem>source(-Stream)
Prolog stream being processed. May be used in the <c/on_begin/, <em>etc.</em> 
callbacks from <pref>sgml_parse/2</pref>. 

    <termitem>dialect(-Dialect)
Return the current dialect used by the parser (<c/sgml/, <c/xml/ or <c/xmlns/).

    <termitem>event_class(-Class)
The <jargon/event class/ can be requested in call-back events. It
denotes the cause of the event, providing useful information for syntax
highlighting.  Defined values are:
    <argdef>
    <termitem><c/explicit/
        The code generating this event is explicitely present in the
        document.
    <termitem><c/omitted/
	The current event is caused by the insertion of an omitted tag.
	This may be a normal event in SGML mode or an error in XML mode.
    <termitem><c/shorttag/
	The current event (<c/begin/ or <c/end/) is caused by an
	element written down using the <jargon/shorttag/ notation
	(<verb><tag/value/></verb>.
    <termitem><c/shortref/
	The current event is caused by the expansion of a
	<jargon/shortref/. This allows for highlighting shortref strings
	in the source-text.
    </argdef>

    <termitem>doctype(-Element)
Return the defined document-type (= toplevel element).  See also
<pref>set_sgml_parser/2</pref>.

    <termitem>dtd(-DTD)
Return the currently used DTD.  See dtd_property/2 for obtaining information
on the DTD such as element and attribute properties.

    <termitem>context(-StackOfElements)
Returns the stack of currently open elements as a list. The head of this
list is the current element. This can be used to determine the context
of, for example, CDATA events in call-back mode. The elements
are passed as atoms.  Currently no access to the attributes is provided.

    <termitem>allowed(-Elements)
Determines which elements may be inserted at the current location.  This
information is returned as a list of element-names. If character data is
allowed in the current location, <c/#pcdata/ is part of <aref/Elements/.
If no element is open, the <jargon/doctype/ is returned.

<p>
This option is intended to support syntax-sensitive editors.  Such an
editor should load the DTD, find an appropriate starting point and then
feed all data between the starting point and the caret into the parser.
Next it can use this option to determine the elements allowed at this
point.  Below is a code fragment illustrating this use given a parser
with loaded DTD, an input stream and a start-location.

<code>
	...,
	seek(In, Start, bof, _),
	set_sgml_parser(Parser, charpos(Start)),
	set_sgml_parser(Parser, doctype(_)),
	Len is Caret - Start,
	sgml_parse(Parser,
		   [ goal(feed(In, Len)),
		     parse(input)	% do not complete document
		   ]),
        get_sgml_parser(Parser, allowed(Allowed)),
	...

feed(In, Len, Parser) :-
	copy_stream_data(In, Parser, Len).
</code>
</argdef>

<p>
    <pred>sgml_parse(+Parser, +Options)
    <summary>Parse the input
Parse an XML file. The parser can operate in two input and two output
modes. Output is either a structured term as described with
<pref>load_structure/2</pref> or call-backs on predefined events. The
first is especially suitable for manipulating not-too-large documents,
while the latter provides a primitive means for handling very large
documents.

<p>
Input is either a stream or an goal that pushes characters into the 
parser. A full description of the option-list is below.

<argdef>
    <termitem>document(+Term)
A variable that will be unified with a list describing the content of 
the document (see <pref>load_structure/2</pref>). 
    <termitem>source(+Stream)
An input stream that is read. Either this option or the <term>goal(Goal)</term>
option must be provided. 
    <termitem>goal(+Goal)
<aref/Goal/ is a callable term. The predicate <pref>sgml_parse/2</pref> 
opens an output stream to the parser and invokes
<term>call(Goal, Stream)</term>, where <aref/Goal/ should write the data
to be parsed to <aref/Stream/. This option is not compatible with
<term>parse(element)</term>. This option can be used, for example, to
parse a Prolog atom:

<code>
parse_atom(Atom, Term) :-
        new_sgml_parser(Parser, []),
        sgml_parse(Parser,
                   [ document(Term),
                     goal(provide_atom(Atom))
                   ]),
        free_sgml_parser(Parser).

provide_atom(Atom, ParserStream) :-
        write(ParserStream, Atom).
</code>

<p>
For example:

<code>
?- parse_atom('<h1>hello world</h1>', X).

X = [element(h1, [], ['hello world'])] 
</code>

    <termitem>parse(Unit)
Defines how much of the input is parsed.  This option is used to parse
only parts of a file.
    <argdef>
	<termitem><c/file/
Default.  Parse everything upto the end of the input.
	<termitem><c/element/
The parser stops after reading the first element. Using <term>
source(Stream)</term>, this implies reading is stopped as soon as the
element is complete, and another call may be issued on the same stream
to read the next element. Using <term>goal(Goal)</term> as input, the
stream reports an I/O error after completing the first element. This
exception destroyes the built <term>document(Term)</term>, making this
option useless using `Goal' driven input.
	<termitem><c/content/
The value <c/content/ is like <c/element/ but assumes the element has
already been opened. It may be used in a call-back from
<term>call(<c/on_begin/, Pred)</term> to parse individual elements after
validating their headers.
	<termitem><c/declaration/
This may be used to stop the parser after reading the first
declaration.  This is especially useful to parse only the <xmp/doctype/
declaration.
	<termitem><c/input/
This option is intended to be used in conjunction with the
<term>allowed(Elements)</term> option of <pref>get_sgml_parser/2</pref>.
It disables the parser's default to complete the parse-tree by closing
all open elements.
    </argdef>

    <termitem>max_errors(+MaxErrors)
Set the maximum number of errors. If this number is exceeded further 
writes to the stream will yield an I/O error exception. Printing of 
errors is suppressed after reaching this value. The default is 100. 
    <termitem>syntax_errors(+ErrorMode)
Defines how syntax errors are handled.
    <argdef>
	<termitem>print
	Default.  Pass messages to <pref builtin>print_message/2</pref>.
        <termitem>quiet
	Suppress all messages.
    </argdef>
    <termitem>call(+Event, :PredicateName)
Issue call-backs on the specified events. <aref/PredicateName/ is the 
name of the predicate to call on this event, possibly prefixed with a 
module identifier. The defined events are:
<argdef>
    <termitem>begin
An open-tag has been parsed. The named handler is called with three 
arguments: <term><aref/Handler/(+Tag, +Attributes, +Parser)</term>. 
    <termitem>end
A close-tag has been parsed. The named handler is called with two 
arguments: <term><aref/Handler/(+Tag, +Parser)</term>. 
<!--Why on earth are the attributes not parsed here?
    I know nsgmls doesn't include them, and a major nuisance I've found it.
-->    <termitem>cdata
CDATA has been parsed. The named handler is called with two arguments: <term><aref/
Handler/(+CDATA, +Parser)</term>, where CDATA is an atom representing 
the data. 
    <termitem>entity
An entity that cannot be represented as CDATA has been parsed. The named
handler is called with two arguments: <term><aref/Handler/(+NameOrCode, 
+Parser)</term>. 
    <termitem>pi
A processing instruction has been parsed. The named handler is called 
with two arguments: <term><aref/Handler/(+Text, +Parser)</term>, where
<aref/Text/ is the text of the processing instruction. 
    <termitem>decl
A declaration (<verb><!...></verb>) has been read. The named handler is
called with two arguments: <term><aref/Handler/(+Text, +Parser)</term>,
where <aref/Text/ is the text of the declaration with comments removed.

<p>
This option is expecially useful for highlighting declarations and comments in
editor support, where the location of the declaration is extracted using
<pref>get_sgml_parser/2</pref>.
    <termitem>error
An error has been encountered.  the named handler is called with three
arguments: <term><aref/Handler/(+Severity, +Message, +Parser)</term>, where
<aref/Severity/ is one of <c/warning/ or <c/error/ and <aref/Message/ is
an atom representing the diagnostic message.  The location of the error
can be determined using <pref>get_sgml_parser/2</pref>

<p>
If this option is present, errors and warnings are not reported using
<pref builtin>print_message/3</pref>
    <termitem>xmlns
When parsing an in <c/xmlns/ mode, a new namespace declaraction is 
pushed on the environment. The named handler is called with three 
arguments: <term><aref/Handler/(+NameSpace, +URL, +Parser)</term>.
See <ref sec/xmlns/ for details. 
    <termitem>urlns
When parsing an in <c/xmlns/ mode, this predicate can be used to map a 
url into either a canonical URL for this namespace or another internal 
identifier. See <ref sec/xmlns/ for details. 
</argdef>
</argdef>
</definitions>


<s3>Partial Parsing</title>

<p>
In some cases, part of a document needs to be parsed. One option is to 
use <pref>load_structure/2</pref> or one of its variations and extract 
the desired elements from the returned structure. This is a clean 
solution, especially on small and medium-sized documents. It however is 
unsuitable for parsing really big documents. Such documents can only be 
handled with the call-back output interface realised by the
<term>call(Event, Action)</term> option of <pref>sgml_parse/2</pref>.
Event-driven processing is not very natural in Prolog.

<p>
The SGML2PL library allows for a mixed approach. Consider the case where
we want to process all descriptions from RDF elements in a document. The
code below calls <xmp>process_rdf_description(Element)</xmp> on each element 
that is directly inside an RDF element.

<code>
:- dynamic
        in_rdf/0.

load_rdf(File) :-
        retractall(in_rdf),
        open(File, read, In),
        new_sgml_parser(Parser, []),
        set_sgml_parser(Parser, file(File)),
        set_sgml_parser(Parser, dialect(xml)),
        sgml_parse(Parser,
                   [ source(In),
                     call(begin, on_begin),
                     call(end, on_end)
                   ]),
        close(In).

on_end('RDF', _) :-
        retractall(in_rdf).

on_begin('RDF', _, _) :-
        assert(in_rdf).
on_begin(Tag, Attr, Parser) :-
        in_rdf, !,
        sgml_parse(Parser,
                   [ document(Content),
                     parse(content)
                   ]),
        process_rdf_description(element(Tag, Attr, Content)).
</code>


<s1 name="indexaccess">Processing Indexed Files</title>

<p>
In some cases applications which to process small portions of large 
SGML, XML or RDF files. For example, the <em>OpenDirectory</em> project 
by Netscape has produced a 90MB RDF file representing the main index. 
The parser described here can process this document as a unit, but 
loading takes 85 seconds on a Pentium-II 450 and the resulting term 
requires about 70MB global stack. One option is to process the entire 
document and output it as a Prolog fact-base of RDF triplets, but in 
many cases this is undesirable. Another example is a large SGML file 
containing online documentation. The application normally wishes to 
provide only small portions at a time to the user. Loading the entire 
document into memory is then undesirable.

<p>
Using the <term>parse(element)</term> option, we open a file, seek 
(using <pref builtin>seek/4</pref>) to the position of the element and
read the desired element.

<p>
The index can be built using the call-back interface of
<pref>sgml_parse/2</pref>. For example, the following code makes an
index of the <ref file/ structure.rdf/ file of the OpenDirectory
project:

<code>
:- dynamic
        location/3.                     % Id, File, Offset

rdf_index(File) :-
        retractall(location(_,_)),
        open(File, read, In, [type(binary)]),
        new_sgml_parser(Parser, []),
        set_sgml_parser(Parser, file(File)),
        set_sgml_parser(Parser, dialect(xml)),
        sgml_parse(Parser,
                   [ source(In),
                     call(begin, index_on_begin)
                   ]),
        close(In).

index_on_begin(_Element, Attributes, Parser) :-
        memberchk('r:id'=Id, Attributes),
        get_sgml_parser(Parser, charpos(Offset)),
        get_sgml_parser(Parser, file(File)),
        assert(location(Id, File, Offset)).
</code>

<p>
The following code extracts the RDF element with required id:

<code>
rdf_element(Id, Term) :-
        location(Id, File, Offset),
        load_structure(File, Term,
                       [ dialect(xml),
                         offset(Offset),
                         parse(element)
                       ]).
</code>

<s1>External entities</title>

<p>
While processing an SGML document the document may refer to external 
data. This occurs in three places: external parameter entities, normal 
external entities and the <c/DOCTYPE/ declaration. The current version 
of this tool deals rather primitively with external data. External 
entities can only be loaded from a file and the mapping between the 
entity names and the file is done using a <jargon/catalog/ file in a 
format compatible with that used by James Clark's SP Parser,
based on the SGML Open (now OASIS) specification.

<p>
Catalog files can be specified using two primitives: the predicate
<pref>sgml_register_catalog_file/2</pref> or the environment variable
<ref env/SGML_CATALOG_FILES/ (compatible with the SP package).

<definitions>
    <pred>sgml_register_catalog_file(+File, +Location)
    <summary>Register a catalog file
Register the indicated <aref/File/ as a catalog file. <aref/Location/ is
either <c/start/ or <c/end/ and defines whether the catalog is 
considered first or last. This predicate has no effect if <aref/File/ is
already part of the catalog.

<p>
If no files are registered using this predicate, the first query on the 
catalog examines <ref env/SGML_CATALOG_FILES/ and fills the catalog with
all files in this path. 
</definitions>


<p>
Two types of lines are used by this package.
<!--Why are DOCTYPE, ENTITY, and NOTATION not supported? 
-->
<xmp sgml block>
<c/DOCTYPE/ <aref/doctype/ <aref/file/
<c/PUBLIC/ <xmp/"/<aref/Id/<xmp/"/ <aref/file/ 
</xmp>

<p>
The specified <aref/file/ path is taken relative to the location of the 
catolog file. For the <c/DOCTYPE/ declaraction, <ref pkg/sgml2pl/ first makes 
an attempt to resolve the <c/SYSTEM/ or <c/PUBLIC/ identifier. If this 
fails it tries to resolve the <aref/doctype/ using the provided catalog 
files.

<p>
Strictly speaking, <ref pkg/sgml2pl/ breaks the rules for XML, where
system identifiers must be Universal Resource Indicators, not local file
names.  Simple uses of relative URIs will work correctly under UNIX
and Windows.

<p>
In the future we will design a call-back mechanism for locating and 
processing external entities, so Prolog-based file-location and Prolog 
resources can be used to store external entities.


<s1>Unsupported features</title>

<p>
The current parser is rather limited. While it is able to deal with many
serious documents, it omits several less-used features of SGML and XML.
Known missing SGML features include

<ul>
    <tag>NOTATION on entities
Though notation is parsed, notation attributes on external entity 
declarations are not handed to the user. 
    <tag>NOTATION attributes
SGML notations may have attributes, declared using
<verb><!ATTLIST #NOTATION name attributes></verb>. Those data attributes
are provided when you declare an external CDATA, NDATA, or SDATA entity.

<p>
XML does not include external CDATA, NDATA, or SDATA entities,
nor any of the other uses to which data attributes are put in SGML,
so it doesn't include data attributes for notations either.

<p>
Sgml2pl does not support this feature and is unlikely to;
you should be aware that SGML documents using this feature cannot
be converted faithfully to XML.
    <tag>SHORTTAG
The SGML SHORTTAG syntax is only partially implemented. Currently,
<verb><tag/content/</verb> is a valid abbreviation for
<verb><tag>content</tag></verb>, which can also be written as
<verb><tag>content</></verb>.
Empty start tags (<verb><></verb>), unclosed start tags
(<verb><a<b</verb>) and unclosed end tags (<verb></a<b</verb>) are not
supported.
    <tag>SGML declaration
The `SGML declaration' is fixed, though most of the parameters are 
handled through indirections in the implementation. 
    <tag>The DATATAG feature
It is regarded as superseeded by SHORTREF, which is supported.
(SP does not support it either.)
    <tag>The RANK feature
It is regarded as obsolete.
    <tag>The LINK feature
It is regarded as too complicated.
    <tag>The CONCUR feature
Concurrent markup allows a document to be tagged according to more than
one DTD at the same time.  It is not supported.
</ul>

<p>
In XML mode the parser recognises SGML constructs that are not allowed 
in XML. Also various extensions of XML over SGML are not yet realised.
In particular, XInclude is not implemented because the designers of
XInclude can't make up their minds whether to base it on elements or
attributes yet, let alone details.


<s1>Installation</title>

<s2>Unix systems</title>

<p>
Installation on Unix system uses the commonly found <sh>configure</sh>,
<sh>make</sh> and <sh>make install</sh> sequence. SWI-Prolog should be 
installed before building this package. If SWI-Prolog is not installed 
as <ref exe/pl/, the environment variable <ref env/PL/ must be set to 
the name of the SWI-Prolog executable. Installation is now accomplished 
using:

<code sh>
% ./configure
% make
% make install
</code>

<p>
This installs the foreign libraries in <ref file>$PLBASE/lib/$PLARCH</ref> and
the Prolog library files in <ref file>$PLBASE/library</ref>, where
<ref file/$PLBASE/ refers to the SWI-Prolog `home-directory'.

<s1>Acknowledgements</title>

<p>
The Prolog representation for parsed documents is based on the 
SWI-Prolog interface to SP by Anjo Anjewierden.

<p>
Richard O'Keefe has put a lot of effort testing and providing bug
reports consisting of an illustrative example and explanation of the
standard.  He also made many suggestions for improving this document.
</pldoc>

