--- /dev/null
+<chapter id="quick-start">
+<title>Quick Start </title>
+
+<para>
+In this section, we will test the system by indexing a small set of sample
+GILS records that are included with the software distribution. Go to the
+<literal remap="tt">test/gils</literal> subdirectory of the distribution archive. There you will
+find a configuration
+file named <literal remap="tt">zebra.cfg</literal> with the following contents:
+
+<screen>
+# Where are the YAZ tables located.
+profilePath: ../../../yaz/tab ../../tab
+
+# Files that describe the attribute sets supported.
+attset: bib1.att
+attset: gils.att
+</screen>
+
+</para>
+
+<para>
+Now, edit the file and set <literal remap="tt">profilePath</literal> to the path of the
+YAZ profile tables (sub directory <literal remap="tt">tab</literal> of the YAZ distribution
+archive).
+</para>
+
+<para>
+The 48 test records are located in the sub directory <literal remap="tt">records</literal>.
+To index these, type:
+
+<screen>
+$ ../../index/zebraidx -t grs.sgml update records
+</screen>
+
+</para>
+
+<para>
+In the command above the option <literal remap="tt">-t</literal> specified the record
+type — in this case <literal remap="tt">grs.sgml</literal>. The word <literal remap="tt">update</literal> followed
+by a directory root updates all files below that directory node.
+</para>
+
+<para>
+If your indexing command was successful, you are now ready to
+fire up a server. To start a server on port 2100, type:
+
+<screen>
+$ ../../index/zebrasrv tcp:@:2100
+</screen>
+
+</para>
+
+<para>
+The Zebra index that you have just created has a single database
+named <literal remap="tt">Default</literal>. The database contains records structured according to
+the GILS profile, and the server will
+return records in either either USMARC, GRS-1, or SUTRS depending
+on what your client asks
+for.
+</para>
+
+<para>
+To test the server, you can use any Z39.50 client (1992 or later). For
+instance, you can use the demo client that comes with YAZ: Just cd to
+the <literal remap="tt">client</literal> subdirectory of the YAZ distribution and type:
+</para>
+
+<para>
+
+<screen>
+$ client tcp:localhost:2100
+</screen>
+
+</para>
+
+<para>
+When the client has connected, you can type:
+</para>
+
+<para>
+
+<screen>
+Z> find surficial
+Z> show 1
+</screen>
+
+</para>
+
+<para>
+The default retrieval syntax for the client is USMARC. To try other
+formats for the same record, try:
+</para>
+
+<para>
+
+<screen>
+Z>format sutrs
+Z>show 1
+Z>format grs-1
+Z>show 1
+Z>elements B
+Z>show 1
+</screen>
+
+</para>
+
+<para>
+<emphasis remap="it">NOTE: You may notice that more fields are returned when your
+client requests SUTRS or GRS-1 records. When retrieving GILS records,
+this is normal - not all of the GILS data elements have mappings in
+the USMARC record format.</emphasis>
+</para>
+
+<para>
+If you've made it this far, there's a good chance that
+you've got through the compilation OK.
+</para>
+
+</chapter>
+
+<chapter id="administration">
+<title>Administrating Zebra</title>
+
+<para>
+Unlike many simpler retrieval systems, Zebra supports safe, incremental
+updates to an existing index.
+</para>
+
+<para>
+Normally, when Zebra modifies the index it reads a number of records
+that you specify.
+Depending on your specifications and on the contents of each record
+one the following events take place for each record:
+<variablelist>
+
+<varlistentry>
+<term>Insert</term>
+<listitem>
+<para>
+The record is indexed as if it never occurred
+before. Either the Zebra system doesn't know how to identify the record or
+Zebra can identify the record but didn't find it to be already indexed.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>Modify</term>
+<listitem>
+<para>
+The record has already been indexed. In this case
+either the contents of the record or the location (file) of the record
+indicates that it has been indexed before.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>Delete</term>
+<listitem>
+<para>
+The record is deleted from the index. As in the
+update-case it must be able to identify the record.
+</para>
+</listitem>
+</varlistentry>
+</variablelist>
+</para>
+
+<para>
+Please note that in both the modify- and delete- case the Zebra
+indexer must be able to generate a unique key that identifies the record in
+question (more on this below).
+</para>
+
+<para>
+To administrate the Zebra retrieval system, you run the
+<literal remap="tt">zebraidx</literal> program. This program supports a number of options
+which are preceded by a minus, and a few commands (not preceded by
+minus).
+</para>
+
+<para>
+Both the Zebra administrative tool and the Z39.50 server share a
+set of index files and a global configuration file. The
+name of the configuration file defaults to <literal remap="tt">zebra.cfg</literal>.
+The configuration file includes specifications on how to index
+various kinds of records and where the other configuration files
+are located. <literal remap="tt">zebrasrv</literal> and <literal remap="tt">zebraidx</literal> <emphasis>must</emphasis>
+be run in the directory where the configuration file lives unless you
+indicate the location of the configuration file by option
+<literal remap="tt">-c</literal>.
+</para>
+
+<sect1 id="record-types">
+<title>Record Types</title>
+
+<para>
+Indexing is a per-record process, in which either insert/modify/delete
+will occur. Before a record is indexed search keys are extracted from
+whatever might be the layout the original record (sgml,html,text, etc..).
+The Zebra system currently supports two fundamantal types of records:
+structured and simple text.
+To specify a particular extraction process, use either the
+command line option <literal remap="tt">-t</literal> or specify a
+<literal remap="tt">recordType</literal> setting in the configuration file.
+</para>
+
+</sect1>
+
+<sect1 id="configuration-file">
+<title>The Zebra Configuration File</title>
+
+<para>
+The Zebra configuration file, read by <literal remap="tt">zebraidx</literal> and
+<literal remap="tt">zebrasrv</literal> defaults to <literal remap="tt">zebra.cfg</literal> unless specified
+by <literal remap="tt">-c</literal> option.
+</para>
+
+<para>
+You can edit the configuration file with a normal text editor.
+parameter names and values are seperated by colons in the file. Lines
+starting with a hash sign (<literal remap="tt">#</literal>) are treated as comments.
+</para>
+
+<para>
+If you manage different sets of records that share common
+characteristics, you can organize the configuration settings for each
+type into "groups".
+When <literal remap="tt">zebraidx</literal> is run and you wish to address a given group
+you specify the group name with the <literal remap="tt">-g</literal> option. In this case
+settings that have the group name as their prefix will be used
+by <literal remap="tt">zebraidx</literal>. If no <literal remap="tt">-g</literal> option is specified, the settings
+with no prefix are used.
+</para>
+
+<para>
+In the configuration file, the group name is placed before the option
+name itself, separated by a dot (.). For instance, to set the record type
+for group <literal remap="tt">public</literal> to <literal remap="tt">grs.sgml</literal> (the SGML-like format for structured
+records) you would write:
+</para>
+
+<para>
+
+<screen>
+public.recordType: grs.sgml
+</screen>
+
+</para>
+
+<para>
+To set the default value of the record type to <literal remap="tt">text</literal> write:
+</para>
+
+<para>
+
+<screen>
+recordType: text
+</screen>
+
+</para>
+
+<para>
+The available configuration settings are summarized below. They will be
+explained further in the following sections.
+</para>
+
+<para>
+<variablelist>
+
+<varlistentry>
+<term><emphasis remap="it">group</emphasis>.recordType[<emphasis remap="it">.name</emphasis>]</term>
+<listitem>
+<para>
+Specifies how records with the file extension <emphasis remap="it">name</emphasis> should
+be handled by the indexer. This option may also be specified
+as a command line option (<literal remap="tt">-t</literal>). Note that if you do not
+specify a <emphasis remap="it">name</emphasis>, the setting applies to all files. In general,
+the record type specifier consists of the elements (each
+element separated by dot), <emphasis remap="it">fundamental-type</emphasis>,
+<emphasis remap="it">file-read-type</emphasis> and arguments. Currently, two
+fundamental types exist, <literal remap="tt">text</literal> and <literal remap="tt">grs</literal>.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term><emphasis remap="it">group</emphasis>.recordId</term>
+<listitem>
+<para>
+Specifies how the records are to be identified when updated. See
+section <xref linkend="locating-records"/>.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term><emphasis remap="it">group</emphasis>.database</term>
+<listitem>
+<para>
+Specifies the Z39.50 database name.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term><emphasis remap="it">group</emphasis>.storeKeys</term>
+<listitem>
+<para>
+Specifies whether key information should be saved for a given
+group of records. If you plan to update/delete this type of
+records later this should be specified as 1; otherwise it
+should be 0 (default), to save register space. See section
+<xref linkend="file-ids"/>.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term><emphasis remap="it">group</emphasis>.storeData</term>
+<listitem>
+<para>
+Specifies whether the records should be stored internally
+in the Zebra system files. If you want to maintain the raw records yourself,
+this option should be false (0). If you want Zebra to take care of the records
+for you, it should be true(1).
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>register</term>
+<listitem>
+<para>
+Specifies the location of the various register files that Zebra uses
+to represent your databases. See section
+<xref linkend="register-location"/>.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>shadow</term>
+<listitem>
+<para>
+Enables the <emphasis remap="it">safe update</emphasis> facility of Zebra, and tells the system
+where to place the required, temporary files. See section
+<xref linkend="shadow-registers"/>.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>lockDir</term>
+<listitem>
+<para>
+Directory in which various lock files are stored.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>keyTmpDir</term>
+<listitem>
+<para>
+Directory in which temporary files used during zebraidx' update
+phase are stored.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>setTmpDir</term>
+<listitem>
+<para>
+Specifies the directory that the server uses for temporary result sets.
+If not specified <literal remap="tt">/tmp</literal> will be used.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>profilePath</term>
+<listitem>
+<para>
+Specifies the location of profile specification files.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>attset</term>
+<listitem>
+<para>
+Specifies the filename(s) of attribute set files for use in
+searching. At least the Bib-1 set should be loaded (<literal remap="tt">bib1.att</literal>).
+The <literal remap="tt">profilePath</literal> setting is used to look for the specified files.
+See section <xref linkend="attset-files"/>
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>memMax</term>
+<listitem>
+<para>
+Specifies size of internal memory to use for the zebraidx program. The
+amount is given in megabytes - default is 4 (4 MB).
+</para>
+</listitem>
+</varlistentry>
+</variablelist>
+</para>
+
+</sect1>
+
+<sect1 id="locating-records">
+<title>Locating Records</title>
+
+<para>
+The default behaviour of the Zebra system is to reference the
+records from their original location, i.e. where they were found when you
+ran <literal remap="tt">zebraidx</literal>. That is, when a client wishes to retrieve a record
+following a search operation, the files are accessed from the place
+where you originally put them - if you remove the files (without
+running <literal remap="tt">zebraidx</literal> again, the client will receive a diagnostic
+message.
+</para>
+
+<para>
+If your input files are not permanent - for example if you retrieve
+your records from an outside source, or if they were temporarily
+mounted on a CD-ROM drive,
+you may want Zebra to make an internal copy of them. To do this,
+you specify 1 (true) in the <literal remap="tt">storeData</literal> setting. When
+the Z39.50 server retrieves the records they will be read from the
+internal file structures of the system.
+</para>
+
+</sect1>
+
+<sect1 id="simple-indexing">
+<title>Indexing with no Record IDs (Simple Indexing)</title>
+
+<para>
+If you have a set of records that are not expected to change over time
+you may can build your database without record IDs.
+This indexing method uses less space than the other methods and
+is simple to use.
+</para>
+
+<para>
+To use this method, you simply omit the <literal remap="tt">recordId</literal> entry
+for the group of files that you index. To add a set of records you use
+<literal remap="tt">zebraidx</literal> with the <literal remap="tt">update</literal> command. The
+<literal remap="tt">update</literal> command will always add all of the records that it
+encounters to the index - whether they have already been indexed or
+not. If the set of indexed files change, you should delete all of the
+index files, and build a new index from scratch.
+</para>
+
+<para>
+Consider a system in which you have a group of text files called
+<literal remap="tt">simple</literal>. That group of records should belong to a Z39.50 database
+called <literal remap="tt">textbase</literal>. The following <literal remap="tt">zebra.cfg</literal> file will suffice:
+</para>
+
+<para>
+
+<screen>
+profilePath: /usr/local/yaz
+attset: bib1.att
+simple.recordType: text
+simple.database: textbase
+</screen>
+
+</para>
+
+<para>
+Since the existing records in an index can not be addressed by their
+IDs, it is impossible to delete or modify records when using this method.
+</para>
+
+</sect1>
+
+<sect1 id="file-ids">
+<title>Indexing with File Record IDs</title>
+
+<para>
+If you have a set of files that regularly change over time: Old files
+are deleted, new ones are added, or existing files are modified, you
+can benefit from using the <emphasis remap="it">file ID</emphasis> indexing methodology. Examples
+of this type of database might include an index of WWW resources, or a
+USENET news spool area. Briefly speaking, the file key methodology
+uses the directory paths of the individual records as a unique
+identifier for each record. To perform indexing of a directory with
+file keys, again, you specify the top-level directory after the
+<literal remap="tt">update</literal> command. The command will recursively traverse the
+directories and compare each one with whatever have been indexed before in
+that same directory. If a file is new (not in the previous version of
+the directory) it is inserted into the registers; if a file was
+already indexed and it has been modified since the last update,
+the index is also modified; if a file has been removed since the last
+visit, it is deleted from the index.
+</para>
+
+<para>
+The resulting system is easy to administrate. To delete a record you
+simply have to delete the corresponding file (say, with the <literal remap="tt">rm</literal>
+command). And to add records you create new files (or directories with
+files). For your changes to take effect in the register you must run
+<literal remap="tt">zebraidx update</literal> with the same directory root again. This mode
+of operation requires more disk space than simpler indexing methods,
+but it makes it easier for you to keep the index in sync with a
+frequently changing set of data. If you combine this system with the
+<emphasis remap="it">safe update</emphasis> facility (see below), you never have to take your
+server offline for maintenance or register updating purposes.
+</para>
+
+<para>
+To enable indexing with pathname IDs, you must specify <literal remap="tt">file</literal> as
+the value of <literal remap="tt">recordId</literal> in the configuration file. In addition,
+you should set <literal remap="tt">storeKeys</literal> to <literal remap="tt">1</literal>, since the Zebra
+indexer must save additional information about the contents of each record
+in order to modify the indices correctly at a later time.
+</para>
+
+<para>
+For example, to update records of group <literal remap="tt">esdd</literal> located below
+<literal remap="tt">/data1/records/</literal> you should type:
+
+<screen>
+$ zebraidx -g esdd update /data1/records
+</screen>
+
+</para>
+
+<para>
+The corresponding configuration file includes:
+
+<screen>
+esdd.recordId: file
+esdd.recordType: grs.sgml
+esdd.storeKeys: 1
+</screen>
+
+</para>
+
+<para>
+<emphasis>Important note: You cannot start out with a group of records with simple
+indexing (no record IDs as in the previous section) and then later
+enable file record Ids. Zebra must know from the first time that you
+index the group that
+the files should be indexed with file record IDs.</emphasis>
+</para>
+
+<para>
+You cannot explicitly delete records when using this method (using the
+<emphasis remap="bf">delete</emphasis> command to <literal remap="tt">zebraidx</literal>. Instead
+you have to delete the files from the file system (or move them to a
+different location)
+and then run <literal remap="tt">zebraidx</literal> with the <emphasis remap="bf">update</emphasis> command.
+</para>
+
+</sect1>
+
+<sect1 id="generic-ids">
+<title>Indexing with General Record IDs</title>
+
+<para>
+When using this method you construct an (almost) arbritrary, internal
+record key based on the contents of the record itself and other system
+information. If you have a group of records that explicitly associates
+an ID with each record, this method is convenient. For example, the
+record format may contain a title or a ID-number - unique within the group.
+In either case you specify the Z39.50 attribute set and use-attribute
+location in which this information is stored, and the system looks at
+that field to determine the identity of the record.
+</para>
+
+<para>
+As before, the record ID is defined by the <literal remap="tt">recordId</literal> setting
+in the configuration file. The value of the record ID specification
+consists of one or more tokens separated by whitespace. The resulting
+ID is
+represented in the index by concatenating the tokens and separating them by
+ASCII value (1).
+</para>
+
+<para>
+There are three kinds of tokens:
+<variablelist>
+
+<varlistentry>
+<term>Internal record info</term>
+<listitem>
+<para>
+The token refers to a key that is
+extracted from the record. The syntax of this token is
+<literal remap="tt">(</literal> <emphasis>set</emphasis> <literal remap="tt">,</literal> <emphasis>use</emphasis> <literal remap="tt">)</literal>, where <emphasis>set</emphasis> is the
+attribute set name <emphasis>use</emphasis> is the name or value of the attribute.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>System variable</term>
+<listitem>
+<para>
+The system variables are preceded by
+
+<screen>
+$
+</screen>
+ and immediately followed by the system variable name, which
+may one of
+<variablelist>
+
+<varlistentry>
+<term>group</term>
+<listitem>
+<para>
+Group name.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>database</term>
+<listitem>
+<para>
+Current database specified.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>type</term>
+<listitem>
+<para>
+Record type.
+</para>
+</listitem>
+</varlistentry>
+</variablelist>
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>Constant string</term>
+<listitem>
+<para>
+A string used as part of the ID — surrounded
+by single- or double quotes.
+</para>
+</listitem>
+</varlistentry>
+</variablelist>
+</para>
+
+<para>
+For instance, the sample GILS records that come with the Zebra
+distribution contain a unique ID in the data tagged Control-Identifier.
+The data is mapped to the Bib-1 use attribute Identifier-standard
+(code 1007). To use this field as a record id, specify
+<literal remap="tt">(bib1,Identifier-standard)</literal> as the value of the
+<literal remap="tt">recordId</literal> in the configuration file.
+If you have other record types that uses the same field for a
+different purpose, you might add the record type
+(or group or database name) to the record id of the gils
+records as well, to prevent matches with other types of records.
+In this case the recordId might be set like this:
+
+<screen>
+gils.recordId: $type (bib1,Identifier-standard)
+</screen>
+
+</para>
+
+<para>
+(see section <xref linkend="data-model"/>
+for details of how the mapping between elements of your records and
+searchable attributes is established).
+</para>
+
+<para>
+As for the file record ID case described in the previous section,
+updating your system is simply a matter of running <literal remap="tt">zebraidx</literal>
+with the <literal remap="tt">update</literal> command. However, the update with general
+keys is considerably slower than with file record IDs, since all files
+visited must be (re)read to discover their IDs.
+</para>
+
+<para>
+As you might expect, when using the general record IDs
+method, you can only add or modify existing records with the <literal remap="tt">update</literal>
+command. If you wish to delete records, you must use the,
+<literal remap="tt">delete</literal> command, with a directory as a parameter.
+This will remove all records that match the files below that root
+directory.
+</para>
+
+</sect1>
+
+<sect1 id="register-location">
+<title>Register Location</title>
+
+<para>
+Normally, the index files that form dictionaries, inverted
+files, record info, etc., are stored in the directory where you run
+<literal remap="tt">zebraidx</literal>. If you wish to store these, possibly large, files
+somewhere else, you must add the <literal remap="tt">register</literal> entry to the
+<literal remap="tt">zebra.cfg</literal> file. Furthermore, the Zebra system allows its file
+structures to
+span multiple file systems, which is useful for managing very large
+databases.
+</para>
+
+<para>
+The value of the <literal remap="tt">register</literal> setting is a sequence of tokens.
+Each token takes the form:
+
+<screen>
+<emphasis>dir</emphasis><literal remap="tt">:</literal><emphasis>size</emphasis>.
+</screen>
+
+The <emphasis>dir</emphasis> specifies a directory in which index files will be
+stored and the <emphasis>size</emphasis> specifies the maximum size of all
+files in that directory. The Zebra indexer system fills each directory
+in the order specified and use the next specified directories as needed.
+The <emphasis>size</emphasis> is an integer followed by a qualifier
+code, <literal remap="tt">M</literal> for megabytes, <literal remap="tt">k</literal> for kilobytes.
+</para>
+
+<para>
+For instance, if you have allocated two disks for your register, and
+the first disk is mounted
+on <literal remap="tt">/d1</literal> and has 200 Mb of free space and the
+second, mounted on <literal remap="tt">/d2</literal> has 300 Mb, you could
+put this entry in your configuration file:
+
+<screen>
+register: /d1:200M /d2:300M
+</screen>
+
+</para>
+
+<para>
+Note that Zebra does not verify that the amount of space specified is
+actually available on the directory (file system) specified - it is
+your responsibility to ensure that enough space is available, and that
+other applications do not attempt to use the free space. In a large production system,
+it is recommended that you allocate one or more filesystem exclusively
+to the Zebra register files.
+</para>
+
+</sect1>
+
+<sect1 id="shadow-registers">
+<title>Safe Updating - Using Shadow Registers</title>
+
+<sect2>
+<title>Description</title>
+
+<para>
+The Zebra server supports <emphasis remap="it">updating</emphasis> of the index structures. That is,
+you can add, modify, or remove records from databases managed by Zebra
+without rebuilding the entire index. Since this process involves
+modifying structured files with various references between blocks of
+data in the files, the update process is inherently sensitive to
+system crashes, or to process interruptions: Anything but a
+successfully completed update process will leave the register files in
+an unknown state, and you will essentially have no recourse but to
+re-index everything, or to restore the register files from a backup
+medium. Further, while the update process is active, users cannot be
+allowed to access the system, as the contents of the register files
+may change unpredictably.
+</para>
+
+<para>
+You can solve these problems by enabling the shadow register system in
+Zebra. During the updating procedure, <literal remap="tt">zebraidx</literal> will temporarily
+write changes to the involved files in a set of "shadow
+files", without modifying the files that are accessed by the
+active server processes. If the update procedure is interrupted by a
+system crash or a signal, you simply repeat the procedure - the
+register files have not been changed or damaged, and the partially
+written shadow files are automatically deleted before the new updating
+procedure commences.
+</para>
+
+<para>
+At the end of the updating procedure (or in a separate operation, if
+you so desire), the system enters a "commit mode". First,
+any active server processes are forced to access those blocks that
+have been changed from the shadow files rather than from the main
+register files; the unmodified blocks are still accessed at their
+normal location (the shadow files are not a complete copy of the
+register files - they only contain those parts that have actually been
+modified). If the commit process is interrupted at any point during the
+commit process, the server processes will continue to access the
+shadow files until you can repeat the commit procedure and complete
+the writing of data to the main register files. You can perform
+multiple update operations to the registers before you commit the
+changes to the system files, or you can execute the commit operation
+at the end of each update operation. When the commit phase has
+completed successfully, any running server processes are instructed to
+switch their operations to the new, operational register, and the
+temporary shadow files are deleted.
+</para>
+
+</sect2>
+
+<sect2>
+<title>How to Use Shadow Register Files</title>
+
+<para>
+The first step is to allocate space on your system for the shadow
+files. You do this by adding a <literal remap="tt">shadow</literal> entry to the <literal remap="tt">zebra.cfg</literal>
+file. The syntax of the <literal remap="tt">shadow</literal> entry is exactly the same as for
+the <literal remap="tt">register</literal> entry (see section <xref linkend="register-location"/>). The location of the shadow area should be
+<emphasis remap="it">different</emphasis> from the location of the main register area (if you
+have specified one - remember that if you provide no <literal remap="tt">register</literal>
+setting, the default register area is the
+working directory of the server and indexing processes).
+</para>
+
+<para>
+The following excerpt from a <literal remap="tt">zebra.cfg</literal> file shows one example of
+a setup that configures both the main register location and the shadow
+file area. Note that two directories or partitions have been set aside
+for the shadow file area. You can specify any number of directories
+for each of the file areas, but remember that there should be no
+overlaps between the directories used for the main registers and the
+shadow files, respectively.
+</para>
+
+<para>
+
+<screen>
+register: /d1:500M
+
+shadow: /scratch1:100M /scratch2:200M
+</screen>
+
+</para>
+
+<para>
+When shadow files are enabled, an extra command is available at the
+<literal remap="tt">zebraidx</literal> command line. In order to make changes to the system
+take effect for the users, you'll have to submit a
+"commit" command after a (sequence of) update
+operation(s). You can ask the indexer to commit the changes
+immediately after the update operation:
+</para>
+
+<para>
+
+<screen>
+$ zebraidx update /d1/records update /d2/more-records commit
+</screen>
+
+</para>
+
+<para>
+Or you can execute multiple updates before committing the changes:
+</para>
+
+<para>
+
+<screen>
+$ zebraidx -g books update /d1/records update /d2/more-records
+$ zebraidx -g fun update /d3/fun-records
+$ zebraidx commit
+</screen>
+
+</para>
+
+<para>
+If one of the update operations above had been interrupted, the commit
+operation on the last line would fail: <literal remap="tt">zebraidx</literal> will not let you
+commit changes that would destroy the running register. You'll have to
+rerun all of the update operations since your last commit operation,
+before you can commit the new changes.
+</para>
+
+<para>
+Similarly, if the commit operation fails, <literal remap="tt">zebraidx</literal> will not let
+you start a new update operation before you have successfully repeated
+the commit operation. The server processes will keep accessing the
+shadow files rather than the (possibly damaged) blocks of the main
+register files until the commit operation has successfully completed.
+</para>
+
+<para>
+You should be aware that update operations may take slightly longer
+when the shadow register system is enabled, since more file access
+operations are involved. Further, while the disk space required for
+the shadow register data is modest for a small update operation, you
+may prefer to disable the system if you are adding a very large number
+of records to an already very large database (we use the terms
+<emphasis remap="it">large</emphasis> and <emphasis remap="it">modest</emphasis> very loosely here, since every
+application will have a different perception of size). To update the system
+without the use of the the shadow files, simply run <literal remap="tt">zebraidx</literal> with
+the <literal remap="tt">-n</literal> option (note that you do not have to execute the
+<emphasis remap="bf">commit</emphasis> command of <literal remap="tt">zebraidx</literal> when you temporarily disable the
+use of the shadow registers in this fashion. Note also that, just as
+when the shadow registers are not enabled, server processes will be
+barred from accessing the main register while the update procedure
+takes place.
+</para>
+
+</sect2>
+
+</sect1>
+
+</chapter>
+
+<chapter id="zebraidx">
+<title>Running the Maintenance Interface (zebraidx)</title>
+
+<para>
+The following is a complete reference to the command line interface to
+the <literal remap="tt">zebraidx</literal> application.
+</para>
+
+<para>
+<emphasis remap="bf">Syntax</emphasis>
+
+<screen>
+$ zebraidx [options] command [directory] ...
+</screen>
+
+<emphasis remap="bf">Options</emphasis>
+<variablelist>
+
+<varlistentry>
+<term>-t <emphasis remap="it">type</emphasis></term>
+<listitem>
+<para>
+Update all files as <emphasis remap="it">type</emphasis>. Currently, the
+types supported are <literal remap="tt">text</literal> and <literal remap="tt">grs</literal><emphasis remap="it">.subtype</emphasis>. If no
+<emphasis remap="it">subtype</emphasis> is provided for the GRS (General Record Structure) type,
+the canonical input format is assumed (see section <xref linkend="local-representation"/>). Generally, it
+is probably advisable to specify the record types in the
+<literal remap="tt">zebra.cfg</literal> file
+(see section <xref linkend="record-types"/>), to avoid
+confusion at subsequent updates.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>-c <emphasis remap="it">config-file</emphasis></term>
+<listitem>
+<para>
+Read the configuration file
+<emphasis remap="it">config-file</emphasis> instead of <literal remap="tt">zebra.cfg</literal>.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>-g <emphasis remap="it">group</emphasis></term>
+<listitem>
+<para>
+Update the files according to the group
+settings for <emphasis remap="it">group</emphasis> (see section
+<xref linkend="configuration-file"/>).
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>-d <emphasis remap="it">database</emphasis></term>
+<listitem>
+<para>
+The records located should be associated
+with the database name <emphasis remap="it">database</emphasis> for access through the Z39.50
+server.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>-m <emphasis remap="it">mbytes</emphasis></term>
+<listitem>
+<para>
+Use <emphasis remap="it">mbytes</emphasis> of megabytes before flushing
+keys to background storage. This setting affects performance when
+updating large databases.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>-n</term>
+<listitem>
+<para>
+Disable the use of shadow registers for this operation
+(see section <xref linkend="shadow-registers"/>).
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>-s</term>
+<listitem>
+<para>
+Show analysis of the indexing process. The maintenance
+program works in a read-only mode and doesn't change the state
+of the index. This options is very useful when you wish to test a
+new profile.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>-V</term>
+<listitem>
+<para>
+Show Zebra version.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>-v <emphasis remap="it">level</emphasis></term>
+<listitem>
+<para>
+Set the log level to <emphasis remap="it">level</emphasis>. <emphasis remap="it">level</emphasis>
+should be one of <literal remap="tt">none</literal>, <literal remap="tt">debug</literal>, and <literal remap="tt">all</literal>.
+</para>
+</listitem>
+</varlistentry>
+</variablelist>
+</para>
+
+<para>
+<emphasis remap="bf">Commands</emphasis>
+<variablelist>
+
+<varlistentry>
+<term>Update <emphasis remap="it">directory</emphasis></term>
+<listitem>
+<para>
+Update the register with the files
+contained in <emphasis remap="it">directory</emphasis>. If no directory is provided, a list of
+files is read from <literal remap="tt">stdin</literal>.
+See section <xref linkend="administration"/>.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>Delete <emphasis remap="it">directory</emphasis></term>
+<listitem>
+<para>
+Remove the records corresponding to
+the files found under <emphasis remap="it">directory</emphasis> from the register.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>Commit</term>
+<listitem>
+<para>
+Write the changes resulting from the last <emphasis remap="bf">update</emphasis>
+commands to the register. This command is only available if the use of
+shadow register files is enabled (see section
+<xref linkend="shadow-registers"/>).
+</para>
+</listitem>
+</varlistentry>
+</variablelist>
+</para>
+
+</chapter>
+
+<chapter id="server">
+<title>The Z39.50 Server</title>
+
+<sect1 id="zebrasrv">
+<title>Running the Z39.50 Server (zebrasrv)</title>
+
+<para>
+<emphasis remap="bf">Syntax</emphasis>
+
+<screen>
+zebrasrv [options] [listener-address ...]
+</screen>
+
+</para>
+
+<para>
+<emphasis remap="bf">Options</emphasis>
+<variablelist>
+
+<varlistentry>
+<term>-a <emphasis remap="it">APDU file</emphasis></term>
+<listitem>
+<para>
+Specify a file for dumping PDUs (for diagnostic purposes).
+The special name "-" sends output to <literal>stderr</literal>.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>-c <emphasis remap="it">config-file</emphasis></term>
+<listitem>
+<para>
+Read configuration information from <emphasis remap="it">config-file</emphasis>. The default configuration is <literal remap="tt">./zebra.cfg</literal>.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>-S</term>
+<listitem>
+<para>
+Don't fork on connection requests. This can be useful for
+symbolic-level debugging. The server can only accept a single
+connection in this mode.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>-s</term>
+<listitem>
+<para>
+Use the SR protocol.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>-z</term>
+<listitem>
+<para>
+Use the Z39.50 protocol (default). These two options complement
+eachother. You can use both multiple times on the same command
+line, between listener-specifications (see below). This way, you
+can set up the server to listen for connections in both protocols
+concurrently, on different local ports.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>-l <emphasis remap="it">logfile</emphasis></term>
+<listitem>
+<para>
+Specify an output file for the diagnostic
+messages. The default is to write this information to <literal remap="tt">stderr</literal>.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>-v <emphasis remap="it">log-level</emphasis></term>
+<listitem>
+<para>
+The log level. Use a comma-separated list of members of the set
+{fatal,debug,warn,log,all,none}.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>-u <emphasis remap="it">username</emphasis></term>
+<listitem>
+<para>
+Set user ID. Sets the real UID of the server process to that of the
+given <emphasis remap="it">username</emphasis>. It's useful if you aren't comfortable with having the
+server run as root, but you need to start it as such to bind a
+privileged port.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>-w <emphasis remap="it">working-directory</emphasis></term>
+<listitem>
+<para>
+Change working directory.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>-i</term>
+<listitem>
+<para>
+Run under the Internet superserver, <literal remap="tt">inetd</literal>. Make
+sure you use the logfile option <literal remap="tt">-l</literal> in conjunction with this
+mode and specify the <literal remap="tt">-l</literal> option before any other options.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>-t <emphasis remap="it">timeout</emphasis></term>
+<listitem>
+<para>
+Set the idle session timeout (default 60 minutes).
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>-k <emphasis remap="it">kilobytes</emphasis></term>
+<listitem>
+<para>
+Set the (approximate) maximum size of
+present response messages. Default is 1024 Kb (1 Mb).
+</para>
+</listitem>
+</varlistentry>
+</variablelist>
+</para>
+
+<para>
+A <emphasis remap="it">listener-address</emphasis> consists of a transport mode followed by a
+colon (:) followed by a listener address. The transport mode is
+either <literal remap="tt">osi</literal> or <literal remap="tt">tcp</literal>.
+</para>
+
+<para>
+For TCP, an address has the form
+</para>
+
+<para>
+
+<screen>
+hostname | IP-number [: portnumber]
+</screen>
+
+</para>
+
+<para>
+The port number defaults to 210 (standard Z39.50 port).
+</para>
+
+<para>
+For OSI (only available if the server is compiled with XTI/mOSI
+support enabled), the address form is
+</para>
+
+<para>
+
+<screen>
+[t-selector /] hostname | IP-number [: portnumber]
+</screen>
+
+</para>
+
+<para>
+The transport selector is given as a string of hex digits (with an even
+number of digits). The default port number is 102 (RFC1006 port).
+</para>
+
+<para>
+Examples
+</para>
+
+<para>
+
+<screen>
+tcp:dranet.dra.com
+
+osi:0402/dbserver.osiworld.com:3000
+</screen>
+
+</para>
+
+<para>
+In both cases, the special hostname "@" is mapped to
+the address INADDR_ANY, which causes the server to listen on any local
+interface. To start the server listening on the registered ports for
+Z39.50 and SR over OSI/RFC1006, and to drop root privileges once the
+ports are bound, execute the server like this (from a root shell):
+</para>
+
+<para>
+
+<screen>
+zebrasrv -u daemon tcp:@ -s osi:@
+</screen>
+
+</para>
+
+<para>
+You can replace <literal remap="tt">daemon</literal> with another user, eg. your own account, or
+a dedicated IR server account.
+</para>
+
+<para>
+The default behavior for <literal remap="tt">zebrasrv</literal> is to establish a single TCP/IP
+listener, for the Z39.50 protocol, on port 9999.
+</para>
+
+</sect1>
+
+<sect1 id="protocol-support">
+<title>Z39.50 Protocol Support and Behavior</title>
+
+<sect2>
+<title>Initialization</title>
+
+<para>
+During initialization, the server will negotiate to version 3 of the
+Z39.50 protocol, and the option bits for Search, Present, Scan,
+NamedResultSets, and concurrentOperations will be set, if requested by
+the client. The maximum PDU size is negotiated down to a maximum of
+1Mb by default.
+</para>
+
+</sect2>
+
+<sect2 id="search">
+<title>Search</title>
+
+<para>
+The supported query type are 1 and 101. All operators are currently
+supported with the restriction that only proximity units of type "word" are
+supported for the proximity operator.
+Queries can be arbitrarily complex.
+Named result sets are supported, and result sets can be used as operands
+without limitations.
+Searches may span multiple databases.
+</para>
+
+<para>
+The server has full support for piggy-backed present requests (see
+also the following section).
+</para>
+
+<para>
+<emphasis remap="bf">Use</emphasis> attributes are interpreted according to the attribute sets which
+have been loaded in the <literal remap="tt">zebra.cfg</literal> file, and are matched against
+specific fields as specified in the <literal remap="tt">.abs</literal> file which describes the
+profile of the records which have been loaded. If no <emphasis remap="bf">Use</emphasis>
+attribute is provided, a default of Bib-1 <emphasis remap="bf">Any</emphasis> is assumed.
+</para>
+
+<para>
+If a <emphasis remap="bf">Structure</emphasis> attribute of <emphasis remap="bf">Phrase</emphasis> is used in conjunction with a
+<emphasis remap="bf">Completeness</emphasis> attribute of <emphasis remap="bf">Complete (Sub)field</emphasis>, the term is
+matched against the contents of the phrase (long word) register, if one
+exists for the given <emphasis remap="bf">Use</emphasis> attribute.
+A phrase register is created for those fields in the <literal remap="tt">.abs</literal>
+file that contains a <literal remap="tt">p</literal>-specifier.
+</para>
+
+<para>
+If <emphasis remap="bf">Structure</emphasis>=<emphasis remap="bf">Phrase</emphasis> is used in conjunction with
+<emphasis remap="bf">Incomplete Field</emphasis> - the default value for <emphasis remap="bf">Completeness</emphasis>, the
+search is directed against the normal word registers, but if the term
+contains multiple words, the term will only match if all of the words
+are found immediately adjacent, and in the given order.
+The word search is performed on those fields that are indexed as
+type <literal remap="tt">w</literal> in the <literal remap="tt">.abs</literal> file.
+</para>
+
+<para>
+If the <emphasis remap="bf">Structure</emphasis> attribute is <emphasis remap="bf">Word List</emphasis>,
+<emphasis remap="bf">Free-form Text</emphasis>, or <emphasis remap="bf">Document Text</emphasis>, the term is treated as a
+natural-language, relevance-ranked query.
+This search type uses the word register, i.e. those fields
+that are indexed as type <literal remap="tt">w</literal> in the <literal remap="tt">.abs</literal> file.
+</para>
+
+<para>
+If the <emphasis remap="bf">Structure</emphasis> attribute is <emphasis remap="bf">Numeric String</emphasis> the
+term is treated as an integer. The search is performed on those
+fields that are indexed as type <literal remap="tt">n</literal> in the <literal remap="tt">.abs</literal> file.
+</para>
+
+<para>
+If the <emphasis remap="bf">Structure</emphasis> attribute is <emphasis remap="bf">URx</emphasis> the
+term is treated as a URX (URL) entity. The search is performed on those
+fields that are indexed as type <literal remap="tt">u</literal> in the <literal remap="tt">.abs</literal> file.
+</para>
+
+<para>
+If the <emphasis remap="bf">Structure</emphasis> attribute is <emphasis remap="bf">Local Number</emphasis> the
+term is treated as native Zebra Record Identifier.
+</para>
+
+<para>
+If the <emphasis remap="bf">Relation</emphasis> attribute is <emphasis remap="bf">Equals</emphasis> (default), the term is
+matched in a normal fashion (modulo truncation and processing of
+individual words, if required). If <emphasis remap="bf">Relation</emphasis> is <emphasis remap="bf">Less Than</emphasis>,
+<emphasis remap="bf">Less Than or Equal</emphasis>, <emphasis remap="bf">Greater than</emphasis>, or <emphasis remap="bf">Greater than or
+Equal</emphasis>, the term is assumed to be numerical, and a standard regular
+expression is constructed to match the given expression. If
+<emphasis remap="bf">Relation</emphasis> is <emphasis remap="bf">Relevance</emphasis>, the standard natural-language query
+processor is invoked.
+</para>
+
+<para>
+For the <emphasis remap="bf">Truncation</emphasis> attribute, <emphasis remap="bf">No Truncation</emphasis> is the default.
+<emphasis remap="bf">Left Truncation</emphasis> is not supported. <emphasis remap="bf">Process #</emphasis> is supported, as
+is <emphasis remap="bf">Regxp-1</emphasis>. <emphasis remap="bf">Regxp-2</emphasis> enables the fault-tolerant (fuzzy)
+search. As a default, a single error (deletion, insertion,
+replacement) is accepted when terms are matched against the register
+contents.
+</para>
+
+<sect3>
+<title>Regular expressions</title>
+
+<para>
+Each term in a query is interpreted as a regular expression if
+the truncation value is either <emphasis remap="bf">Regxp-1</emphasis> (102) or <emphasis remap="bf">Regxp-2</emphasis> (103).
+Both query types follow the same syntax with the operands:
+<variablelist>
+
+<varlistentry>
+<term>x</term>
+<listitem>
+<para>
+Matches the character <emphasis remap="it">x</emphasis>.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>.</term>
+<listitem>
+<para>
+Matches any character.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term><literal remap="tt">[</literal>..<literal remap="tt">]</literal></term>
+<listitem>
+<para>
+Matches the set of characters specified;
+such as <literal remap="tt">[abc]</literal> or <literal remap="tt">[a-c]</literal>.
+</para>
+</listitem>
+</varlistentry>
+</variablelist>
+and the operators:
+<variablelist>
+
+<varlistentry>
+<term>x*</term>
+<listitem>
+<para>
+Matches <emphasis remap="it">x</emphasis> zero or more times. Priority: high.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>x+</term>
+<listitem>
+<para>
+Matches <emphasis remap="it">x</emphasis> one or more times. Priority: high.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>x?</term>
+<listitem>
+<para>
+Matches <emphasis remap="it">x</emphasis> once or twice. Priority: high.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>xy</term>
+<listitem>
+<para>
+Matches <emphasis remap="it">x</emphasis>, then <emphasis remap="it">y</emphasis>. Priority: medium.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>x|y</term>
+<listitem>
+<para>
+Matches either <emphasis remap="it">x</emphasis> or <emphasis remap="it">y</emphasis>. Priority: low.
+</para>
+</listitem>
+</varlistentry>
+</variablelist>
+The order of evaluation may be changed by using parentheses.
+</para>
+
+<para>
+If the first character of the <emphasis remap="bf">Regxp-2</emphasis> query is a plus character
+(<literal remap="tt">+</literal>) it marks the beginning of a section with non-standard
+specifiers. The next plus character marks the end of the section.
+Currently Zebra only supports one specifier, the error tolerance,
+which consists one digit.
+</para>
+
+<para>
+Since the plus operator is normally a suffix operator the addition to
+the query syntax doesn't violate the syntax for standard regular
+expressions.
+</para>
+
+</sect3>
+
+<sect3>
+<title>Query examples</title>
+
+<para>
+Phrase search for <emphasis remap="bf">information retrieval</emphasis> in the title-register:
+
+<screen>
+ @attr 1=4 "information retrieval"
+</screen>
+
+</para>
+
+<para>
+Ranked search for the same thing:
+
+<screen>
+ @attr 1=4 @attr 2=102 "Information retrieval"
+</screen>
+
+</para>
+
+<para>
+Phrase search with a regular expression:
+
+<screen>
+ @attr 1=4 @attr 5=102 "informat.* retrieval"
+</screen>
+
+</para>
+
+<para>
+Ranked search with a regular expression:
+
+<screen>
+ @attr 1=4 @attr 5=102 @attr 2=102 "informat.* retrieval"
+</screen>
+
+</para>
+
+<para>
+In the GILS schema (<literal remap="tt">gils.abs</literal>), the west-bounding-coordinate is
+indexed as type <literal remap="tt">n</literal>, and is therefore searched by specifying
+<emphasis remap="bf">structure</emphasis>=<emphasis remap="bf">Numeric String</emphasis>.
+To match all those records with west-bounding-coordinate greater
+than -114 we use the following query:
+
+<screen>
+ @attr 4=109 @attr 2=5 @attr gils 1=2038 -114
+</screen>
+
+</para>
+
+</sect3>
+
+</sect2>
+
+<sect2>
+<title>Present</title>
+
+<para>
+The present facility is supported in a standard fashion. The requested
+record syntax is matched against the ones supported by the profile of
+each record retrieved. If no record syntax is given, SUTRS is the
+default. The requested element set name, again, is matched against any
+provided by the relevant record profiles.
+</para>
+
+</sect2>
+
+<sect2>
+<title>Scan</title>
+
+<para>
+The attribute combinations provided with the termListAndStartPoint are
+processed in the same way as operands in a query (see above).
+Currently, only the term and the globalOccurrences are returned with
+the termInfo structure.
+</para>
+
+</sect2>
+
+<sect2>
+<title>Sort</title>
+
+<para>
+Z39.50 specifies three diffent types of sort criterias.
+Of these Zebra supports the attribute specification type in which
+case the use attribute specifies the "Sort register".
+Sort registers are created for those fields that are of type "sort" in
+the default.idx file.
+The corresponding character mapping file in default.idx specifies the
+ordinal of each character used in the actual sort.
+</para>
+
+<para>
+Z39.50 allows the client to specify sorting on one or more input
+result sets and one output result set.
+Zebra supports sorting on one result set only which may or may not
+be the same as the output result set.
+</para>
+
+</sect2>
+
+<sect2>
+<title>Close</title>
+
+<para>
+If a Close PDU is received, the server will respond with a Close PDU
+with reason=FINISHED, no matter which protocol version was negotiated
+during initialization. If the protocol version is 3 or more, the
+server will generate a Close PDU under certain circumstances,
+including a session timeout (60 minutes by default), and certain kinds of
+protocol errors. Once a Close PDU has been sent, the protocol
+association is considered broken, and the transport connection will be
+closed immediately upon receipt of further data, or following a short
+timeout.
+</para>
+
+</sect2>
+
+</sect1>
+
+</chapter>
+
+<chapter id="record-model">
+<title>The Record Model</title>
+
+<para>
+The Zebra system is designed to support a wide range of data management
+applications. The system can be configured to handle virtually any
+kind of structured data. Each record in the system is associated with
+a <emphasis remap="it">record schema</emphasis> which lends context to the data elements of the
+record. Any number of record schema can coexist in the system.
+Although it may be wise to use only a single schema within
+one database, the system poses no such restrictions.
+</para>
+
+<para>
+The record model described in this chapter applies to the fundamental,
+structured
+record type <literal remap="tt">grs</literal> as introduced in
+section <xref linkend="record-types"/>.
+</para>
+
+<para>
+Records pass through three different states during processing in the
+system.
+</para>
+
+<para>
+
+<itemizedlist>
+<listitem>
+
+<para>
+When records are accessed by the system, they are represented
+in their local, or native format. This might be SGML or HTML files,
+News or Mail archives, MARC records. If the system doesn't already
+know how to read the type of data you need to store, you can set up an
+input filter by preparing conversion rules based on regular
+expressions and possibly augmented by a flexible scripting language (Tcl). The input filter
+produces as output an internal representation:
+
+</para>
+</listitem>
+<listitem>
+
+<para>
+When records are processed by the system, they are represented
+in a tree-structure, constructed by tagged data elements hanging off a
+root node. The tagged elements may contain data or yet more tagged
+elements in a recursive structure. The system performs various
+actions on this tree structure (indexing, element selection, schema
+mapping, etc.),
+
+</para>
+</listitem>
+<listitem>
+
+<para>
+Before transmitting records to the client, they are first
+converted from the internal structure to a form suitable for exchange
+over the network - according to the Z39.50 standard.
+</para>
+</listitem>
+
+</itemizedlist>
+
+</para>
+
+<sect1 id="local-representation">
+<title>Local Representation</title>
+
+<para>
+As mentioned earlier, Zebra places few restrictions on the type of
+data that you can index and manage. Generally, whatever the form of
+the data, it is parsed by an input filter specific to that format, and
+turned into an internal structure that Zebra knows how to handle. This
+process takes place whenever the record is accessed - for indexing and
+retrieval.
+</para>
+
+<para>
+The RecordType parameter in the <literal remap="tt">zebra.cfg</literal> file, or the <literal remap="tt">-t</literal>
+option to the indexer tells Zebra how to process input records. Two
+basic types of processing are available - raw text and structured
+data. Raw text is just that, and it is selected by providing the
+argument <emphasis remap="bf">text</emphasis> to Zebra. Structured records are all handled
+internally using the basic mechanisms described in the subsequent
+sections. Zebra can read structured records in many different formats.
+How this is done is governed by additional parameters after the
+"grs" keyboard, separated by "." characters.
+</para>
+
+<para>
+Three basic subtypes to the <emphasis remap="bf">grs</emphasis> type are currently available:
+</para>
+
+<para>
+<variablelist>
+
+<varlistentry>
+<term>grs.sgml</term>
+<listitem>
+<para>
+This is the canonical input format —
+described below. It is a simple SGML-like syntax.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>grs.regx.<emphasis remap="it">filter</emphasis></term>
+<listitem>
+<para>
+This enables a user-supplied input
+filter. The mechanisms of these filters are described below.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>grs.marc.<emphasis remap="it">abstract syntax</emphasis></term>
+<listitem>
+<para>
+This allows Zebra to read
+records in the ISO2709 (MARC) encoding standard. In this case, the
+last paramemeter <emphasis remap="it">abstract syntax</emphasis> names the .abs file (see below)
+which describes the specific MARC structure of the input record as
+well as the indexing rules.
+</para>
+</listitem>
+</varlistentry>
+</variablelist>
+</para>
+
+<sect2>
+<title>Canonical Input Format</title>
+
+<para>
+Although input data can take any form, it is sometimes useful to
+describe the record processing capabilities of the system in terms of
+a single, canonical input format that gives access to the full
+spectrum of structure and flexibility in the system. In Zebra, this
+canonical format is an "SGML-like" syntax.
+</para>
+
+<para>
+To use the canonical format specify <literal remap="tt">grs.sgml</literal> as the record
+type,
+</para>
+
+<para>
+Consider a record describing an information resource (such a record is
+sometimes known as a <emphasis remap="it">locator record</emphasis>). It might contain a field
+describing the distributor of the information resource, which might in
+turn be partitioned into various fields providing details about the
+distributor, like this:
+</para>
+
+<para>
+
+<screen>
+<Distributor>
+ <Name> USGS/WRD </Name>
+ <Organization> USGS/WRD </Organization>
+ <Street-Address>
+ U.S. GEOLOGICAL SURVEY, 505 MARQUETTE, NW
+ </Street-Address>
+ <City> ALBUQUERQUE </City>
+ <State> NM </State>
+ <Zip-Code> 87102 </Zip-Code>
+ <Country> USA </Country>
+ <Telephone> (505) 766-5560 </Telephone>
+</Distributor>
+</screen>
+
+</para>
+
+<para>
+<emphasis remap="it">NOTE: The indentation used above is used to illustrate how Zebra
+interprets the markup. The indentation, in itself, has no
+significance to the parser for the canonical input format, which
+discards superfluous whitespace.</emphasis>
+</para>
+
+<para>
+The keywords surrounded by <...> are <emphasis remap="it">tags</emphasis>, while the
+sections of text in between are the <emphasis remap="it">data elements</emphasis>. A data element
+is characterized by its location in the tree that is made up by the
+nested elements. Each element is terminated by a closing tag -
+beginning with <literal remap="tt"><</literal>/, and containing the same symbolic tag-name as
+the corresponding opening tag. The general closing tag - <literal remap="tt"><</literal>>/ -
+terminates the element started by the last opening tag. The
+structuring of elements is significant. The element <emphasis remap="bf">Telephone</emphasis>,
+for instance, may be indexed and presented to the client differently,
+depending on whether it appears inside the <emphasis remap="bf">Distributor</emphasis> element,
+or some other, structured data element such a <emphasis remap="bf">Supplier</emphasis> element.
+</para>
+
+<sect3>
+<title>Record Root</title>
+
+<para>
+The first tag in a record describes the root node of the tree that
+makes up the total record. In the canonical input format, the root tag
+should contain the name of the schema that lends context to the
+elements of the record (see section
+<xref linkend="internal-representation"/>).
+The following is a GILS record that
+contains only a single element (strictly speaking, that makes it an
+illegal GILS record, since the GILS profile includes several mandatory
+elements - Zebra does not validate the contents of a record against
+the Z39.50 profile, however - it merely attempts to match up elements
+of a local representation with the given schema):
+</para>
+
+<para>
+
+<screen>
+<gils>
+ <title>Zen and the Art of Motorcycle Maintenance</title>
+</gils>
+</screen>
+
+</para>
+
+</sect3>
+
+<sect3>
+<title>Variants</title>
+
+<para>
+Zebra allows you to provide individual data elements in a number of
+<emphasis remap="it">variant forms</emphasis>. Examples of variant forms are textual data
+elements which might appear in different languages, and images which
+may appear in different formats or layouts. The variant system in
+Zebra is
+essentially a representation of the variant mechanism of
+Z39.50-1995.
+</para>
+
+<para>
+The following is an example of a title element which occurs in two
+different languages.
+</para>
+
+<para>
+
+<screen>
+<title>
+ <var lang lang "eng">
+ Zen and the Art of Motorcycle Maintenance</>
+ <var lang lang "dan">
+ Zen og Kunsten at Vedligeholde en Motorcykel</>
+</title>
+</screen>
+
+</para>
+
+<para>
+The syntax of the <emphasis remap="it">variant element</emphasis> is <literal remap="tt"><var class
+type value></literal>. The available values for the <emphasis remap="it">class</emphasis> and
+<emphasis remap="it">type</emphasis> fields are given by the variant set that is associated with the
+current schema (see section <xref linkend="variant-set"/>).
+</para>
+
+<para>
+Variant elements are terminated by the general end-tag </>, by
+the variant end-tag </var>, by the appearance of another variant
+tag with the same <emphasis remap="it">class</emphasis> and <emphasis remap="it">value</emphasis> settings, or by the
+appearance of another, normal tag. In other words, the end-tags for
+the variants used in the example above could have been saved.
+</para>
+
+<para>
+Variant elements can be nested. The element
+</para>
+
+<para>
+
+<screen>
+<title>
+ <var lang lang "eng"><var body iana "text/plain">
+ Zen and the Art of Motorcycle Maintenance
+</title>
+</screen>
+
+</para>
+
+<para>
+Associates two variant components to the variant list for the title
+element.
+</para>
+
+<para>
+Given the nesting rules described above, we could write
+</para>
+
+<para>
+
+<screen>
+<title>
+ <var body iana "text/plain>
+ <var lang lang "eng">
+ Zen and the Art of Motorcycle Maintenance
+ <var lang lang "dan">
+ Zen og Kunsten at Vedligeholde en Motorcykel
+</title>
+</screen>
+
+</para>
+
+<para>
+The title element above comes in two variants. Both have the IANA body
+type "text/plain", but one is in English, and the other in
+Danish. The client, using the element selection mechanism of Z39.50,
+can retrieve information about the available variant forms of data
+elements, or it can select specific variants based on the requirements
+of the end-user.
+</para>
+
+</sect3>
+
+</sect2>
+
+<sect2>
+<title>Input Filters</title>
+
+<para>
+In order to handle general input formats, Zebra allows the
+operator to define filters which read individual records in their native format
+and produce an internal representation that the system can
+work with.
+</para>
+
+<para>
+Input filters are ASCII files, generally with the suffix <literal remap="tt">.flt</literal>.
+The system looks for the files in the directories given in the
+<emphasis remap="bf">profilePath</emphasis> setting in the <literal remap="tt">zebra.cfg</literal> files. The record type
+for the filter is <literal remap="tt">grs.regx.</literal><emphasis remap="it">filter-filename</emphasis>
+(fundamental type <literal remap="tt">grs</literal>, file read type <literal remap="tt">regx</literal>, argument
+<emphasis remap="it">filter-filename</emphasis>).
+</para>
+
+<para>
+Generally, an input filter consists of a sequence of rules, where each
+rule consists of a sequence of expressions, followed by an action. The
+expressions are evaluated against the contents of the input record,
+and the actions normally contribute to the generation of an internal
+representation of the record.
+</para>
+
+<para>
+An expression can be either of the following:
+</para>
+
+<para>
+<variablelist>
+
+<varlistentry>
+<term>INIT</term>
+<listitem>
+<para>
+The action associated with this expression is evaluated
+exactly once in the lifetime of the application, before any records
+are read. It can be used in conjunction with an action that
+initializes tables or other resources that are used in the processing
+of input records.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>BEGIN</term>
+<listitem>
+<para>
+Matches the beginning of the record. It can be used to
+initialize variables, etc. Typically, the <emphasis remap="bf">BEGIN</emphasis> rule is also used
+to establish the root node of the record.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>END</term>
+<listitem>
+<para>
+Matches the end of the record - when all of the contents
+of the record has been processed.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>/pattern/</term>
+<listitem>
+<para>
+Matches a string of characters from the input
+record.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>BODY</term>
+<listitem>
+<para>
+This keyword may only be used between two patterns. It
+matches everything between (not including) those patterns.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>FINISH</term>
+<listitem>
+<para>
+The expression asssociated with this pattern is evaluated
+once, before the application terminates. It can be used to release
+system resources - typically ones allocated in the <emphasis remap="bf">INIT</emphasis> step.
+</para>
+</listitem>
+</varlistentry>
+</variablelist>
+</para>
+
+<para>
+An action is surrounded by curly braces ({...}), and consists of a
+sequence of statements. Statements may be separated by newlines or
+semicolons (;). Within actions, the strings that matched the
+expressions immediately preceding the action can be referred to as
+$0, $1, $2, etc.
+</para>
+
+<para>
+The available statements are:
+</para>
+
+<para>
+<variablelist>
+
+<varlistentry>
+<term>begin <emphasis remap="it">type [parameter ... ]</emphasis></term>
+<listitem>
+<para>
+Begin a new
+data element. The type is one of the following:
+<variablelist>
+
+<varlistentry>
+<term>record</term>
+<listitem>
+<para>
+Begin a new record. The followingparameter should be the
+name of the schema that describes the structure of the record, eg.
+<literal remap="tt">gils</literal> or <literal remap="tt">wais</literal> (see below). The <literal remap="tt">begin record</literal> call should
+precede
+any other use of the <emphasis remap="bf">begin</emphasis> statement.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>element</term>
+<listitem>
+<para>
+Begin a new tagged element. The parameter is the
+name of the tag. If the tag is not matched anywhere in the tagsets
+referenced by the current schema, it is treated as a local string
+tag.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>variant</term>
+<listitem>
+<para>
+Begin a new node in a variant tree. The parameters are
+<emphasis remap="it">class type value</emphasis>.
+</para>
+</listitem>
+</varlistentry>
+</variablelist>
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>data</term>
+<listitem>
+<para>
+Create a data element. The concatenated arguments make
+up the value of the data element. The option <literal remap="tt">-text</literal> signals that
+the layout (whitespace) of the data should be retained for
+transmission. The option <literal remap="tt">-element</literal> <emphasis remap="it">tag</emphasis> wraps the data up in
+the <emphasis remap="it">tag</emphasis>. The use of the <literal remap="tt">-element</literal> option is equivalent to
+preceding the command with a <emphasis remap="bf">begin element</emphasis> command, and following
+it with the <emphasis remap="bf">end</emphasis> command.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>end <emphasis remap="it">[type]</emphasis></term>
+<listitem>
+<para>
+Close a tagged element. If no parameter is given,
+the last element on the stack is terminated. The first parameter, if
+any, is a type name, similar to the <emphasis remap="bf">begin</emphasis> statement. For the
+<emphasis remap="bf">element</emphasis> type, a tag name can be provided to terminate a specific tag.
+</para>
+</listitem>
+</varlistentry>
+</variablelist>
+</para>
+
+<para>
+The following input filter reads a Usenet news file, producing a
+record in the WAIS schema. Note that the body of a news posting is
+separated from the list of headers by a blank line (or rather a
+sequence of two newline characters.
+</para>
+
+<para>
+
+<screen>
+BEGIN { begin record wais }
+
+/^From:/ BODY /$/ { data -element name $1 }
+/^Subject:/ BODY /$/ { data -element title $1 }
+/^Date:/ BODY /$/ { data -element lastModified $1 }
+/\n\n/ BODY END {
+ begin element bodyOfDisplay
+ begin variant body iana "text/plain"
+ data -text $1
+ end record
+ }
+</screen>
+
+</para>
+
+<para>
+If Zebra is compiled with support for Tcl (Tool Command Language)
+enabled, the statements described above are supplemented with a complete
+scripting environment, including control structures (conditional
+expressions and loop constructs), and powerful string manipulation
+mechanisms for modifying the elements of a record. Tcl is a popular
+scripting environment, with several tutorials available both online
+and in hardcopy.
+</para>
+
+<para>
+<emphasis remap="it">NOTE: Tcl support is not currently available, but will be
+included with one of the next alpha or beta releases.</emphasis>
+</para>
+
+<para>
+<emphasis remap="it">NOTE: Variant support is not currently available in the input
+filter, but will be included with one of the next alpha or beta
+releases.</emphasis>
+</para>
+
+</sect2>
+
+</sect1>
+
+<sect1 id="internal-representation">
+<title>Internal Representation</title>
+
+<para>
+When records are manipulated by the system, they're represented in a
+tree-structure, with data elements at the leaf nodes, and tags or
+variant components at the non-leaf nodes. The root-node identifies the
+schema that lends context to the tagging and structuring of the
+record. Imagine a simple record, consisting of a 'title' element and
+an 'author' element:
+</para>
+
+<para>
+
+<screen>
+ TITLE "Zen and the Art of Motorcycle Maintenance"
+ROOT
+ AUTHOR "Robert Pirsig"
+</screen>
+
+</para>
+
+<para>
+A slightly more complex record would have the author element consist
+of two elements, a surname and a first name:
+</para>
+
+<para>
+
+<screen>
+ TITLE "Zen and the Art of Motorcycle Maintenance"
+ROOT
+ FIRST-NAME "Robert"
+ AUTHOR
+ SURNAME "Pirsig"
+</screen>
+
+</para>
+
+<para>
+The root of the record will refer to the record schema that describes
+the structuring of this particular record. The schema defines the
+element tags (TITLE, FIRST-NAME, etc.) that may occur in the record, as
+well as the structuring (SURNAME should appear below AUTHOR, etc.). In
+addition, the schema establishes element set names that are used by
+the client to request a subset of the elements of a given record. The
+schema may also establish rules for converting the record to a
+different schema, by stating, for each element, a mapping to a
+different tag path.
+</para>
+
+<sect2>
+<title>Tagged Elements</title>
+
+<para>
+A data element is characterized by its tag, and its position in the
+structure of the record. For instance, while the tag "telephone
+number" may be used different places in a record, we may need to
+distinguish between these occurrences, both for searching and
+presentation purposes. For instance, while the phone numbers for the
+"customer" and the "service provider" are both
+representatives for the same type of resource (a telephone number), it
+is essential that they be kept separate. The record schema provides
+the structure of the record, and names each data element (defined by
+the sequence of tags - the tag path - by which the element can be
+reached from the root of the record).
+</para>
+
+</sect2>
+
+<sect2>
+<title>Variants</title>
+
+<para>
+The children of a tag node may be either more tag nodes, a data node
+(possibly accompanied by tag nodes),
+or a tree of variant nodes. The children of variant nodes are either
+more variant nodes or a data node (possibly accompanied by more
+variant nodes). Each leaf node, which is normally a
+data node, corresponds to a <emphasis remap="it">variant form</emphasis> of the tagged element
+identified by the tag which parents the variant tree. The following
+title element occurs in two different languages:
+</para>
+
+<para>
+
+<screen>
+ VARIANT LANG=ENG "War and Peace"
+TITLE
+ VARIANT LANG=DAN "Krig og Fred"
+</screen>
+
+</para>
+
+<para>
+Which of the two elements are transmitted to the client by the server
+depends on the specifications provided by the client, if any.
+</para>
+
+<para>
+In practice, each variant node is associated with a triple of class,
+type, value, corresponding to the variant mechanism of Z39.50.
+</para>
+
+</sect2>
+
+<sect2>
+<title>Data Elements</title>
+
+<para>
+Data nodes have no children (they are always leaf nodes in the record
+tree).
+</para>
+
+<para>
+<emphasis remap="it">NOTE: Documentation needs extension here about types of nodes - numerical,
+textual, etc., plus the various types of inclusion notes.</emphasis>
+</para>
+
+</sect2>
+
+</sect1>
+
+<sect1 id="data-model">
+<title>Configuring Your Data Model</title>
+
+<para>
+The following sections describe the configuration files that govern
+the internal management of data records. The system searches for the files
+in the directories specified by the <emphasis remap="bf">profilePath</emphasis> setting in the
+<literal remap="tt">zebra.cfg</literal> file.
+</para>
+
+<sect2>
+<title>The Abstract Syntax</title>
+
+<para>
+The abstract syntax definition (also known as an Abstract Record
+Structure, or ARS) is the focal point of the
+record schema description. For a given schema, the ABS file may state any
+or all of the following:
+</para>
+
+<para>
+
+<itemizedlist>
+<listitem>
+
+<para>
+The object identifier of the Z39.50 schema associated
+with the ARS, so that it can be referred to by the client.
+
+</para>
+</listitem>
+<listitem>
+
+<para>
+The attribute set (which can possibly be a compound of multiple
+sets) which applies in the profile. This is used when indexing and
+searching the records belonging to the given profile.
+
+</para>
+</listitem>
+<listitem>
+
+<para>
+The Tag set (again, this can consist of several different sets).
+This is used when reading the records from a file, to recognize the
+different tags, and when transmitting the record to the client -
+mapping the tags to their numerical representation, if they are
+known.
+
+</para>
+</listitem>
+<listitem>
+
+<para>
+The variant set which is used in the profile. This provides a
+vocabulary for specifying the <emphasis remap="it">forms</emphasis> of data that appear inside
+the records.
+
+</para>
+</listitem>
+<listitem>
+
+<para>
+Element set names, which are a shorthand way for the client to
+ask for a subset of the data elements contained in a record. Element
+set names, in the retrieval module, are mapped to <emphasis remap="it">element
+specifications</emphasis>, which contain information equivalent to the
+<emphasis remap="it">Espec-1</emphasis> syntax of Z39.50.
+
+</para>
+</listitem>
+<listitem>
+
+<para>
+Map tables, which may specify mappings to <emphasis remap="it">other</emphasis> database
+profiles, if desired.
+
+</para>
+</listitem>
+<listitem>
+
+<para>
+Possibly, a set of rules describing the mapping of elements to a
+MARC representation.
+
+</para>
+</listitem>
+<listitem>
+
+<para>
+A list of element descriptions (this is the actual ARS of the
+schema, in Z39.50 terms), which lists the ways in which the various
+tags can be used and organized hierarchically.
+</para>
+</listitem>
+
+</itemizedlist>
+
+</para>
+
+<para>
+Several of the entries above simply refer to other files, which
+describe the given objects.
+</para>
+
+</sect2>
+
+<sect2>
+<title>The Configuration Files</title>
+
+<para>
+This section describes the syntax and use of the various tables which
+are used by the retrieval module.
+</para>
+
+<para>
+The number of different file types may appear daunting at first, but
+each type corresponds fairly clearly to a single aspect of the Z39.50
+retrieval facilities. Further, the average database administrator,
+who is simply reusing an existing profile for which tables already
+exist, shouldn't have to worry too much about the contents of these tables.
+</para>
+
+<para>
+Generally, the files are simple ASCII files, which can be maintained
+using any text editor. Blank lines, and lines beginning with a (#) are
+ignored. Any characters on a line followed by a (#) are also ignored.
+All other
+lines contain <emphasis remap="it">directives</emphasis>, which provide some setting or value
+to the system. Generally, settings are characterized by a single
+keyword, identifying the setting, followed by a number of parameters.
+Some settings are repeatable (r), while others may occur only once in a
+file. Some settings are optional (o), whicle others again are
+mandatory (m).
+</para>
+
+</sect2>
+
+<sect2>
+<title>The Abstract Syntax (.abs) Files</title>
+
+<para>
+The name of this file type is slightly misleading in Z39.50 terms,
+since, apart from the actual abstract syntax of the profile, it also
+includes most of the other definitions that go into a database
+profile.
+</para>
+
+<para>
+When a record in the canonical, SGML-like format is read from a file
+or from the database, the first tag of the file should reference the
+profile that governs the layout of the record. If the first tag of the
+record is, say, <literal remap="tt"><gils></literal>, the system will look for the profile
+definition in the file <literal remap="tt">gils.abs</literal>. Profile definitions are cached,
+so they only have to be read once during the lifespan of the current
+process.
+</para>
+
+<para>
+When writing your own input filters, the <emphasis remap="bf">record-begin</emphasis> command
+introduces the profile, and should always be called first thing when
+introducing a new record.
+</para>
+
+<para>
+The file may contain the following directives:
+</para>
+
+<para>
+<variablelist>
+
+<varlistentry>
+<term>name <emphasis remap="it">symbolic-name</emphasis></term>
+<listitem>
+<para>
+(m) This provides a shorthand name or
+description for the profile. Mostly useful for diagnostic purposes.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>reference <emphasis remap="it">OID-name</emphasis></term>
+<listitem>
+<para>
+(m) The reference name of the OID for
+the profile. The reference names can be found in the <emphasis remap="bf">util</emphasis>
+module of <emphasis remap="bf">YAZ</emphasis>.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>attset <emphasis remap="it">filename</emphasis></term>
+<listitem>
+<para>
+(m) The attribute set that is used for
+indexing and searching records belonging to this profile.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>tagset <emphasis remap="it">filename</emphasis></term>
+<listitem>
+<para>
+(o) The tag set (if any) that describe
+that fields of the records.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>varset <emphasis remap="it">filename</emphasis></term>
+<listitem>
+<para>
+(o) The variant set used in the profile.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>maptab <emphasis remap="it">filename</emphasis></term>
+<listitem>
+<para>
+(o,r) This points to a
+conversion table that might be used if the client asks for the record
+in a different schema from the native one.
+</para>
+</listitem></varlistentry>
+<varlistentry>
+<term>marc <emphasis remap="it">filename</emphasis></term>
+<listitem>
+<para>
+(o) Points to a file containing parameters
+for representing the record contents in the ISO2709 syntax. Read the
+description of the MARC representation facility below.
+</para>
+</listitem></varlistentry>
+<varlistentry>
+<term>esetname <emphasis remap="it">name filename</emphasis></term>
+<listitem>
+<para>
+(o,r) Associates the
+given element set name with an element selection file. If an (@) is
+given in place of the filename, this corresponds to a null mapping for
+the given element set name.
+</para>
+</listitem></varlistentry>
+<varlistentry>
+<term>any <emphasis remap="it">tags</emphasis></term>
+<listitem>
+<para>
+(o) This directive specifies a list of
+attributes which should be appended to the attribute list given for each
+element. The effect is to make every single element in the abstract
+syntax searchable by way of the given attributes. This directive
+provides an efficient way of supporting free-text searching across all
+elements. However, it does increase the size of the index
+significantly. The attributes can be qualified with a structure, as in
+the <emphasis remap="bf">elm</emphasis> directive below.
+</para>
+</listitem></varlistentry>
+<varlistentry>
+<term>elm <emphasis remap="it">path name attributes</emphasis></term>
+<listitem>
+<para>
+(o,r) Adds an element
+to the abstract record syntax of the schema. The <emphasis remap="it">path</emphasis> follows the
+syntax which is suggested by the Z39.50 document - that is, a sequence
+of tags separated by slashes (/). Each tag is given as a
+comma-separated pair of tag type and -value surrounded by parenthesis.
+The <emphasis remap="it">name</emphasis> is the name of the element, and the <emphasis remap="it">attributes</emphasis>
+specifies which attributes to use when indexing the element in a
+comma-separated list. A ! in
+place of the attribute name is equivalent to specifying an attribute
+name identical to the element name. A - in place of the attribute name
+specifies that no indexing is to take place for the given element. The
+attributes can be qualified with <emphasis remap="it">field types</emphasis> to specify which
+character set should govern the indexing procedure for that field. The
+same data element may be indexed into several different fields, using
+different character set definitions. See the section
+<xref linkend="field-structure-and-character-sets"/>.
+The default field type is "w" for
+<emphasis remap="it">word</emphasis>.
+</para>
+</listitem></varlistentry>
+</variablelist>
+</para>
+
+<para>
+<emphasis remap="it">NOTE: The mechanism for controlling indexing is not adequate for
+complex databases, and will probably be moved into a separate
+configuration table eventually.</emphasis>
+</para>
+
+<para>
+The following is an excerpt from the abstract syntax file for the GILS
+profile.
+</para>
+
+<para>
+
+<screen>
+name gils
+reference GILS-schema
+attset gils.att
+tagset gils.tag
+varset var1.var
+
+maptab gils-usmarc.map
+
+# Element set names
+
+esetname VARIANT gils-variant.est # for WAIS-compliance
+esetname B gils-b.est
+esetname G gils-g.est
+esetname F @
+
+elm (1,10) rank -
+elm (1,12) url -
+elm (1,14) localControlNumber Local-number
+elm (1,16) dateOfLastModification Date/time-last-modified
+elm (2,1) title w:!,p:!
+elm (4,1) controlIdentifier Identifier-standard
+elm (2,6) abstract Abstract
+elm (4,51) purpose !
+elm (4,52) originator -
+elm (4,53) accessConstraints !
+elm (4,54) useConstraints !
+elm (4,70) availability -
+elm (4,70)/(4,90) distributor -
+elm (4,70)/(4,90)/(2,7) distributorName !
+elm (4,70)/(4,90)/(2,10 distributorOrganization !
+elm (4,70)/(4,90)/(4,2) distributorStreetAddress !
+elm (4,70)/(4,90)/(4,3) distributorCity !
+</screen>
+
+</para>
+
+</sect2>
+
+<sect2 id="attset-files">
+<title>The Attribute Set (.att) Files</title>
+
+<para>
+This file type describes the <emphasis remap="bf">Use</emphasis> elements of an attribute set.
+It contains the following directives.
+</para>
+
+<para>
+<variablelist>
+
+<varlistentry>
+<term>name <emphasis remap="it">symbolic-name</emphasis></term>
+<listitem>
+<para>
+(m) This provides a shorthand name or
+description for the attribute set. Mostly useful for diagnostic purposes.
+</para>
+</listitem></varlistentry>
+<varlistentry>
+<term>reference <emphasis remap="it">OID-name</emphasis></term>
+<listitem>
+<para>
+(m) The reference name of the OID for
+the attribute set. The reference names can be found in the <emphasis remap="bf">util</emphasis>
+module of <emphasis remap="bf">YAZ</emphasis>.
+</para>
+</listitem></varlistentry>
+<varlistentry>
+<term>ordinal <emphasis remap="it">integer</emphasis></term>
+<listitem>
+<para>
+(m) This value will be used to represent the
+attribute set in the index. Care should be taken that each attribute
+set has a unique ordinal value.
+</para>
+</listitem></varlistentry>
+<varlistentry>
+<term>include <emphasis remap="it">filename</emphasis></term>
+<listitem>
+<para>
+(o,r) This directive is used to
+include another attribute set as a part of the current one. This is
+used when a new attribute set is defined as an extension to another
+set. For instance, many new attribute sets are defined as extensions
+to the <emphasis remap="bf">bib-1</emphasis> set. This is an important feature of the retrieval
+system of Z39.50, as it ensures the highest possible level of
+interoperability, as those access points of your database which are
+derived from the external set (say, bib-1) can be used even by clients
+who are unaware of the new set.
+</para>
+</listitem></varlistentry>
+<varlistentry>
+<term>att <emphasis remap="it">att-value att-name [local-value]</emphasis></term>
+<listitem>
+<para>
+(o,r) This
+repeatable directive introduces a new attribute to the set. The
+attribute value is stored in the index (unless a <emphasis remap="it">local-value</emphasis> is
+given, in which case this is stored). The name is used to refer to the
+attribute from the <emphasis remap="it">abstract syntax</emphasis>.
+</para>
+</listitem></varlistentry>
+</variablelist>
+</para>
+
+<para>
+This is an excerpt from the GILS attribute set definition. Notice how
+the file describing the <emphasis remap="it">bib-1</emphasis> attribute set is referenced.
+</para>
+
+<para>
+
+<screen>
+name gils
+reference GILS-attset
+include bib1.att
+ordinal 2
+
+att 2001 distributorName
+att 2002 indextermsControlled
+att 2003 purpose
+att 2004 accessConstraints
+att 2005 useConstraints
+</screen>
+
+</para>
+
+</sect2>
+
+<sect2>
+<title>The Tag Set (.tag) Files</title>
+
+<para>
+This file type defines the tagset of the profile, possibly by
+referencing other tag sets (most tag sets, for instance, will include
+tagsetG and tagsetM from the Z39.50 specification. The file may
+contain the following directives.
+</para>
+
+<para>
+<variablelist>
+
+<varlistentry>
+<term>name <emphasis remap="it">symbolic-name</emphasis></term>
+<listitem>
+<para>
+(m) This provides a shorthand name or
+description for the tag set. Mostly useful for diagnostic purposes.
+</para>
+</listitem></varlistentry>
+<varlistentry>
+<term>reference <emphasis remap="it">OID-name</emphasis></term>
+<listitem>
+<para>
+(o) The reference name of the OID for
+the tag set. The reference names can be found in the <emphasis remap="bf">util</emphasis>
+module of <emphasis remap="bf">YAZ</emphasis>. The directive is optional, since not all tag sets
+are registered outside of their schema.
+</para>
+</listitem></varlistentry>
+<varlistentry>
+<term>type <emphasis remap="it">integer</emphasis></term>
+<listitem>
+<para>
+(m) The type number of the tagset within the schema
+profile (note: this specification really should belong to the .abs
+file. This will be fixed in a future release).
+</para>
+</listitem></varlistentry>
+<varlistentry>
+<term>include <emphasis remap="it">filename</emphasis></term>
+<listitem>
+<para>
+(o,r) This directive is used
+to include the definitions of other tag sets into the current one.
+</para>
+</listitem></varlistentry>
+<varlistentry>
+<term>tag <emphasis remap="it">number names type</emphasis></term>
+<listitem>
+<para>
+(o,r) Introduces a new
+tag to the set. The <emphasis remap="it">number</emphasis> is the tag number as used in the protocol
+(there is currently no mechanism for specifying string tags at this
+point, but this would be quick work to add). The <emphasis remap="it">names</emphasis> parameter
+is a list of names by which the tag should be recognized in the input
+file format. The names should be separated by slashes (/). The
+<emphasis remap="it">type</emphasis> is th recommended datatype of the tag. It should be one of
+the following:
+
+<itemizedlist>
+<listitem>
+
+<para>
+structured
+</para>
+</listitem>
+<listitem>
+
+<para>
+string
+</para>
+</listitem>
+<listitem>
+
+<para>
+numeric
+</para>
+</listitem>
+<listitem>
+
+<para>
+bool
+</para>
+</listitem>
+<listitem>
+
+<para>
+oid
+</para>
+</listitem>
+<listitem>
+
+<para>
+generalizedtime
+</para>
+</listitem>
+<listitem>
+
+<para>
+intunit
+</para>
+</listitem>
+<listitem>
+
+<para>
+int
+</para>
+</listitem>
+<listitem>
+
+<para>
+octetstring
+</para>
+</listitem>
+<listitem>
+
+<para>
+null
+</para>
+</listitem>
+
+</itemizedlist>
+
+</para>
+</listitem></varlistentry>
+</variablelist>
+</para>
+
+<para>
+The following is an excerpt from the TagsetG definition file.
+</para>
+
+<para>
+
+<screen>
+name tagsetg
+reference TagsetG
+type 2
+
+tag 1 title string
+tag 2 author string
+tag 3 publicationPlace string
+tag 4 publicationDate string
+tag 5 documentId string
+tag 6 abstract string
+tag 7 name string
+tag 8 date generalizedtime
+tag 9 bodyOfDisplay string
+tag 10 organization string
+</screen>
+
+</para>
+
+</sect2>
+
+<sect2 id="variant-set">
+<title>The Variant Set (.var) Files</title>
+
+<para>
+The variant set file is a straightforward representation of the
+variant set definitions associated with the protocol. At present, only
+the <emphasis remap="it">Variant-1</emphasis> set is known.
+</para>
+
+<para>
+These are the directives allowed in the file.
+</para>
+
+<para>
+<variablelist>
+
+<varlistentry>
+<term>name <emphasis remap="it">symbolic-name</emphasis></term>
+<listitem>
+<para>
+(m) This provides a shorthand name or
+description for the variant set. Mostly useful for diagnostic purposes.
+</para>
+</listitem></varlistentry>
+<varlistentry>
+<term>reference <emphasis remap="it">OID-name</emphasis></term>
+<listitem>
+<para>
+(o) The reference name of the OID for
+the variant set, if one is required. The reference names can be found
+in the <emphasis remap="bf">util</emphasis> module of <emphasis remap="bf">YAZ</emphasis>.
+</para>
+</listitem></varlistentry>
+<varlistentry>
+<term>class <emphasis remap="it">integer class-name</emphasis></term>
+<listitem>
+<para>
+(m,r) Introduces a new
+class to the variant set.
+</para>
+</listitem></varlistentry>
+<varlistentry>
+<term>type <emphasis remap="it">integer type-name datatype</emphasis></term>
+<listitem>
+<para>
+(m,r) Addes a
+new type to the current class (the one introduced by the most recent
+<emphasis remap="bf">class</emphasis> directive). The type names belong to the same name space as
+the one used in the tag set definition file.
+</para>
+</listitem></varlistentry>
+</variablelist>
+</para>
+
+<para>
+The following is an excerpt from the file describing the variant set
+<emphasis remap="it">Variant-1</emphasis>.
+</para>
+
+<para>
+
+<screen>
+name variant-1
+reference Variant-1
+
+class 1 variantId
+
+ type 1 variantId octetstring
+
+class 2 body
+
+ type 1 iana string
+ type 2 z39.50 string
+ type 3 other string
+</screen>
+
+</para>
+
+</sect2>
+
+<sect2>
+<title>The Element Set (.est) Files</title>
+
+<para>
+The element set specification files describe a selection of a subset
+of the elements of a database record. The element selection mechanism
+is equivalent to the one supplied by the <emphasis remap="it">Espec-1</emphasis> syntax of the
+Z39.50 specification. In fact, the internal representation of an
+element set specification is identical to the <emphasis remap="it">Espec-1</emphasis> structure,
+and we'll refer you to the description of that structure for most of
+the detailed semantics of the directives below.
+</para>
+
+<para>
+<emphasis remap="it">NOTE: Not all of the Espec-1 functionality has been implemented yet.
+The fields that are mentioned below all work as expected, unless
+otherwise is noted.</emphasis>
+</para>
+
+<para>
+The directives available in the element set file are as follows:
+</para>
+
+<para>
+<variablelist>
+
+<varlistentry>
+<term>defaultVariantSetId <emphasis remap="it">OID-name</emphasis></term>
+<listitem>
+<para>
+(o) If variants are used in
+the following, this should provide the name of the variantset used
+(it's not currently possible to specify a different set in the
+individual variant request). In almost all cases (certainly all
+profiles known to us), the name <literal remap="tt">Variant-1</literal> should be given here.
+</para>
+</listitem></varlistentry>
+<varlistentry>
+<term>defaultVariantRequest <emphasis remap="it">variant-request</emphasis></term>
+<listitem>
+<para>
+(o) This directive
+provides a default variant request for
+use when the individual element requests (see below) do not contain a
+variant request. Variant requests consist of a blank-separated list of
+variant components. A variant compont is a comma-separated,
+parenthesized triple of variant class, type, and value (the two former
+values being represented as integers). The value can currently only be
+entered as a string (this will change to depend on the definition of
+the variant in question). The special value (@) is interpreted as a
+null value, however.
+</para>
+</listitem></varlistentry>
+<varlistentry>
+<term>simpleElement <emphasis remap="it">path ['variant' variant-request]</emphasis></term>
+<listitem>
+<para>
+(o,r) This corresponds to a simple element request in <emphasis remap="it">Espec-1</emphasis>. The
+path consists of a sequence of tag-selectors, where each of these can
+consist of either:
+</para>
+
+<para>
+
+<itemizedlist>
+<listitem>
+
+<para>
+A simple tag, consisting of a comma-separated type-value pair in
+parenthesis, possibly followed by a colon (:) followed by an
+occurrences-specification (see below). The tag-value can be a number
+or a string. If the first character is an apostrophe ('), this forces
+the value to be interpreted as a string, even if it appears to be numerical.
+
+</para>
+</listitem>
+<listitem>
+
+<para>
+A WildThing, represented as a question mark (?), possibly
+followed by a colon (:) followed by an occurrences specification (see
+below).
+
+</para>
+</listitem>
+<listitem>
+
+<para>
+A WildPath, represented as an asterisk (*). Note that the last
+element of the path should not be a wildPath (wildpaths don't work in
+this version).
+</para>
+</listitem>
+
+</itemizedlist>
+
+</para>
+
+<para>
+The occurrences-specification can be either the string <literal remap="tt">all</literal>, the
+string <literal remap="tt">last</literal>, or an explicit value-range. The value-range is
+represented as an integer (the starting point), possibly followed by a
+plus (+) and a second integer (the number of elements, default being
+one).
+</para>
+
+<para>
+The variant-request has the same syntax as the defaultVariantRequest
+above. Note that it may sometimes be useful to give an empty variant
+request, simply to disable the default for a specific set of fields
+(we aren't certain if this is proper <emphasis remap="it">Espec-1</emphasis>, but it works in
+this implementation).
+</para>
+</listitem></varlistentry>
+</variablelist>
+</para>
+
+<para>
+The following is an example of an element specification belonging to
+the GILS profile.
+</para>
+
+<para>
+
+<screen>
+simpleelement (1,10)
+simpleelement (1,12)
+simpleelement (2,1)
+simpleelement (1,14)
+simpleelement (4,1)
+simpleelement (4,52)
+</screen>
+
+</para>
+
+</sect2>
+
+<sect2 id="schema-mapping">
+<title>The Schema Mapping (.map) Files</title>
+
+<para>
+Sometimes, the client might want to receive a database record in
+a schema that differs from the native schema of the record. For
+instance, a client might only know how to process WAIS records, while
+the database record is represented in a more specific schema, such as
+GILS. In this module, a mapping of data to one of the MARC formats is
+also thought of as a schema mapping (mapping the elements of the
+record into fields consistent with the given MARC specification, prior
+to actually converting the data to the ISO2709). This use of the
+object identifier for USMARC as a schema identifier represents an
+overloading of the OID which might not be entirely proper. However,
+it represents the dual role of schema and record syntax which
+is assumed by the MARC family in Z39.50.
+</para>
+
+<para>
+<emphasis remap="it">NOTE: The schema-mapping functions are so far limited to a
+straightforward mapping of elements. This should be extended with
+mechanisms for conversions of the element contents, and conditional
+mappings of elements based on the record contents.</emphasis>
+</para>
+
+<para>
+These are the directives of the schema mapping file format:
+</para>
+
+<para>
+<variablelist>
+
+<varlistentry>
+<term>targetName <emphasis remap="it">name</emphasis></term>
+<listitem>
+<para>
+(m) A symbolic name for the target schema
+of the table. Useful mostly for diagnostic purposes.
+</para>
+</listitem></varlistentry>
+<varlistentry>
+<term>targetRef <emphasis remap="it">OID-name</emphasis></term>
+<listitem>
+<para>
+(m) An OID name for the target schema.
+This is used, for instance, by a server receiving a request to present
+a record in a different schema from the native one. The name, again,
+is found in the <emphasis remap="bf">oid</emphasis> module of <emphasis remap="bf">YAZ</emphasis>.
+</para>
+</listitem></varlistentry>
+<varlistentry>
+<term>map <emphasis remap="it">element-name target-path</emphasis></term>
+<listitem>
+<para>
+(o,r) Adds
+an element mapping rule to the table.
+</para>
+</listitem></varlistentry>
+</variablelist>
+</para>
+
+</sect2>
+
+<sect2>
+<title>The MARC (ISO2709) Representation (.mar) Files</title>
+
+<para>
+This file provides rules for representing a record in the ISO2709
+format. The rules pertain mostly to the values of the constant-length
+header of the record.
+</para>
+
+<para>
+<emphasis remap="it">NOTE: This will be described better. We're in the process of
+re-evaluating and most likely changing the way that MARC records are
+handled by the system.</emphasis>
+</para>
+
+</sect2>
+
+<sect2 id="field-structure-and-character-sets">
+<title>Field Structure and Character Sets
+</title>
+
+<para>
+In order to provide a flexible approach to national character set
+handling, Zebra allows the administrator to configure the set up the
+system to handle any 8-bit character set — including sets that
+require multi-octet diacritics or other multi-octet characters. The
+definition of a character set includes a specification of the
+permissible values, their sort order (this affects the display in the
+SCAN function), and relationships between upper- and lowercase
+characters. Finally, the definition includes the specification of
+space characters for the set.
+</para>
+
+<para>
+The operator can define different character sets for different fields,
+typical examples being standard text fields, numerical fields, and
+special-purpose fields such as WWW-style linkages (URx).
+</para>
+
+<para>
+The field types, and hence character sets, are associated with data
+elements by the .abs files (see above). The file <literal remap="tt">default.idx</literal>
+provides the association between field type codes (as used in the .abs
+files) and the character map files (with the .chr suffix). The format
+of the .idx file is as follows
+</para>
+
+<para>
+<variablelist>
+
+<varlistentry>
+<term>index <emphasis remap="it">field type code</emphasis></term>
+<listitem>
+<para>
+This directive introduces a new
+search index code. The argument is a one-character code to be used in the
+.abs files to select this particular index type. An index, roughly,
+corresponds to a particular structure attribute during search. Refer
+to section <xref linkend="search"/>.
+</para>
+</listitem></varlistentry>
+<varlistentry>
+<term>sort <emphasis remap="it">field code type</emphasis></term>
+<listitem>
+<para>
+This directive introduces a
+sort index. The argument is a one-character code to be used in the
+.abs fie to select this particular index type. The corresponding
+use attribute must be used in the sort request to refer to this
+particular sort index. The corresponding character map (see below)
+is used in the sort process.
+</para>
+</listitem></varlistentry>
+<varlistentry>
+<term>completeness <emphasis remap="it">boolean</emphasis></term>
+<listitem>
+<para>
+This directive enables or disables
+complete field indexing. The value of the <emphasis remap="it">boolean</emphasis> should be 0
+(disable) or 1. If completeness is enabled, the index entry will
+contain the complete contents of the field (up to a limit), with words
+(non-space characters) separated by single space characters
+(normalized to " " on display). When completeness is
+disabled, each word is indexed as a separate entry. Complete subfield
+indexing is most useful for fields which are typically browsed (eg.
+titles, authors, or subjects), or instances where a match on a
+complete subfield is essential (eg. exact title searching). For fields
+where completeness is disabled, the search engine will interpret a
+search containing space characters as a word proximity search.
+</para>
+</listitem></varlistentry>
+<varlistentry>
+<term>charmap <emphasis remap="it">filename</emphasis></term>
+<listitem>
+<para>
+This is the filename of the character
+map to be used for this index for field type.
+</para>
+</listitem></varlistentry>
+</variablelist>
+</para>
+
+<para>
+The contents of the character map files are structured as follows:
+</para>
+
+<para>
+<variablelist>
+
+<varlistentry>
+<term>lowercase <emphasis remap="it">value-set</emphasis></term>
+<listitem>
+<para>
+This directive introduces the basic
+value set of the field type. The format is an ordered list (without
+spaces) of the characters which may occur in "words" of
+the given type. The order of the entries in the list determines the
+sort order of the index. In addition to single characters, the
+following combinations are legal:
+</para>
+
+<para>
+
+<itemizedlist>
+<listitem>
+
+<para>
+Backslashes may be used to introduce three-digit octal, or
+two-digit hex representations of single characters (preceded by <literal remap="tt">x</literal>).
+In addition, the combinations
+\\, \\r, \\n, \\t, \\s (space — remember that real space-characters
+may ot occur in the value definition), and \\ are recognised,
+with their usual interpretation.
+
+</para>
+</listitem>
+<listitem>
+
+<para>
+Curly braces {} may be used to enclose ranges of single
+characters (possibly using the escape convention described in the
+preceding point), eg. {a-z} to entroduce the standard range of ASCII
+characters. Note that the interpretation of such a range depends on
+the concrete representation in your local, physical character set.
+
+</para>
+</listitem>
+<listitem>
+
+<para>
+paranthesises () may be used to enclose multi-byte characters -
+eg. diacritics or special national combinations (eg. Spanish
+"ll"). When found in the input stream (or a search term),
+these characters are viewed and sorted as a single character, with a
+sorting value depending on the position of the group in the value
+statement.
+</para>
+</listitem>
+
+</itemizedlist>
+
+</para>
+</listitem></varlistentry>
+<varlistentry>
+<term>uppercase <emphasis remap="it">value-set</emphasis></term>
+<listitem>
+<para>
+This directive introduces the
+upper-case equivalencis to the value set (if any). The number and
+order of the entries in the list should be the same as in the
+<literal remap="tt">lowercase</literal> directive.
+</para>
+</listitem></varlistentry>
+<varlistentry>
+<term>space <emphasis remap="it">value-set</emphasis></term>
+<listitem>
+<para>
+This directive introduces the character
+which separate words in the input stream. Depending on the
+completeness mode of the field in question, these characters either
+terminate an index entry, or delimit individual "words" in
+the input stream. The order of the elements is not significant —
+otherwise the representation is the same as for the <literal remap="tt">upercase</literal> and
+<literal remap="tt">lowercase</literal> directives.
+</para>
+</listitem></varlistentry>
+<varlistentry>
+<term>map <emphasis remap="it">value-set</emphasis> <emphasis remap="it">target</emphasis></term>
+<listitem>
+<para>
+This directive introduces a
+mapping between each of the members of the value-set on the left to
+the character on the right. The character on the right must occur in
+the value set (the <literal remap="tt">lowercase</literal> directive) of the character set, but
+it may be a paranthesis-enclosed multi-octet character. This directive
+may be used to map diacritics to their base characters, or to map
+HTML-style character-representations to their natural form, etc.
+</para>
+</listitem></varlistentry>
+</variablelist>
+</para>
+
+</sect2>
+
+</sect1>
+
+<sect1 id="formats">
+<title>Exchange Formats</title>
+
+<para>
+Converting records from the internal structure to en exchange format
+is largely an automatic process. Currently, the following exchange
+formats are supported:
+</para>
+
+<para>
+
+<itemizedlist>
+<listitem>
+
+<para>
+GRS-1. The internal representation is based on GRS-1, so the
+conversion here is straightforward. The system will create
+applied variant and supported variant lists as required, if a record
+contains variant information.
+
+</para>
+</listitem>
+<listitem>
+
+<para>
+SUTRS. Again, the mapping is fairly straighforward. Indentation
+is used to show the hierarchical structure of the record. All
+"GRS" type records support both the GRS-1 and SUTRS
+representations.
+
+</para>
+</listitem>
+<listitem>
+
+<para>
+ISO2709-based formats (USMARC, etc.). Only records with a
+two-level structure (corresponding to fields and subfields) can be
+directly mapped to ISO2709. For records with a different structuring
+(eg., GILS), the representation in a structure like USMARC involves a
+schema-mapping (see section <xref linkend="schema-mapping"/>), to an
+"implied" USMARC schema (implied,
+because there is no formal schema which specifies the use of the
+USMARC fields outside of ISO2709). The resultant, two-level record is
+then mapped directly from the internal representation to ISO2709. See
+the GILS schema definition files for a detailed example of this
+approach.
+
+</para>
+</listitem>
+<listitem>
+
+<para>
+Explain. This representation is only available for records
+belonging to the Explain schema.
+
+</para>
+</listitem>
+<listitem>
+
+<para>
+Summary. This ASN-1 based structure is only available for records
+belonging to the Summary schema - or schema which provide a mapping
+to this schema (see the description of the schema mapping facility
+above).
+
+</para>
+</listitem>
+<listitem>
+
+<para>
+SOIF. Support for this syntax is experimental, and is currently
+keyed to a private Index Data OID (1.2.840.10003.5.1000.81.2). All
+abstract syntaxes can be mapped to the SOIF format, although nested
+elements are represented by concatenation of the tag names at each
+level.
+
+</para>
+</listitem>
+
+</itemizedlist>
+
+</para>
+
+</sect1>
+
+</chapter>
projects / idzebra-moved-to-github.git / commitdiff