+Note that Zebra does not verify that the amount of space specified is
+actually available on the directory (file system) specified - it is
+your responsibility to ensure that enough space is available, and that
+other applications do not use the free space. In a large production system,
+it is recommended that you allocate one or more filesystem exclusively
+to the Zebra register files.
+
+<sect1>Safe Updating - Using Shadow Registers<label id="shadow-registers">
+
+<sect2>Description
+
+<p>
+The Zebra server supports updating of the index structures. That is,
+you can add records to databases managed by Zebra without rebuilding
+the entire index. Since this process involves modifying structured
+files with various references between blocks of data in the files, the
+update process is inherently sensitive to system crashes, or to
+process interruptions: Anything but a successfully completed update
+process will leave the register files in an unknown state, and you
+will essentially have no recourse but to re-index everything, or to
+restore the register files from a backup medium. Further, while the
+update process is active, users cannot be allowed to access the
+system, as the contents of the register files may change unpredictably.
+
+You can solve these problems by enabling the shadow register system in
+Zebra. During the updating procedure, <tt/zebraidx/ will temporarily
+write changes to the involved files in a set of &dquot;shadow
+files&dquot;, without modifying the files that are accessed by the
+active server processes. If the update procedure is interrupted by a
+system crash or a signal, you simply repeat the procedure - the
+register files have not been changed or damaged, and the partially
+written shadow files are automatically deleted before the new updating
+procedure commences.
+
+At the end of the updating procedure (or in a separate operation, if
+you so desire), the system enters a &dquot;commit mode&dquot;. First,
+any active server processes are forced to access those blocks that
+have been changed from the shadow files rather than from the main
+register files; the unmodified blocks are still accessed at their
+normal location (the shadow files are not a complete copy of the
+register files - they only contain those parts that have actually been
+modified). If the process is interrupted at any point during the
+commit process, the server processes will continue to access the
+shadow files until you can repeat the commit procedure and complete
+the writing of data to the main register files. You can perform
+multiple update operations to the registers before you commit the
+changes to the system files, or you can execute the commit operation
+at the end of each update operation. When the commit phase has
+completed successfully, any running server processes are instructed to
+switch their operations to the new, operational register, and the
+temporary shadow files are deleted.
+
+<sect2>How to Use Shadow Register Files
+
+<p>
+The first step is to allocate space on your system for the shadow
+files. You do this by adding a <tt/shadow/ entry to the <tt/zebra.cfg/
+file. The syntax of the <tt/shadow/ entry is exactly the same as for
+the <tt/register/ entry (see section <ref name="Register Location"
+id="register-location">). The location of the shadow area should be
+<it/different/ from the location of the main register area (if you
+have specified one - remember that the default register area is the
+working directory of the server and indexing processes).
+
+The following excerpt from a <tt/zebra.cfg/ file shows one example of
+a setup that configures both the main register location and the shadow
+file area. Note that two directories or partitions have been set aside
+for the shadow file area. You can specify any number of directories
+for each of the file areas.
+
+<tscreen><verb>
+register: /d1:500M
+
+shadow: /scratch1:100M /scratch2:200M
+</verb></tscreen>
+
+When shadow files are enabled, an extra command is available at the
+<tt/zebraidx/ command line. In order to make changes to the system
+take effect for the users, you'll have to submit a
+&dquot;commit&dquot; command after a (sequence of) update
+operation(s). You can ask the indexer to commit the changes
+immediately after the update operation:
+
+<tscreen><verb>
+$ zebraidx update /d1/records update /d2/more-records commit
+</verb></tscreen>
+
+Or you can execute multiple updates before committing the changes:
+
+<tscreen><verb>
+$ zebraidx -g books update /d1/records update /d2/more-records
+$ zebraidx -g fun update /d3/fun-records
+$ zebraidx commit
+</verb></tscreen>
+
+If one of the update operations above had been interrupted, the commit
+operation on the last line would fail: <tt/zebraidx/ will not let you
+commit changes that would destroy the running register. You'll have to
+rerun all of the update operations since your last commit operation,
+before you can commit the new changes.
+
+Similarly, if the commit operation fails, <tt/zebraidx/ will not let
+you start a new update operation before you have successfully repeated
+the commit operation. The server processes will keep accessing the
+shadow files rather than the (possibly damaged) blocks of the main
+register files until the commit operation has successfully completed.
+
+You should be aware that update operations may take slightly longer
+when the shadow register system is enabled, since more file access
+operations are involved. Further, while the disk space required for
+the shadow register data is modest for a small update operation, you
+may prefer to disable the system if you are adding a very large number
+of records to an already very large database (we use the terms
+<it/large/ and <it/modest/ very loosely here, since every
+application's perception of size is different). To update the system
+without the use of the the shadow files, simply run <tt/zebraidx/ with
+the <tt/-n/ option (note that you do not have to execute the
+<bf/commit/ command of <tt/zebraidx/ when you temporarily disable the
+use of the shadow registers in this fashion. Note also that, just as
+when the shadow registers are not enabled, server processes will be
+barred from accessing the main register while the update procedure
+takes place.
+
+<sect>Running the Maintenance Interface (zebraidx)
+
+<p>
+The following is a complete reference to the command line interface to
+the <tt/zebraidx/ application.
+
+<bf/Syntax/
+<tscreen><verb>
+$ zebraidx [options] command [directory] ...
+</verb></tscreen>
+<bf/Options/
+<descrip>
+<tag>-t <it/type/</tag>Update all files as <it/type/. Currently, the
+types supported are <tt/text/ and <tt/grs/<it/.filter/. If no
+<it/filter/ is provided for the GRS (General Record Structure) type,
+the canonical input format is assumed (see section <ref
+id="local-representation" name="Local Representation">). Generally, it
+is probably advisable to specify the record types in the
+<tt/zebra.cfg/ file (see section <ref id="record-types" name="Record Types">).
+
+<tag>-c <it/config-file/</tag>Read the configuration file
+<it/config-file/ instead of <tt/zebra.cfg/.
+
+<tag>-g <it/group/</tag>Update the files according to the group
+settings for <it/group/ (see section <ref id="configuration-file"
+name="The Zebra Configuration File">).
+
+<tag>-d <it/database/</tag>The records located should be associated
+with the database name <it/database/ for access through the Z39.50
+server.
+
+<tag>-d <it/mbytes/</tag>Use <it/mbytes/ of megabytes before flushing
+keys to background storage. This setting affects performance when
+updating large databases.
+
+<tag>-n</tag>Disable the use of shadow registers for this operation
+(see section <ref id="shadow-registers" name="Robust Updating - Using
+Shadow Registers">).
+
+<tag>-v <it/level/</tag>Set the log level to <it/level/. <it/level/
+should be one of <tt/none/, <tt/debug/, and <tt/all/.
+
+</descrip>
+
+<bf/Commands/
+<descrip>
+<tag>Update <it/directory/</tag>Update the register with the files
+contained in <it/directory/. If no directory is provided, a list of
+files is read from <tt/stdin/. See section <ref
+id="administrating" name="Administrating Zebra">.
+
+<tag>Delete <it/directory/</tag>Remove the records corresponding to
+the files found under <it/directory/ from the register.
+
+<tag/Commit/Write the changes resulting from the last <bf/update/
+commands to the register. This command is only available if the use of
+shadow register files is enabled (see section <ref
+id="shadow-registers" name="Robust Updating - Using Shadow
+Registers">).
+
+</descrip>
+
+<sect>Running the Z39.50 Server (zebrasrv)
+
+<p>
+<bf/Syntax/
+<tscreen><verb>
+zebrasrv [options] [listener-address ...]
+</verb></tscreen>
+
+<bf/Options/
+<descrip>
+<tag>-a <it/APDU file/</tag> Specify a file for dumping PDUs (for diagnostic purposes).
+The special name &dquot;-&dquot; sends output to <tt/stderr/.
+
+<tag>-c <it/config-file/</tag> Read configuration information from <it/config-file/. The default configuration is <tt/./zebra.cfg/.
+
+<tag/-S/Don't fork on connection requests. This can be useful for
+symbolic-level debugging. The server can only accept a single
+connection in this mode.
+
+<tag/-s/Use the SR protocol.
+
+<tag/-z/Use the Z39.50 protocol (default). These two options complement
+eachother. You can use both multiple times on the same command
+line, between listener-specifications (see below). This way, you
+can set up the server to listen for connections in both protocols
+concurrently, on different local ports.
+
+<tag>-l <it/logfile/</tag>Specify an output file for the diagnostic
+messages. The default is to write this information to <tt/stderr/.
+
+<tag>-v <it/log-level/</tag>The log level. Use a comma-separated list of members of the set
+{fatal,debug,warn,log,all,none}.
+
+<tag>-u <it/username/</tag>Set user ID. Sets the real UID of the server process to that of the
+given <it/username/. It's useful if you aren't comfortable with having the
+server run as root, but you need to start it as such to bind a
+privileged port.
+</descrip>
+
+A <it/listener-address/ consists of a transport mode followed by a
+colon (:) followed by a listener address. The transport mode is
+either <tt/osi/ or <tt/tcp/.
+
+For TCP, an address has the form
+
+<tscreen><verb>
+hostname | IP-number [: portnumber]
+</verb></tscreen>
+
+The port number defaults to 210 (standard Z39.50 port).
+
+For OSI (only available if the server is compiled with XTI/mOSI
+support enabled), the address form is
+
+<tscreen><verb>
+[t-selector /] hostname | IP-number [: portnumber]
+</verb></tscreen>
+
+The transport selector is given as a string of hex digits (with an even
+number of digits). The default port number is 102 (RFC1006 port).
+
+Examples
+
+<tscreen>
+<verb>
+tcp:dranet.dra.com
+
+osi:0402/dbserver.osiworld.com:3000
+</verb>
+</tscreen>
+
+In both cases, the special hostname &dquot;@&dquot; is mapped to
+the address INADDR_ANY, which causes the server to listen on any local
+interface. To start the server listening on the registered ports for
+Z39.50 and SR over OSI/RFC1006, and to drop root privileges once the
+ports are bound, execute the server like this (from a root shell):
+
+<tscreen><verb>
+zebrasrv -u daemon tcp:@ -s osi:@
+</verb></tscreen>
+
+You can replace <tt/daemon/ with another user, eg. your own account, or
+a dedicated IR server account.
+
+The default behavior for <tt/zebrasrv/ is to establish a single TCP/IP
+listener, for the Z39.50 protocol, on port 9999.
+
+<sect>The Record Model
+
+<p>
+The Zebra system is designed to span a wide range of data management
+applications. The system can be configured to handle virtually any
+kind of structured data. Each record in the system is associated with
+a <it/record schema/ which lends context to the data elements of the
+record. Any number of record schema can coexist in the system.
+Although it may be wise to use only a single schema within
+one database, the system poses no such restrictions.
+
+Records pass through three different states during processing in the
+system.
+
+<itemize>
+<item>When records are first entered into the system, they are represented
+in their local, or native format. This might be SGML or HTML files,
+News or Mail archives, MARC records. If the system doesn't already
+know how to read the type of data you need to store, you can set up an
+input filter by preparing conversion rules based on regular
+expressions and a flexible scripting language (Tcl). The input filter
+produces as output an internal representation:
+
+<item>When records are processed by the system, they are represented
+in a tree-structure, constructed by tagged data elements hanging off a
+root node. The tagged elements may contain data or yet more tagged
+elements in a recursive structure. The system performs various
+actions on this tree structure (indexing, element selection, schema
+mapping, etc.),
+
+<item>Before transmitting records to the client, they are first
+converted from the internal structure to a form suitable for exchange
+over the network - according to the Z39.50 standard.
+</itemize>
+
+<sect1>Local Representation<label id="local-representation">
+
+<p>
+As mentioned earlier, Zebra places few restrictions on the type of
+data that you can index and manage. Generally, whatever the form of
+the data, it is parsed by an input filter specific to that format, and
+turned into an internal structure that Zebra knows how to handle. This
+process takes place whenever the record is accessed - for indexing and
+retrieval.
+
+<sect2>Canonical Input Format
+
+<p>
+Although input data can take any form, it is sometimes useful to
+describe the record processing capabilities of the system in terms of
+a single, canonical input format that gives access to the full
+spectrum of structure and flexibility in the system. In Zebra, this
+canonical format is an &dquot;SGML-like&dquot; syntax.
+
+Consider a record describing an information resource (such a record is
+sometimes known as a <it/locator record/). It might contain a field
+describing the distributor of the information resource, which might in
+turn be partitioned into various fields providing details about the
+distributor, like this:
+
+<tscreen><verb>
+<Distributor>
+ <Name> USGS/WRD &etago;Name>
+ <Organization> USGS/WRD &etago;Organization>
+ <Street-Address>
+ U.S. GEOLOGICAL SURVEY, 505 MARQUETTE, NW
+ &etago;Street-Address>
+ <City> ALBUQUERQUE &etago;City>
+ <State> NM &etago;State>
+ <Zip-Code> 87102 &etago;Zip-Code>
+ <Country> USA &etago;Country>
+ <Telephone> (505) 766-5560 &etago;Telephone>
+&etago;Distributor>
+</verb></tscreen>
+
+<it>NOTE: The indentation used above is used to illustrate how Zebra
+interprets the expression. The indentation, in itself, has no
+significance to the parser for the canonical input format, which
+ignores all whitespace.</it>
+
+The keywords surrounded by <...> are <it/tags/, while the
+sections of text in between are the <it/data elements/. A data element
+is characterized by its location in the tree that is made up by the
+nested elements. Each element is terminated by a closing tag -
+beginning with &etago;, and containing the same symbolic tag-name as
+the corresponding opening tag. The general closing tag - &etago;> -
+terminates the element started by the last opening tag. The
+structuring of elements is significant. The element <bf/Telephone/,
+for instance, may be indexed and presented to the client differently,
+depending on whether it appears inside the <bf/Distributor/ element,
+or some other data element.
+
+<sect3>Record Root
+
+<p>
+The first tag in a record describes the root node of the tree that
+makes up the total record. In the canonical input format, the root tag
+should contain the name of the schema that lends context to the
+elements of the record (see section <ref id="internal-representation"
+name="Internal Representation">). The following is a GILS record that
+contains only a single element (strictly speaking, that makes it an
+illegal GILS record, since the GILS profile includes several mandatory
+elements - Zebra does not validate the contents of a record against
+the Z39.50 profile, however):
+
+<tscreen><verb>
+<gils>
+ <title>Zen and the Art of Motorcycle Maintenance&etago;title>
+&etago;gils>
+</verb></tscreen>
+
+<sect3>Variants
+
+<p>
+Zebra allows you to provide individual data elements in a number of
+<it/variant forms/. Examples of variant forms are textual data
+elements which might appear in different languages, and images which
+may appear in different formats or layouts. The variant system is
+essentially a clean representation of the variant mechanism of
+Z39.50-1995.
+
+The following is an example of a title element which occurs in two
+different languages.
+
+<tscreen><verb>
+<title>
+ <var lang lang "eng">
+ Zen and the Art of Motorcycle Maintenance&etago;>
+ <var lang lang "dan">
+ Zen og Kunsten at Vedligeholde en Motorcykel&etago;>
+&etago;title>
+</verb></tscreen>
+
+The syntax of the <it/variant element/ is <tt><<bf/var/ <it/class
+type value/></tt>. The available values for the <it/class/ and
+<it/type/ fields are given by the variant set that is associated with the
+current schema (see section <ref id="variant-set" name="Variant Set
+File">).
+
+Variant elements are terminated by the general end-tag &etago;>, by
+the variant end-tag &etago;var>, by the appearance of another variant
+tag with the same <it/class/ and <it/value/ settings, or by the
+appearance of another, normal tag. In other words, the end-tags for
+the variants used in the example above could have been saved.
+
+Variant elements can be nested. The element
+
+<tscreen><verb>
+<title>
+ <var lang lang "eng"><var body iana "text/plain">
+ Zen and the Art of Motorcycle Maintenance
+&etago;title>
+</verb></tscreen>
+
+Associates two variant components to the variant list for the title
+element. Given the nesting rules described above, we could write
+
+<tscreen><verb>
+<title>
+ <var body iana "text/plain>
+ <var lang lang "eng">
+ Zen and the Art of Motorcycle Maintenance
+ <var lang lang "dan">
+ Zen og Kunsten at Vedligeholde en Motorcykel
+&etago;title>
+</verb></tscreen>
+
+The title element above comes in two variants. Both have the IANA body
+type &dquot;text/plain&dquot;, but one is in English, and the other in
+Danish.
+
+<sect2>Input Filters
+
+<p>
+In order to handle general, text-based input formats, Zebra allows the
+operator to specify filters which read individual records in their native format
+and produce an internal representation that the system can
+work with.
+
+Input filters are ASCII files, generally with the suffix <tt/.flt/.
+The system looks for the files in the directories given in the
+<bf/profilePath/ setting in the <tt/zebra.cfg/ file.
+
+Generally, an input filter consists of a sequence of rules, where each
+rule consists of a sequence of expressions, followed by an action. The
+expressions are evaluated against the contents of the input record,
+and the actions normally contribute to the generation of an internal
+representation of the record.
+
+An expression can be either of the following:
+
+<descrip>
+<tag/INIT/The action associated with this expression is evaluated
+exactly once in the lifetime of the application, before any records
+are read. It can be used in conjunction with an action that
+initializes tables or other resources that are used in the processing
+of input records.
+
+<tag/BEGIN/Matches the beginning of the record. It can be used to
+initialize variables, etc. Typically, the <bf/BEGIN/ rule is also used
+to establish the root node of the record.
+
+<tag/END/Matches the end of the record - when all of the contents
+of the record has been processed.
+
+<tag>/pattern/</tag>Matches a string of characters from the input
+record.
+
+<tag/BODY/This keyword may only be used between two patterns. It
+matches everything between (not including) those patterns.
+
+<tag/FINISH/THe expression asssociated with this pattern is evaluated
+once, before the application terminates. It can be used to release
+system resources - typically ones allocated in the <bf/INIT/ step.
+
+</descrip>
+
+An action is surrounded by curly braces ({...}), and consists of a
+sequence of statements. Statements may be separated by newlines or
+semicolons (;). Within actions, the strings that matched the
+expressions immediately preceding the action can be referred to as
+$0, $1, $2, etc.
+
+The available statements are:
+
+<descrip>
+
+<tag>begin <it/type [parameter ... ]/</tag>Begin a new
+data element. The type is one of the following:
+<descrip>
+<tag/record/Begin a new record. The parameter should be the
+name of the schema that describes the structure of the record, eg.
+<tt/gils/ or <tt/wais/. The <tt/begin record/ call should come before
+any other call to <bf/begin/.
+
+<tag/element/Begin a new tagged element. The parameter is the
+name of the tag. If the tag is not matched anywhere in the tagsets
+referenced by the current schema, it is treated as a local string
+tag.
+
+<tag/variant/Begin a new node in a variant tree. The parameters are
+<it/class type value/.
+
+</descrip>
+
+<tag/data/Create a data element. The concatenated arguments make
+up the value of the data element. The option <tt/-text/ signals that
+the layout (whitespace) of the data should be retained for
+transmission. The option <tt/-element/ <it/tag/ wraps the data up in
+the <it/tag/. The use of the <tt/-element/ option is equivalent to
+preceding the command with a <bf/begin element/ command, and following
+it with the <bf/end/ command.
+
+<tag>end <it/[type]/</tag>Close a tagged element. If no parameter is given,
+the last element on the stack is terminated. The first parameter, if
+any, is a type name, similar to the <bf/begin/ statement. For the
+<bf/element/ type, a tag name can be provided to terminate a specific tag.
+
+</descrip>
+
+The following input filter reads a Usenet news file, producing a
+record in the WAIS schema. Note that the body of the news posting is
+separated from the list of headers by a blank line (or rather a
+sequence of two newline characters.
+
+<tscreen><verb>
+BEGIN { begin record wais }
+
+/^From:/ BODY /$/ { data -element name $1 }
+/^Subject:/ BODY /$/ { data -element title $1 }
+/^Date:/ BODY /$/ { data -element lastModified $1 }
+/\n\n/ BODY END {
+ begin element bodyOfDisplay
+ begin variant body iana "text/plain"
+ data -text $1
+ end record
+ }
+</verb></tscreen>
+
+If Zebra is compiled with support for Tcl (Tool Command Language)
+enabled, the statements described above are supplemented with a complete
+scripting environment, including control structures (conditional
+expressions and loop constructs), and powerful string manipulation
+mechanisms for modifying the elements of a record. Tcl is a popular
+scripting environment, with several tutorials available both online
+and in hardcopy.
+
+<it>NOTE: Tcl support is not currently available, but will be
+included with the next release.</it>
+
+<it>NOTE: Variant support is not currently available in the input filter, but will be included with the next release.</it>
+
+<sect1>Internal Representation<label id="internal-representation">
+
+<p>
+When records are manipulated by the system, they're represented in a
+tree-structure, with data elements at the leaf nodes, and tags or
+variant components at the non-leaf nodes. The root-node identifies the
+schema that lends context to the tagging and structuring of the
+record. Imagine a simple record, consisting of a 'title' element and
+an 'author' element:
+
+<tscreen><verb>
+ TITLE "Zen and the Art of Motorcycle Maintenance"
+ROOT
+ AUTHOR "Robert Pirsig"
+</verb></tscreen>
+
+A slightly more complex record would have the author element consist
+of two elements, a surname and a first name:
+
+<tscreen><verb>
+ TITLE "Zen and the Art of Motorcycle Maintenance"
+ROOT
+ FIRST-NAME "Robert"
+ AUTHOR
+ SURNAME "Pirsig"
+</verb></tscreen>
+
+The root of the record will refer to the record schema that describes
+the structuring of this particular record. The schema defines the
+element tags (TITLE, FIRST-NAME, etc.) that occur in the record, as
+well as the structuring (SURNAME should appear below AUTHOR, etc.). In
+addition, the schema establishes element set names that are used by
+the client to request a subset of the elements of a given record. The
+schema may also establish rules for converting the record to a
+different schema, by stating, for each element, a mapping to a
+different tagging.
+
+<sect2>Tagged Elements
+
+<p>
+A data element is characterized by its tag, and its position in the
+structure of the record. For instance, while the tag &dquot;telephone
+number&dquot; may be used different places in a record, we may need to
+distinguish between these occurrences, both for searching and
+presentation purposes. For instance, while the phone numbers for the
+&dquot;customer&dquot; and the &dquot;service provider&dquot; are both
+representatives for the same type of resource (a telephone number), it
+is essential that they be kept separate. The record schema provides
+the structure of the record, and names each data element (defined by
+the sequence of tags - the tag path - by which the element can be
+reached from the root of the record).
+
+<sect2>Variants
+
+<p>
+The children of a tag node may be either more tag nodes, a data node,
+or a tree of variant nodes. The children of variant nodes are either
+more variant nodes or data nodes. Each leaf node, which is normally a
+data node, corresponds to a <it/variant form/ or the tagged element
+identified by the tag which parents the variant tree. The following
+title element occurs in two different languages:
+
+<tscreen><verb>
+ VARIANT LANG=ENG "War and Peace"
+TITLE
+ VARIANT LANG=DAN "Krig og Fred"
+</verb></tscreen>
+
+Which of the two elements are transmitted to the client by the server
+depends on the specifications provided by the client, if any.
+
+In practice, each variant node is associated with a triple of class,
+type, value, corresponding to the variant mechanism of Z39.50.
+
+<sect2>Data Elements