2 <!-- $Id: recordmodel-grs.xml,v 1.9 2007-05-24 13:44:09 adam Exp $ -->
3 <title>&acro.grs1; Record Model and Filter Modules</title>
7 The functionality of this record model has been improved and
8 replaced by the DOM &acro.xml; record model. See
9 <xref linkend="record-model-domxml"/>.
14 The record model described in this chapter applies to the fundamental,
16 record type <literal>grs</literal>, introduced in
17 <xref linkend="componentmodulesgrs"/>.
21 <section id="grs-filters">
22 <title>&acro.grs1; Record Filters</title>
24 Many basic subtypes of the <emphasis>grs</emphasis> type are
31 <term><literal>grs.sgml</literal></term>
34 This is the canonical input format
35 described <xref linkend="grs-canonical-format"/>. It is using
36 simple &acro.sgml;-like syntax.
41 <term><literal>grs.marc.</literal><replaceable>type</replaceable></term>
44 This allows &zebra; to read
45 records in the ISO2709 (&acro.marc;) encoding standard.
46 Last parameter <replaceable>type</replaceable> names the
47 <literal>.abs</literal> file (see below)
48 which describes the specific &acro.marc; structure of the input record as
49 well as the indexing rules.
51 <para>The <literal>grs.marc</literal> uses an internal represtantion
52 which is not &acro.xml; conformant. In particular &acro.marc; tags are
53 presented as elements with the same name. And &acro.xml; elements
54 may not start with digits. Therefore this filter is only
55 suitable for systems returning &acro.grs1; and &acro.marc; records. For &acro.xml;
56 use <literal>grs.marcxml</literal> filter instead (see below).
59 The loadable <literal>grs.marc</literal> filter module
60 is packaged in the GNU/Debian package
61 <literal>libidzebra2.0-mod-grs-marc</literal>
66 <term><literal>grs.marcxml.</literal><replaceable>type</replaceable></term>
69 This allows &zebra; to read ISO2709 encoded records.
70 Last parameter <replaceable>type</replaceable> names the
71 <literal>.abs</literal> file (see below)
72 which describes the specific &acro.marc; structure of the input record as
73 well as the indexing rules.
76 The internal representation for <literal>grs.marcxml</literal>
77 is the same as for <ulink url="&url.marcxml;">&acro.marcxml;</ulink>.
78 It slightly more complicated to work with than
79 <literal>grs.marc</literal> but &acro.xml; conformant.
82 The loadable <literal>grs.marcxml</literal> filter module
83 is also contained in the GNU/Debian package
84 <literal>libidzebra2.0-mod-grs-marc</literal>
89 <term><literal>grs.xml</literal></term>
92 This filter reads &acro.xml; records and uses
93 <ulink url="http://expat.sourceforge.net/">Expat</ulink> to
94 parse them and convert them into ID&zebra;'s internal
95 <literal>grs</literal> record model.
96 Only one record per file is supported, due to the fact &acro.xml; does
97 not allow two documents to "follow" each other (there is no way
98 to know when a document is finished).
99 This filter is only available if &zebra; is compiled with EXPAT support.
102 The loadable <literal>grs.xml</literal> filter module
103 is packagged in the GNU/Debian package
104 <literal>libidzebra2.0-mod-grs-xml</literal>
109 <term><literal>grs.regx.</literal><replaceable>filter</replaceable></term>
112 This enables a user-supplied Regular Expressions input
113 filter described in <xref linkend="grs-regx-tcl"/>.
116 The loadable <literal>grs.regx</literal> filter module
117 is packaged in the GNU/Debian package
118 <literal>libidzebra2.0-mod-grs-regx</literal>
123 <term><literal>grs.tcl.</literal><replaceable>filter</replaceable></term>
126 Similar to grs.regx but using Tcl for rules, described in
127 <xref linkend="grs-regx-tcl"/>.
130 The loadable <literal>grs.tcl</literal> filter module
131 is also packaged in the GNU/Debian package
132 <literal>libidzebra2.0-mod-grs-regx</literal>
140 <section id="grs-canonical-format">
141 <title>&acro.grs1; Canonical Input Format</title>
144 Although input data can take any form, it is sometimes useful to
145 describe the record processing capabilities of the system in terms of
146 a single, canonical input format that gives access to the full
147 spectrum of structure and flexibility in the system. In &zebra;, this
148 canonical format is an "&acro.sgml;-like" syntax.
152 To use the canonical format specify <literal>grs.sgml</literal> as
157 Consider a record describing an information resource (such a record is
158 sometimes known as a <emphasis>locator record</emphasis>).
159 It might contain a field describing the distributor of the
160 information resource, which might in turn be partitioned into
161 various fields providing details about the distributor, like this:
167 <Distributor>
168 <Name> USGS/WRD </Name>
169 <Organization> USGS/WRD </Organization>
170 <Street-Address>
171 U.S. GEOLOGICAL SURVEY, 505 MARQUETTE, NW
172 </Street-Address>
173 <City> ALBUQUERQUE </City>
174 <State> NM </State>
175 <Zip-Code> 87102 </Zip-Code>
176 <Country> USA </Country>
177 <Telephone> (505) 766-5560 </Telephone>
178 </Distributor>
183 <!-- There is no indentation in the example above! -H
186 The indentation used above is used to illustrate how &zebra;
187 interprets the mark-up. The indentation, in itself, has no
188 significance to the parser for the canonical input format, which
189 discards superfluous whitespace.
195 The keywords surrounded by <...> are
196 <emphasis>tags</emphasis>, while the sections of text
197 in between are the <emphasis>data elements</emphasis>.
198 A data element is characterized by its location in the tree
199 that is made up by the nested elements.
200 Each element is terminated by a closing tag - beginning
201 with <literal><</literal>/, and containing the same symbolic
202 tag-name as the corresponding opening tag.
203 The general closing tag - <literal></></literal> -
204 terminates the element started by the last opening tag. The
205 structuring of elements is significant.
206 The element <emphasis>Telephone</emphasis>,
207 for instance, may be indexed and presented to the client differently,
208 depending on whether it appears inside the
209 <emphasis>Distributor</emphasis> element, or some other,
210 structured data element such a <emphasis>Supplier</emphasis> element.
213 <section id="grs-record-root">
214 <title>Record Root</title>
217 The first tag in a record describes the root node of the tree that
218 makes up the total record. In the canonical input format, the root tag
219 should contain the name of the schema that lends context to the
220 elements of the record
221 (see <xref linkend="grs-internal-representation"/>).
222 The following is a GILS record that
223 contains only a single element (strictly speaking, that makes it an
224 illegal GILS record, since the GILS profile includes several mandatory
225 elements - &zebra; does not validate the contents of a record against
226 the &acro.z3950; profile, however - it merely attempts to match up elements
227 of a local representation with the given schema):
234 <title>Zen and the Art of Motorcycle Maintenance</title>
242 <section id="grs-variants">
243 <title>Variants</title>
246 &zebra; allows you to provide individual data elements in a number of
247 <emphasis>variant forms</emphasis>. Examples of variant forms are
248 textual data elements which might appear in different languages, and
249 images which may appear in different formats or layouts.
250 The variant system in &zebra; is essentially a representation of
251 the variant mechanism of &acro.z3950;-1995.
255 The following is an example of a title element which occurs in two
263 <var lang lang "eng">
264 Zen and the Art of Motorcycle Maintenance</>
265 <var lang lang "dan">
266 Zen og Kunsten at Vedligeholde en Motorcykel</>
273 The syntax of the <emphasis>variant element</emphasis> is
274 <literal><var class type value></literal>.
275 The available values for the <emphasis>class</emphasis> and
276 <emphasis>type</emphasis> fields are given by the variant set
277 that is associated with the current schema
278 (see <xref linkend="grs-variants"/>).
282 Variant elements are terminated by the general end-tag </>, by
283 the variant end-tag </var>, by the appearance of another variant
284 tag with the same <emphasis>class</emphasis> and
285 <emphasis>value</emphasis> settings, or by the
286 appearance of another, normal tag. In other words, the end-tags for
287 the variants used in the example above could have been omitted.
291 Variant elements can be nested. The element
298 <var lang lang "eng"><var body iana "text/plain">
299 Zen and the Art of Motorcycle Maintenance
306 Associates two variant components to the variant list for the title
311 Given the nesting rules described above, we could write
318 <var body iana "text/plain>
319 <var lang lang "eng">
320 Zen and the Art of Motorcycle Maintenance
321 <var lang lang "dan">
322 Zen og Kunsten at Vedligeholde en Motorcykel
329 The title element above comes in two variants. Both have the IANA body
330 type "text/plain", but one is in English, and the other in
331 Danish. The client, using the element selection mechanism of &acro.z3950;,
332 can retrieve information about the available variant forms of data
333 elements, or it can select specific variants based on the requirements
341 <section id="grs-regx-tcl">
342 <title>&acro.grs1; REGX And TCL Input Filters</title>
345 In order to handle general input formats, &zebra; allows the
346 operator to define filters which read individual records in their
347 native format and produce an internal representation that the system
352 Input filters are ASCII files, generally with the suffix
353 <literal>.flt</literal>.
354 The system looks for the files in the directories given in the
355 <emphasis>profilePath</emphasis> setting in the
356 <literal>zebra.cfg</literal> files.
357 The record type for the filter is
358 <literal>grs.regx.</literal><emphasis>filter-filename</emphasis>
359 (fundamental type <literal>grs</literal>, file read
360 type <literal>regx</literal>, argument
361 <emphasis>filter-filename</emphasis>).
365 Generally, an input filter consists of a sequence of rules, where each
366 rule consists of a sequence of expressions, followed by an action. The
367 expressions are evaluated against the contents of the input record,
368 and the actions normally contribute to the generation of an internal
369 representation of the record.
373 An expression can be either of the following:
380 <term><literal>INIT</literal></term>
383 The action associated with this expression is evaluated
384 exactly once in the lifetime of the application, before any records
385 are read. It can be used in conjunction with an action that
386 initializes tables or other resources that are used in the processing
392 <term><literal>BEGIN</literal></term>
395 Matches the beginning of the record. It can be used to
396 initialize variables, etc. Typically, the
397 <emphasis>BEGIN</emphasis> rule is also used
398 to establish the root node of the record.
403 <term><literal>END</literal></term>
406 Matches the end of the record - when all of the contents
407 of the record has been processed.
413 <literal>/</literal><replaceable>reg</replaceable><literal>/</literal>
417 Matches regular expression pattern <replaceable>reg</replaceable>
418 from the input record. The operators supported are the same
419 as for regular expression queries. Refer to
420 <xref linkend="querymodel-regular"/>.
425 <term><literal>BODY</literal></term>
428 This keyword may only be used between two patterns.
429 It matches everything between (not including) those patterns.
434 <term><literal>FINISH</literal></term>
437 The expression associated with this pattern is evaluated
438 once, before the application terminates. It can be used to release
439 system resources - typically ones allocated in the
440 <emphasis>INIT</emphasis> step.
448 An action is surrounded by curly braces ({...}), and
449 consists of a sequence of statements. Statements may be separated
450 by newlines or semicolons (;).
451 Within actions, the strings that matched the expressions
452 immediately preceding the action can be referred to as
457 The available statements are:
464 <term>begin <replaceable>type [parameter ... ]</replaceable></term>
468 data element. The <replaceable>type</replaceable> is one of
476 Begin a new record. The following parameter should be the
477 name of the schema that describes the structure of the record, eg.
478 <literal>gils</literal> or <literal>wais</literal> (see below).
479 The <literal>begin record</literal> call should precede
480 any other use of the <replaceable>begin</replaceable> statement.
488 Begin a new tagged element. The parameter is the
489 name of the tag. If the tag is not matched anywhere in the tagsets
490 referenced by the current schema, it is treated as a local string
499 Begin a new node in a variant tree. The parameters are
500 <replaceable>class type value</replaceable>.
509 <term>data <replaceable>parameter</replaceable></term>
512 Create a data element. The concatenated arguments make
513 up the value of the data element.
514 The option <literal>-text</literal> signals that
515 the layout (whitespace) of the data should be retained for
517 The option <literal>-element</literal>
518 <replaceable>tag</replaceable> wraps the data up in
519 the <replaceable>tag</replaceable>.
520 The use of the <literal>-element</literal> option is equivalent to
521 preceding the command with a <replaceable>begin
522 element</replaceable> command, and following
523 it with the <replaceable>end</replaceable> command.
528 <term>end <replaceable>[type]</replaceable></term>
531 Close a tagged element. If no parameter is given,
532 the last element on the stack is terminated.
533 The first parameter, if any, is a type name, similar
534 to the <replaceable>begin</replaceable> statement.
535 For the <replaceable>element</replaceable> type, a tag
536 name can be provided to terminate a specific tag.
542 <term>unread <replaceable>no</replaceable></term>
545 Move the input pointer to the offset of first character that
546 match rule given by <replaceable>no</replaceable>.
547 The first rule from left-to-right is numbered zero,
548 the second rule is named 1 and so on.
557 The following input filter reads a Usenet news file, producing a
558 record in the WAIS schema. Note that the body of a news posting is
559 separated from the list of headers by a blank line (or rather a
560 sequence of two newline characters.
566 BEGIN { begin record wais }
568 /^From:/ BODY /$/ { data -element name $1 }
569 /^Subject:/ BODY /$/ { data -element title $1 }
570 /^Date:/ BODY /$/ { data -element lastModified $1 }
572 begin element bodyOfDisplay
573 begin variant body iana "text/plain"
582 If &zebra; is compiled with support for Tcl enabled, the statements
583 described above are supplemented with a complete
584 scripting environment, including control structures (conditional
585 expressions and loop constructs), and powerful string manipulation
586 mechanisms for modifying the elements of a record.
593 <section id="grs-internal-representation">
594 <title>&acro.grs1; Internal Record Representation</title>
597 When records are manipulated by the system, they're represented in a
598 tree-structure, with data elements at the leaf nodes, and tags or
599 variant components at the non-leaf nodes. The root-node identifies the
600 schema that lends context to the tagging and structuring of the
601 record. Imagine a simple record, consisting of a 'title' element and
609 TITLE "Zen and the Art of Motorcycle Maintenance"
610 AUTHOR "Robert Pirsig"
616 A slightly more complex record would have the author element consist
617 of two elements, a surname and a first name:
624 TITLE "Zen and the Art of Motorcycle Maintenance"
633 The root of the record will refer to the record schema that describes
634 the structuring of this particular record. The schema defines the
635 element tags (TITLE, FIRST-NAME, etc.) that may occur in the record, as
636 well as the structuring (SURNAME should appear below AUTHOR, etc.). In
637 addition, the schema establishes element set names that are used by
638 the client to request a subset of the elements of a given record. The
639 schema may also establish rules for converting the record to a
640 different schema, by stating, for each element, a mapping to a
644 <section id="grs-tagged-elements">
645 <title>Tagged Elements</title>
648 A data element is characterized by its tag, and its position in the
649 structure of the record. For instance, while the tag "telephone
650 number" may be used different places in a record, we may need to
651 distinguish between these occurrences, both for searching and
652 presentation purposes. For instance, while the phone numbers for the
653 "customer" and the "service provider" are both
654 representatives for the same type of resource (a telephone number), it
655 is essential that they be kept separate. The record schema provides
656 the structure of the record, and names each data element (defined by
657 the sequence of tags - the tag path - by which the element can be
658 reached from the root of the record).
663 <section id="grs-variant-details">
664 <title>Variants</title>
667 The children of a tag node may be either more tag nodes, a data node
668 (possibly accompanied by tag nodes),
669 or a tree of variant nodes. The children of variant nodes are either
670 more variant nodes or a data node (possibly accompanied by more
671 variant nodes). Each leaf node, which is normally a
672 data node, corresponds to a <emphasis>variant form</emphasis> of the
673 tagged element identified by the tag which parents the variant tree.
674 The following title element occurs in two different languages:
680 VARIANT LANG=ENG "War and Peace"
682 VARIANT LANG=DAN "Krig og Fred"
688 Which of the two elements are transmitted to the client by the server
689 depends on the specifications provided by the client, if any.
693 In practice, each variant node is associated with a triple of class,
694 type, value, corresponding to the variant mechanism of &acro.z3950;.
699 <section id="grs-data-elements">
700 <title>Data Elements</title>
703 Data nodes have no children (they are always leaf nodes in the record
708 FIXME! Documentation needs extension here about types of nodes - numerical,
709 textual, etc., plus the various types of inclusion notes.
717 <section id="grs-conf">
718 <title>&acro.grs1; Record Model Configuration</title>
721 The following sections describe the configuration files that govern
722 the internal management of <literal>grs</literal> records.
723 The system searches for the files
724 in the directories specified by the <emphasis>profilePath</emphasis>
725 setting in the <literal>zebra.cfg</literal> file.
728 <section id="grs-abstract-syntax">
729 <title>The Abstract Syntax</title>
732 The abstract syntax definition (also known as an Abstract Record
733 Structure, or ARS) is the focal point of the
734 record schema description. For a given schema, the ABS file may state any
735 or all of the following:
739 FIXME - Need a diagram here, or a simple explanation how it all hangs together -H
748 The object identifier of the &acro.z3950; schema associated
749 with the ARS, so that it can be referred to by the client.
755 The attribute set (which can possibly be a compound of multiple
756 sets) which applies in the profile. This is used when indexing and
757 searching the records belonging to the given profile.
763 The tag set (again, this can consist of several different sets).
764 This is used when reading the records from a file, to recognize the
765 different tags, and when transmitting the record to the client -
766 mapping the tags to their numerical representation, if they are
773 The variant set which is used in the profile. This provides a
774 vocabulary for specifying the <emphasis>forms</emphasis> of
775 data that appear inside the records.
781 Element set names, which are a shorthand way for the client to
782 ask for a subset of the data elements contained in a record. Element
783 set names, in the retrieval module, are mapped to <emphasis>element
784 specifications</emphasis>, which contain information equivalent to the
785 <emphasis>Espec-1</emphasis> syntax of &acro.z3950;.
791 Map tables, which may specify mappings to
792 <emphasis>other</emphasis> database profiles, if desired.
798 Possibly, a set of rules describing the mapping of elements to a
799 &acro.marc; representation.
806 A list of element descriptions (this is the actual ARS of the
807 schema, in &acro.z3950; terms), which lists the ways in which the various
808 tags can be used and organized hierarchically.
817 Several of the entries above simply refer to other files, which
818 describe the given objects.
823 <section id="grs-configuration-files">
824 <title>The Configuration Files</title>
827 This section describes the syntax and use of the various tables which
828 are used by the retrieval module.
832 The number of different file types may appear daunting at first, but
833 each type corresponds fairly clearly to a single aspect of the &acro.z3950;
834 retrieval facilities. Further, the average database administrator,
835 who is simply reusing an existing profile for which tables already
836 exist, shouldn't have to worry too much about the contents of these tables.
840 Generally, the files are simple ASCII files, which can be maintained
841 using any text editor. Blank lines, and lines beginning with a (#) are
842 ignored. Any characters on a line followed by a (#) are also ignored.
843 All other lines contain <emphasis>directives</emphasis>, which provide
844 some setting or value to the system.
845 Generally, settings are characterized by a single
846 keyword, identifying the setting, followed by a number of parameters.
847 Some settings are repeatable (r), while others may occur only once in a
848 file. Some settings are optional (o), while others again are
854 <section id="abs-file">
855 <title>The Abstract Syntax (.abs) Files</title>
858 The name of this file type is slightly misleading in &acro.z3950; terms,
859 since, apart from the actual abstract syntax of the profile, it also
860 includes most of the other definitions that go into a database
865 When a record in the canonical, &acro.sgml;-like format is read from a file
866 or from the database, the first tag of the file should reference the
867 profile that governs the layout of the record. If the first tag of the
868 record is, say, <literal><gils></literal>, the system will look
869 for the profile definition in the file <literal>gils.abs</literal>.
870 Profile definitions are cached, so they only have to be read once
871 during the lifespan of the current process.
875 When writing your own input filters, the
876 <emphasis>record-begin</emphasis> command
877 introduces the profile, and should always be called first thing when
878 introducing a new record.
882 The file may contain the following directives:
889 <term>name <replaceable>symbolic-name</replaceable></term>
892 (m) This provides a shorthand name or
893 description for the profile. Mostly useful for diagnostic purposes.
898 <term>reference <replaceable>OID-name</replaceable></term>
901 (m) The reference name of the OID for the profile.
902 The reference names can be found in the <emphasis>util</emphasis>
908 <term>attset <replaceable>filename</replaceable></term>
911 (m) The attribute set that is used for
912 indexing and searching records belonging to this profile.
917 <term>tagset <replaceable>filename</replaceable></term>
920 (o) The tag set (if any) that describe
921 that fields of the records.
926 <term>varset <replaceable>filename</replaceable></term>
929 (o) The variant set used in the profile.
934 <term>maptab <replaceable>filename</replaceable></term>
937 (o,r) This points to a
938 conversion table that might be used if the client asks for the record
939 in a different schema from the native one.
944 <term>marc <replaceable>filename</replaceable></term>
947 (o) Points to a file containing parameters
948 for representing the record contents in the ISO2709 syntax.
949 Read the description of the &acro.marc; representation facility below.
954 <term>esetname <replaceable>name filename</replaceable></term>
958 given element set name with an element selection file. If an (@) is
959 given in place of the filename, this corresponds to a null mapping for
960 the given element set name.
965 <term>all <replaceable>tags</replaceable></term>
968 (o) This directive specifies a list of attributes
969 which should be appended to the attribute list given for each
970 element. The effect is to make every single element in the abstract
971 syntax searchable by way of the given attributes. This directive
972 provides an efficient way of supporting free-text searching across all
973 elements. However, it does increase the size of the index
974 significantly. The attributes can be qualified with a structure, as in
975 the <replaceable>elm</replaceable> directive below.
980 <term>elm <replaceable>path name attributes</replaceable></term>
983 (o,r) Adds an element to the abstract record syntax of the schema.
984 The <replaceable>path</replaceable> follows the
985 syntax which is suggested by the &acro.z3950; document - that is, a sequence
986 of tags separated by slashes (/). Each tag is given as a
987 comma-separated pair of tag type and -value surrounded by parenthesis.
988 The <replaceable>name</replaceable> is the name of the element, and
989 the <replaceable>attributes</replaceable>
990 specifies which attributes to use when indexing the element in a
991 comma-separated list.
992 A <literal>!</literal> in place of the attribute name is equivalent
993 to specifying an attribute name identical to the element name.
994 A <literal>-</literal> in place of the attribute name
995 specifies that no indexing is to take place for the given element.
996 The attributes can be qualified with <replaceable>field
997 types</replaceable> to specify which
998 character set should govern the indexing procedure for that field.
999 The same data element may be indexed into several different
1000 fields, using different character set definitions.
1001 See the <xref linkend="fields-and-charsets"/>.
1002 The default field type is <literal>w</literal> for
1003 <emphasis>word</emphasis>.
1009 <term>xelm <replaceable>xpath attributes</replaceable></term>
1012 Specifies indexing for record nodes given by
1013 <replaceable>xpath</replaceable>. Unlike directive
1014 elm, this directive allows you to index attribute
1015 contents. The <replaceable>xpath</replaceable> uses
1016 a syntax similar to XPath. The <replaceable>attributes</replaceable>
1017 have same syntax and meaning as directive elm, except that operator
1018 ! refers to the nodes selected by <replaceable>xpath</replaceable>.
1020 xelm / !:w default index
1021 xelm // !:w additional index
1022 xelm /gils/title/@att myatt:w index attribute @att in myatt
1023 xelm title/@att myatt:w same meaning.
1029 <term>melm <replaceable>field$subfield attributes</replaceable></term>
1032 This directive is specifically for &acro.marc;-formatted records,
1033 ingested either in the form of &acro.marcxml; documents, or in the
1034 ISO2709/Z39.2 format using the grs.marcxml input filter. You can
1035 specify indexing rules for any subfield, or you can leave off the
1036 <replaceable>$subfield</replaceable> part and specify default rules
1037 for all subfields of the given field (note: default rules should come
1038 after any subfield-specific rules in the configuration file). The
1039 <replaceable>attributes</replaceable> have the same syntax and meaning
1040 as for the 'elm' directive above.
1045 <term>encoding <replaceable>encodingname</replaceable></term>
1048 This directive specifies character encoding for external records.
1049 For records such as &acro.xml; that specifies encoding within the
1050 file via a header this directive is ignored.
1051 If neither this directive is given, nor an encoding is set
1052 within external records, ISO-8859-1 encoding is assumed.
1057 <term>xpath <literal>enable</literal>/<literal>disable</literal></term>
1060 If this directive is followed by <literal>enable</literal>,
1061 then extra indexing is performed to allow for XPath-like queries.
1062 If this directive is not specified - equivalent to
1063 <literal>disable</literal> - no extra XPath-indexing is performed.
1070 <term>systag <replaceable>systemtag</replaceable> <replaceable>element</replaceable></term>
1073 This directive maps system information to an element during
1074 retrieval. This information is dynamically created. The
1075 following system tags are defined
1081 Size of record in bytes. By default this
1082 is mapped to element <literal>size</literal>.
1091 Score/rank of record. By default this
1092 is mapped to element <literal>rank</literal>.
1093 If no score was calculated for the record (non-ranked
1094 searched) search this directive is ignored.
1103 &zebra;'s system number (record ID) for the
1104 record. By default this is mapped to element
1105 <literal>localControlNumber</literal>.
1110 If you do not want a particular system tag to be applied,
1111 then set the resulting element to something undefined in the
1112 abs file (such as <literal>none</literal>).
1118 <!-- Mike's version -->
1122 <replaceable>systemTag</replaceable>
1123 <replaceable>actualTag</replaceable>
1127 Specifies what information, if any, &zebra; should
1128 automatically include in retrieval records for the
1129 ``system fields'' that it supports.
1130 <replaceable>systemTag</replaceable> may
1131 be any of the following:
1134 <term><literal>rank</literal></term>
1136 An integer indicating the relevance-ranking score
1137 assigned to the record.
1141 <term><literal>sysno</literal></term>
1143 An automatically generated identifier for the record,
1144 unique within this database. It is represented by the
1145 <literal><localControlNumber></literal> element in
1146 &acro.xml; and the <literal>(1,14)</literal> tag in &acro.grs1;.
1150 <term><literal>size</literal></term>
1152 The size, in bytes, of the retrieved record.
1158 The <replaceable>actualTag</replaceable> parameter may be
1159 <literal>none</literal> to indicate that the named element
1160 should be omitted from retrieval records.
1169 The mechanism for controlling indexing is not adequate for
1170 complex databases, and will probably be moved into a separate
1171 configuration table eventually.
1176 The following is an excerpt from the abstract syntax file for the GILS
1184 reference GILS-schema
1189 maptab gils-usmarc.map
1193 esetname VARIANT gils-variant.est # for WAIS-compliance
1194 esetname B gils-b.est
1195 esetname G gils-g.est
1200 elm (1,14) localControlNumber Local-number
1201 elm (1,16) dateOfLastModification Date/time-last-modified
1202 elm (2,1) title w:!,p:!
1203 elm (4,1) controlIdentifier Identifier-standard
1204 elm (2,6) abstract Abstract
1205 elm (4,51) purpose !
1206 elm (4,52) originator -
1207 elm (4,53) accessConstraints !
1208 elm (4,54) useConstraints !
1209 elm (4,70) availability -
1210 elm (4,70)/(4,90) distributor -
1211 elm (4,70)/(4,90)/(2,7) distributorName !
1212 elm (4,70)/(4,90)/(2,10) distributorOrganization !
1213 elm (4,70)/(4,90)/(4,2) distributorStreetAddress !
1214 elm (4,70)/(4,90)/(4,3) distributorCity !
1221 <section id="attset-files">
1222 <title>The Attribute Set (.att) Files</title>
1225 This file type describes the <replaceable>Use</replaceable> elements of
1227 It contains the following directives.
1233 <term>name <replaceable>symbolic-name</replaceable></term>
1236 (m) This provides a shorthand name or
1237 description for the attribute set.
1238 Mostly useful for diagnostic purposes.
1240 </listitem></varlistentry>
1242 <term>reference <replaceable>OID-name</replaceable></term>
1245 (m) The reference name of the OID for
1247 The reference names can be found in the <replaceable>util</replaceable>
1248 module of <replaceable>&yaz;</replaceable>.
1250 </listitem></varlistentry>
1252 <term>include <replaceable>filename</replaceable></term>
1255 (o,r) This directive is used to
1256 include another attribute set as a part of the current one. This is
1257 used when a new attribute set is defined as an extension to another
1258 set. For instance, many new attribute sets are defined as extensions
1259 to the <replaceable>bib-1</replaceable> set.
1260 This is an important feature of the retrieval
1261 system of &acro.z3950;, as it ensures the highest possible level of
1262 interoperability, as those access points of your database which are
1263 derived from the external set (say, bib-1) can be used even by clients
1264 who are unaware of the new set.
1266 </listitem></varlistentry>
1269 <replaceable>att-value att-name [local-value]</replaceable></term>
1273 repeatable directive introduces a new attribute to the set. The
1274 attribute value is stored in the index (unless a
1275 <replaceable>local-value</replaceable> is
1276 given, in which case this is stored). The name is used to refer to the
1277 attribute from the <replaceable>abstract syntax</replaceable>.
1279 </listitem></varlistentry>
1284 This is an excerpt from the GILS attribute set definition.
1285 Notice how the file describing the <emphasis>bib-1</emphasis>
1286 attribute set is referenced.
1293 reference GILS-attset
1296 att 2001 distributorName
1297 att 2002 indextermsControlled
1299 att 2004 accessConstraints
1300 att 2005 useConstraints
1307 <section id="grs-tag-files">
1308 <title>The Tag Set (.tag) Files</title>
1311 This file type defines the tagset of the profile, possibly by
1312 referencing other tag sets (most tag sets, for instance, will include
1313 tagsetG and tagsetM from the &acro.z3950; specification. The file may
1314 contain the following directives.
1321 <term>name <emphasis>symbolic-name</emphasis></term>
1324 (m) This provides a shorthand name or
1325 description for the tag set. Mostly useful for diagnostic purposes.
1327 </listitem></varlistentry>
1329 <term>reference <emphasis>OID-name</emphasis></term>
1332 (o) The reference name of the OID for the tag set.
1333 The reference names can be found in the <emphasis>util</emphasis>
1334 module of <emphasis>&yaz;</emphasis>.
1335 The directive is optional, since not all tag sets
1336 are registered outside of their schema.
1338 </listitem></varlistentry>
1340 <term>type <emphasis>integer</emphasis></term>
1343 (m) The type number of the tagset within the schema
1344 profile (note: this specification really should belong to the .abs
1345 file. This will be fixed in a future release).
1347 </listitem></varlistentry>
1349 <term>include <emphasis>filename</emphasis></term>
1352 (o,r) This directive is used
1353 to include the definitions of other tag sets into the current one.
1355 </listitem></varlistentry>
1357 <term>tag <emphasis>number names type</emphasis></term>
1360 (o,r) Introduces a new tag to the set.
1361 The <emphasis>number</emphasis> is the tag number as used
1362 in the protocol (there is currently no mechanism for
1363 specifying string tags at this point, but this would be quick
1365 The <emphasis>names</emphasis> parameter is a list of names
1366 by which the tag should be recognized in the input file format.
1367 The names should be separated by slashes (/).
1368 The <emphasis>type</emphasis> is the recommended data type of
1370 It should be one of the following:
1436 </listitem></varlistentry>
1441 The following is an excerpt from the TagsetG definition file.
1452 tag 3 publicationPlace string
1453 tag 4 publicationDate string
1454 tag 5 documentId string
1455 tag 6 abstract string
1457 tag 8 date generalizedtime
1458 tag 9 bodyOfDisplay string
1459 tag 10 organization string
1465 <section id="grs-var-files">
1466 <title>The Variant Set (.var) Files</title>
1469 The variant set file is a straightforward representation of the
1470 variant set definitions associated with the protocol. At present, only
1471 the <emphasis>Variant-1</emphasis> set is known.
1475 These are the directives allowed in the file.
1482 <term>name <emphasis>symbolic-name</emphasis></term>
1485 (m) This provides a shorthand name or
1486 description for the variant set. Mostly useful for diagnostic purposes.
1488 </listitem></varlistentry>
1490 <term>reference <emphasis>OID-name</emphasis></term>
1493 (o) The reference name of the OID for
1494 the variant set, if one is required. The reference names can be found
1495 in the <emphasis>util</emphasis> module of <emphasis>&yaz;</emphasis>.
1497 </listitem></varlistentry>
1499 <term>class <emphasis>integer class-name</emphasis></term>
1502 (m,r) Introduces a new
1503 class to the variant set.
1505 </listitem></varlistentry>
1507 <term>type <emphasis>integer type-name datatype</emphasis></term>
1511 new type to the current class (the one introduced by the most recent
1512 <emphasis>class</emphasis> directive).
1513 The type names belong to the same name space as the one used
1514 in the tag set definition file.
1516 </listitem></varlistentry>
1521 The following is an excerpt from the file describing the variant set
1522 <emphasis>Variant-1</emphasis>.
1533 type 1 variantId octetstring
1538 type 2 z39.50 string
1546 <section id="grs-est-files">
1547 <title>The Element Set (.est) Files</title>
1550 The element set specification files describe a selection of a subset
1551 of the elements of a database record. The element selection mechanism
1552 is equivalent to the one supplied by the <emphasis>Espec-1</emphasis>
1553 syntax of the &acro.z3950; specification.
1554 In fact, the internal representation of an element set
1555 specification is identical to the <emphasis>Espec-1</emphasis> structure,
1556 and we'll refer you to the description of that structure for most of
1557 the detailed semantics of the directives below.
1562 Not all of the Espec-1 functionality has been implemented yet.
1563 The fields that are mentioned below all work as expected, unless
1569 The directives available in the element set file are as follows:
1575 <term>defaultVariantSetId <emphasis>OID-name</emphasis></term>
1578 (o) If variants are used in
1579 the following, this should provide the name of the variantset used
1580 (it's not currently possible to specify a different set in the
1581 individual variant request). In almost all cases (certainly all
1582 profiles known to us), the name
1583 <literal>Variant-1</literal> should be given here.
1585 </listitem></varlistentry>
1587 <term>defaultVariantRequest <emphasis>variant-request</emphasis></term>
1591 provides a default variant request for
1592 use when the individual element requests (see below) do not contain a
1593 variant request. Variant requests consist of a blank-separated list of
1594 variant components. A variant compont is a comma-separated,
1595 parenthesized triple of variant class, type, and value (the two former
1596 values being represented as integers). The value can currently only be
1597 entered as a string (this will change to depend on the definition of
1598 the variant in question). The special value (@) is interpreted as a
1599 null value, however.
1601 </listitem></varlistentry>
1604 <emphasis>path ['variant' variant-request]</emphasis></term>
1607 (o,r) This corresponds to a simple element request
1608 in <emphasis>Espec-1</emphasis>.
1609 The path consists of a sequence of tag-selectors, where each of
1610 these can consist of either:
1617 A simple tag, consisting of a comma-separated type-value pair in
1618 parenthesis, possibly followed by a colon (:) followed by an
1619 occurrences-specification (see below). The tag-value can be a number
1620 or a string. If the first character is an apostrophe ('), this
1621 forces the value to be interpreted as a string, even if it
1622 appears to be numerical.
1628 A WildThing, represented as a question mark (?), possibly
1629 followed by a colon (:) followed by an occurrences
1630 specification (see below).
1636 A WildPath, represented as an asterisk (*). Note that the last
1637 element of the path should not be a wildPath (wildpaths don't
1638 work in this version).
1647 The occurrences-specification can be either the string
1648 <literal>all</literal>, the string <literal>last</literal>, or
1649 an explicit value-range. The value-range is represented as
1650 an integer (the starting point), possibly followed by a
1651 plus (+) and a second integer (the number of elements, default
1656 The variant-request has the same syntax as the defaultVariantRequest
1657 above. Note that it may sometimes be useful to give an empty variant
1658 request, simply to disable the default for a specific set of fields
1659 (we aren't certain if this is proper <emphasis>Espec-1</emphasis>,
1660 but it works in this implementation).
1662 </listitem></varlistentry>
1667 The following is an example of an element specification belonging to
1674 simpleelement (1,10)
1675 simpleelement (1,12)
1677 simpleelement (1,14)
1679 simpleelement (4,52)
1686 <section id="schema-mapping">
1687 <title>The Schema Mapping (.map) Files</title>
1690 Sometimes, the client might want to receive a database record in
1691 a schema that differs from the native schema of the record. For
1692 instance, a client might only know how to process WAIS records, while
1693 the database record is represented in a more specific schema, such as
1694 GILS. In this module, a mapping of data to one of the &acro.marc; formats is
1695 also thought of as a schema mapping (mapping the elements of the
1696 record into fields consistent with the given &acro.marc; specification, prior
1697 to actually converting the data to the ISO2709). This use of the
1698 object identifier for &acro.usmarc; as a schema identifier represents an
1699 overloading of the OID which might not be entirely proper. However,
1700 it represents the dual role of schema and record syntax which
1701 is assumed by the &acro.marc; family in &acro.z3950;.
1705 <emphasis>NOTE: FIXME! The schema-mapping functions are so far limited to a
1706 straightforward mapping of elements. This should be extended with
1707 mechanisms for conversions of the element contents, and conditional
1708 mappings of elements based on the record contents.</emphasis>
1712 These are the directives of the schema mapping file format:
1719 <term>targetName <emphasis>name</emphasis></term>
1722 (m) A symbolic name for the target schema
1723 of the table. Useful mostly for diagnostic purposes.
1725 </listitem></varlistentry>
1727 <term>targetRef <emphasis>OID-name</emphasis></term>
1730 (m) An OID name for the target schema.
1731 This is used, for instance, by a server receiving a request to present
1732 a record in a different schema from the native one.
1733 The name, again, is found in the <emphasis>oid</emphasis>
1734 module of <emphasis>&yaz;</emphasis>.
1736 </listitem></varlistentry>
1738 <term>map <emphasis>element-name target-path</emphasis></term>
1742 an element mapping rule to the table.
1744 </listitem></varlistentry>
1750 <section id="grs-mar-files">
1751 <title>The &acro.marc; (ISO2709) Representation (.mar) Files</title>
1754 This file provides rules for representing a record in the ISO2709
1755 format. The rules pertain mostly to the values of the constant-length
1756 header of the record.
1760 NOTE: FIXME! This will be described better. We're in the process of
1761 re-evaluating and most likely changing the way that &acro.marc; records are
1762 handled by the system.</emphasis>
1768 <section id="grs-exchange-formats">
1769 <title>&acro.grs1; Exchange Formats</title>
1772 Converting records from the internal structure to an exchange format
1773 is largely an automatic process. Currently, the following exchange
1774 formats are supported:
1781 &acro.grs1;. The internal representation is based on &acro.grs1;/&acro.xml;, so the
1782 conversion here is straightforward. The system will create
1783 applied variant and supported variant lists as required, if a record
1784 contains variant information.
1790 &acro.xml;. The internal representation is based on &acro.grs1;/&acro.xml; so
1791 the mapping is trivial. Note that &acro.xml; schemas, preprocessing
1792 instructions and comments are not part of the internal representation
1793 and therefore will never be part of a generated &acro.xml; record.
1794 Future versions of the &zebra; will support that.
1800 &acro.sutrs;. Again, the mapping is fairly straightforward. Indentation
1801 is used to show the hierarchical structure of the record. All
1802 "&acro.grs1;" type records support both the &acro.grs1; and &acro.sutrs;
1804 <!-- FIXME - What is &acro.sutrs; - should be expanded here -->
1810 ISO2709-based formats (&acro.usmarc;, etc.). Only records with a
1811 two-level structure (corresponding to fields and subfields) can be
1812 directly mapped to ISO2709. For records with a different structuring
1813 (eg., GILS), the representation in a structure like &acro.usmarc; involves a
1814 schema-mapping (see <xref linkend="schema-mapping"/>), to an
1815 "implied" &acro.usmarc; schema (implied,
1816 because there is no formal schema which specifies the use of the
1817 &acro.usmarc; fields outside of ISO2709). The resultant, two-level record is
1818 then mapped directly from the internal representation to ISO2709. See
1819 the GILS schema definition files for a detailed example of this
1826 Explain. This representation is only available for records
1827 belonging to the Explain schema.
1833 Summary. This ASN-1 based structure is only available for records
1834 belonging to the Summary schema - or schema which provide a mapping
1835 to this schema (see the description of the schema mapping facility
1840 <!-- FIXME - Is this used anywhere ? -H -->
1843 SOIF. Support for this syntax is experimental, and is currently
1844 keyed to a private Index Data OID (1.2.840.10003.5.1000.81.2). All
1845 abstract syntaxes can be mapped to the SOIF format, although nested
1846 elements are represented by concatenation of the tag names at each
1855 <section id="grs-extended-marc-indexing">
1856 <title>Extended indexing of &acro.marc; records</title>
1858 <para>Extended indexing of &acro.marc; records will help you if you need index a
1859 combination of subfields, or index only a part of the whole field,
1860 or use during indexing process embedded fields of &acro.marc; record.
1863 <para>Extended indexing of &acro.marc; records additionally allows:
1867 <para>to index data in LEADER of &acro.marc; record</para>
1871 <para>to index data in control fields (with fixed length)</para>
1875 <para>to use during indexing the values of indicators</para>
1879 <para>to index linked fields for UNI&acro.marc; based formats</para>
1885 <note><para>In compare with simple indexing process the extended indexing
1886 may increase (about 2-3 times) the time of indexing process for &acro.marc;
1887 records.</para></note>
1889 <section id="formula">
1890 <title>The index-formula</title>
1892 <para>At the beginning, we have to define the term
1893 <emphasis>index-formula</emphasis> for &acro.marc; records. This term helps
1894 to understand the notation of extended indexing of &acro.marc; records by &zebra;.
1895 Our definition is based on the document
1896 <ulink url="http://www.rba.ru/rusmarc/soft/Z39-50.htm">"The table
1897 of conformity for &acro.z3950; use attributes and R&acro.usmarc; fields"</ulink>.
1898 The document is available only in russian language.</para>
1901 The <emphasis>index-formula</emphasis> is the combination of
1902 subfields presented in such way:
1906 71-00$a, $g, $h ($c){.$b ($c)} , (1)
1910 We know that &zebra; supports a &acro.bib1; attribute - right truncation.
1911 In this case, the <emphasis>index-formula</emphasis> (1) consists from
1912 forms, defined in the same way as (1)</para>
1921 <para>The original &acro.marc; record may be without some elements, which included in <emphasis>index-formula</emphasis>.
1925 <para>This notation includes such operands as:
1930 <listitem><para>It means whitespace character.</para></listitem>
1935 <listitem><para>The position may contain any value, defined by
1937 For example, <emphasis>index-formula</emphasis></para>
1943 <para>includes</para>
1957 <para>The repeatable elements are defined in figure-brackets {}.
1959 <emphasis>index-formula</emphasis></para>
1962 71-00$a, $g, $h ($c){.$b ($c)} , (3)
1965 <para>includes</para>
1968 71-00$a, $g, $h ($c). $b ($c)
1969 71-00$a, $g, $h ($c). $b ($c). $b ($c)
1970 71-00$a, $g, $h ($c). $b ($c). $b ($c). $b ($c)
1979 All another operands are the same as accepted in &acro.marc; world.
1985 <section id="notation">
1986 <title>Notation of <emphasis>index-formula</emphasis> for &zebra;</title>
1989 <para>Extended indexing overloads <literal>path</literal> of
1990 <literal>elm</literal> definition in abstract syntax file of &zebra;
1991 (<literal>.abs</literal> file). It means that names beginning with
1992 <literal>"mc-"</literal> are interpreted by &zebra; as
1993 <emphasis>index-formula</emphasis>. The database index is created and
1994 linked with <emphasis>access point</emphasis> (&acro.bib1; use attribute)
1995 according to this formula.</para>
1997 <para>For example, <emphasis>index-formula</emphasis></para>
2000 71-00$a, $g, $h ($c){.$b ($c)} , (4)
2003 <para>in <literal>.abs</literal> file looks like:</para>
2006 mc-71.00_$a,_$g,_$h_(_$c_){.$b_(_$c_)}
2010 <para>The notation of <emphasis>index-formula</emphasis> uses the operands:
2015 <listitem><para>It means whitespace character.</para></listitem>
2020 <listitem><para>The position may contain any value, defined by
2021 &acro.marc; format. For example,
2022 <emphasis>index-formula</emphasis></para>
2028 <para>matches <literal>mc-70._1_$a,_$g_</literal> and includes</para>
2040 <listitem><para>The repeatable elements are defined in
2041 figure-brackets {}. For example,
2042 <emphasis>index-formula</emphasis></para>
2045 71#00$a, $g, $h ($c) {.$b ($c)} , (6)
2049 <literal>mc-71.00_$a,_$g,_$h_(_$c_){.$b_(_$c_)}</literal> and
2053 71.00_$a,_$g,_$h_(_$c_).$b_(_$c_)
2054 71.00_$a,_$g,_$h_(_$c_).$b_(_$c_).$b_(_$c_)
2055 71.00_$a,_$g,_$h_(_$c_).$b_(_$c_).$b_(_$c_).$b_(_$c_)
2061 <term><...></term>
2062 <listitem><para>Embedded <emphasis>index-formula</emphasis> (for
2063 linked fields) is between <>. For example,
2064 <emphasis>index-formula</emphasis>
2068 4--#-$170-#1$a, $g ($c) , (7)
2072 <literal>mc-4.._._$1<70._1_$a,_$g_(_$c_)>_</literal> and
2076 463_._$1<70._1_$a,_$g_(_$c_)>_
2085 <para>All another operands are the same as accepted in &acro.marc; world.</para>
2088 <section id="grs-examples">
2089 <title>Examples</title>
2096 <para>indexing LEADER</para>
2098 <para>You need to use keyword "ldr" to index leader. For example,
2099 indexing data from 6th and 7th position of LEADER</para>
2102 elm mc-ldr[6] Record-type !
2103 elm mc-ldr[7] Bib-level !
2110 <para>indexing data from control fields</para>
2112 <para>indexing date (the time added to database)</para>
2115 elm mc-008[0-5] Date/time-added-to-db !
2118 <para>or for R&acro.usmarc; (this data included in 100th field)</para>
2121 elm mc-100___$a[0-7]_ Date/time-added-to-db !
2128 <para>using indicators while indexing</para>
2130 <para>For R&acro.usmarc; <emphasis>index-formula</emphasis>
2131 <literal>70-#1$a, $g</literal> matches</para>
2134 elm 70._1_$a,_$g_ Author !:w,!:p
2137 <para>When &zebra; finds a field according to
2138 <literal>"70."</literal> pattern it checks the indicators. In this
2139 case the value of first indicator doesn't mater, but the value of
2140 second one must be whitespace, in another case a field is not
2146 <para>indexing embedded (linked) fields for UNI&acro.marc; based
2149 <para>For R&acro.usmarc; <emphasis>index-formula</emphasis>
2150 <literal>4--#-$170-#1$a, $g ($c)</literal> matches</para>
2153 elm mc-4.._._$1<70._1_$a,_$g_(_$c_)>_ Author !:w,!:p
2156 <para>Data are extracted from record if the field matches to
2157 <literal>"4.._."</literal> pattern and data in linked field
2159 <emphasis>index-formula</emphasis>
2160 <literal>70._1_$a,_$g_(_$c_)</literal>.</para>
2174 <!-- Keep this comment at the end of the file
2179 sgml-minimize-attributes:nil
2180 sgml-always-quote-attributes:t
2183 sgml-parent-document: "zebra.xml"
2184 sgml-local-catalogs: nil
2185 sgml-namecase-general:t