2 <!-- $Id: recordmodel-grs.xml,v 1.4 2006-09-03 21:37:27 adam Exp $ -->
3 <title>GRS Record Model and Filter Modules</title>
6 The record model described in this chapter applies to the fundamental,
8 record type <literal>grs</literal>, introduced in
9 <xref linkend="componentmodulesgrs"/>.
13 <section id="grs-filters">
14 <title>GRS Record Filters</title>
16 Many basic subtypes of the <emphasis>grs</emphasis> type are
23 <term><literal>grs.sgml</literal></term>
26 This is the canonical input format
27 described <xref linkend="grs-canonical-format"/>. It is using
28 simple SGML-like syntax.
33 <term><literal>grs.marc.</literal><replaceable>type</replaceable></term>
36 This allows Zebra to read
37 records in the ISO2709 (MARC) encoding standard.
38 Last parameter <replaceable>type</replaceable> names the
39 <literal>.abs</literal> file (see below)
40 which describes the specific MARC structure of the input record as
41 well as the indexing rules.
43 <para>The <literal>grs.marc</literal> uses an internal represtantion
44 which is not XML conformant. In particular MARC tags are
45 presented as elements with the same name. And XML elements
46 may not start with digits. Therefore this filter is only
47 suitable for systems returning GRS-1 and MARC records. For XML
48 use <literal>grs.marcxml</literal> filter instead (see below).
51 The loadable <literal>grs.marc</literal> filter module
52 is packaged in the GNU/Debian package
53 <literal>libidzebra2.0-mod-grs-marc</literal>
58 <term><literal>grs.marcxml.</literal><replaceable>type</replaceable></term>
61 This allows Zebra to read ISO2709 encoded records.
62 Last parameter <replaceable>type</replaceable> names the
63 <literal>.abs</literal> file (see below)
64 which describes the specific MARC structure of the input record as
65 well as the indexing rules.
68 The internal representation for <literal>grs.marcxml</literal>
69 is the same as for <ulink url="&url.marcxml;">MARCXML</ulink>.
70 It slightly more complicated to work with than
71 <literal>grs.marc</literal> but XML conformant.
74 The loadable <literal>grs.marcxml</literal> filter module
75 is also contained in the GNU/Debian package
76 <literal>libidzebra2.0-mod-grs-marc</literal>
81 <term><literal>grs.xml</literal></term>
84 This filter reads XML records and uses
85 <ulink url="http://expat.sourceforge.net/">Expat</ulink> to
86 parse them and convert them into IDZebra's internal
87 <literal>grs</literal> record model.
88 Only one record per file is supported, due to the fact XML does
89 not allow two documents to "follow" each other (there is no way
90 to know when a document is finished).
91 This filter is only available if Zebra is compiled with EXPAT support.
94 The loadable <literal>grs.xml</literal> filter module
95 is packagged in the GNU/Debian package
96 <literal>libidzebra2.0-mod-grs-xml</literal>
101 <term><literal>grs.regx.</literal><replaceable>filter</replaceable></term>
104 This enables a user-supplied Regular Expressions input
105 filter described in <xref linkend="grs-regx-tcl"/>.
108 The loadable <literal>grs.regx</literal> filter module
109 is packaged in the GNU/Debian package
110 <literal>libidzebra2.0-mod-grs-regx</literal>
115 <term><literal>grs.tcl.</literal><replaceable>filter</replaceable></term>
118 Similar to grs.regx but using Tcl for rules, described in
119 <xref linkend="grs-regx-tcl"/>.
122 The loadable <literal>grs.tcl</literal> filter module
123 is also packaged in the GNU/Debian package
124 <literal>libidzebra2.0-mod-grs-regx</literal>
132 <section id="grs-canonical-format">
133 <title>GRS Canonical Input Format</title>
136 Although input data can take any form, it is sometimes useful to
137 describe the record processing capabilities of the system in terms of
138 a single, canonical input format that gives access to the full
139 spectrum of structure and flexibility in the system. In Zebra, this
140 canonical format is an "SGML-like" syntax.
144 To use the canonical format specify <literal>grs.sgml</literal> as
149 Consider a record describing an information resource (such a record is
150 sometimes known as a <emphasis>locator record</emphasis>).
151 It might contain a field describing the distributor of the
152 information resource, which might in turn be partitioned into
153 various fields providing details about the distributor, like this:
159 <Distributor>
160 <Name> USGS/WRD </Name>
161 <Organization> USGS/WRD </Organization>
162 <Street-Address>
163 U.S. GEOLOGICAL SURVEY, 505 MARQUETTE, NW
164 </Street-Address>
165 <City> ALBUQUERQUE </City>
166 <State> NM </State>
167 <Zip-Code> 87102 </Zip-Code>
168 <Country> USA </Country>
169 <Telephone> (505) 766-5560 </Telephone>
170 </Distributor>
175 <!-- There is no indentation in the example above! -H
178 The indentation used above is used to illustrate how Zebra
179 interprets the mark-up. The indentation, in itself, has no
180 significance to the parser for the canonical input format, which
181 discards superfluous whitespace.
187 The keywords surrounded by <...> are
188 <emphasis>tags</emphasis>, while the sections of text
189 in between are the <emphasis>data elements</emphasis>.
190 A data element is characterized by its location in the tree
191 that is made up by the nested elements.
192 Each element is terminated by a closing tag - beginning
193 with <literal><</literal>/, and containing the same symbolic
194 tag-name as the corresponding opening tag.
195 The general closing tag - <literal></></literal> -
196 terminates the element started by the last opening tag. The
197 structuring of elements is significant.
198 The element <emphasis>Telephone</emphasis>,
199 for instance, may be indexed and presented to the client differently,
200 depending on whether it appears inside the
201 <emphasis>Distributor</emphasis> element, or some other,
202 structured data element such a <emphasis>Supplier</emphasis> element.
205 <section id="grs-record-root">
206 <title>Record Root</title>
209 The first tag in a record describes the root node of the tree that
210 makes up the total record. In the canonical input format, the root tag
211 should contain the name of the schema that lends context to the
212 elements of the record
213 (see <xref linkend="grs-internal-representation"/>).
214 The following is a GILS record that
215 contains only a single element (strictly speaking, that makes it an
216 illegal GILS record, since the GILS profile includes several mandatory
217 elements - Zebra does not validate the contents of a record against
218 the Z39.50 profile, however - it merely attempts to match up elements
219 of a local representation with the given schema):
226 <title>Zen and the Art of Motorcycle Maintenance</title>
234 <section id="grs-variants">
235 <title>Variants</title>
238 Zebra allows you to provide individual data elements in a number of
239 <emphasis>variant forms</emphasis>. Examples of variant forms are
240 textual data elements which might appear in different languages, and
241 images which may appear in different formats or layouts.
242 The variant system in Zebra is essentially a representation of
243 the variant mechanism of Z39.50-1995.
247 The following is an example of a title element which occurs in two
255 <var lang lang "eng">
256 Zen and the Art of Motorcycle Maintenance</>
257 <var lang lang "dan">
258 Zen og Kunsten at Vedligeholde en Motorcykel</>
265 The syntax of the <emphasis>variant element</emphasis> is
266 <literal><var class type value></literal>.
267 The available values for the <emphasis>class</emphasis> and
268 <emphasis>type</emphasis> fields are given by the variant set
269 that is associated with the current schema
270 (see <xref linkend="grs-variants"/>).
274 Variant elements are terminated by the general end-tag </>, by
275 the variant end-tag </var>, by the appearance of another variant
276 tag with the same <emphasis>class</emphasis> and
277 <emphasis>value</emphasis> settings, or by the
278 appearance of another, normal tag. In other words, the end-tags for
279 the variants used in the example above could have been omitted.
283 Variant elements can be nested. The element
290 <var lang lang "eng"><var body iana "text/plain">
291 Zen and the Art of Motorcycle Maintenance
298 Associates two variant components to the variant list for the title
303 Given the nesting rules described above, we could write
310 <var body iana "text/plain>
311 <var lang lang "eng">
312 Zen and the Art of Motorcycle Maintenance
313 <var lang lang "dan">
314 Zen og Kunsten at Vedligeholde en Motorcykel
321 The title element above comes in two variants. Both have the IANA body
322 type "text/plain", but one is in English, and the other in
323 Danish. The client, using the element selection mechanism of Z39.50,
324 can retrieve information about the available variant forms of data
325 elements, or it can select specific variants based on the requirements
333 <section id="grs-regx-tcl">
334 <title>GRS REGX And TCL Input Filters</title>
337 In order to handle general input formats, Zebra allows the
338 operator to define filters which read individual records in their
339 native format and produce an internal representation that the system
344 Input filters are ASCII files, generally with the suffix
345 <literal>.flt</literal>.
346 The system looks for the files in the directories given in the
347 <emphasis>profilePath</emphasis> setting in the
348 <literal>zebra.cfg</literal> files.
349 The record type for the filter is
350 <literal>grs.regx.</literal><emphasis>filter-filename</emphasis>
351 (fundamental type <literal>grs</literal>, file read
352 type <literal>regx</literal>, argument
353 <emphasis>filter-filename</emphasis>).
357 Generally, an input filter consists of a sequence of rules, where each
358 rule consists of a sequence of expressions, followed by an action. The
359 expressions are evaluated against the contents of the input record,
360 and the actions normally contribute to the generation of an internal
361 representation of the record.
365 An expression can be either of the following:
375 The action associated with this expression is evaluated
376 exactly once in the lifetime of the application, before any records
377 are read. It can be used in conjunction with an action that
378 initializes tables or other resources that are used in the processing
387 Matches the beginning of the record. It can be used to
388 initialize variables, etc. Typically, the
389 <emphasis>BEGIN</emphasis> rule is also used
390 to establish the root node of the record.
398 Matches the end of the record - when all of the contents
399 of the record has been processed.
404 <term>/pattern/</term>
407 Matches a string of characters from the input record.
415 This keyword may only be used between two patterns.
416 It matches everything between (not including) those patterns.
424 The expression associated with this pattern is evaluated
425 once, before the application terminates. It can be used to release
426 system resources - typically ones allocated in the
427 <emphasis>INIT</emphasis> step.
435 An action is surrounded by curly braces ({...}), and
436 consists of a sequence of statements. Statements may be separated
437 by newlines or semicolons (;).
438 Within actions, the strings that matched the expressions
439 immediately preceding the action can be referred to as
444 The available statements are:
451 <term>begin <replaceable>type [parameter ... ]</replaceable></term>
455 data element. The <replaceable>type</replaceable> is one of
463 Begin a new record. The following parameter should be the
464 name of the schema that describes the structure of the record, eg.
465 <literal>gils</literal> or <literal>wais</literal> (see below).
466 The <literal>begin record</literal> call should precede
467 any other use of the <replaceable>begin</replaceable> statement.
475 Begin a new tagged element. The parameter is the
476 name of the tag. If the tag is not matched anywhere in the tagsets
477 referenced by the current schema, it is treated as a local string
486 Begin a new node in a variant tree. The parameters are
487 <replaceable>class type value</replaceable>.
496 <term>data <replaceable>parameter</replaceable></term>
499 Create a data element. The concatenated arguments make
500 up the value of the data element.
501 The option <literal>-text</literal> signals that
502 the layout (whitespace) of the data should be retained for
504 The option <literal>-element</literal>
505 <replaceable>tag</replaceable> wraps the data up in
506 the <replaceable>tag</replaceable>.
507 The use of the <literal>-element</literal> option is equivalent to
508 preceding the command with a <replaceable>begin
509 element</replaceable> command, and following
510 it with the <replaceable>end</replaceable> command.
515 <term>end <replaceable>[type]</replaceable></term>
518 Close a tagged element. If no parameter is given,
519 the last element on the stack is terminated.
520 The first parameter, if any, is a type name, similar
521 to the <replaceable>begin</replaceable> statement.
522 For the <replaceable>element</replaceable> type, a tag
523 name can be provided to terminate a specific tag.
529 <term>unread <replaceable>no</replaceable></term>
532 Move the input pointer to the offset of first character that
533 match rule given by <replaceable>no</replaceable>.
534 The first rule from left-to-right is numbered zero,
535 the second rule is named 1 and so on.
544 The following input filter reads a Usenet news file, producing a
545 record in the WAIS schema. Note that the body of a news posting is
546 separated from the list of headers by a blank line (or rather a
547 sequence of two newline characters.
553 BEGIN { begin record wais }
555 /^From:/ BODY /$/ { data -element name $1 }
556 /^Subject:/ BODY /$/ { data -element title $1 }
557 /^Date:/ BODY /$/ { data -element lastModified $1 }
559 begin element bodyOfDisplay
560 begin variant body iana "text/plain"
569 If Zebra is compiled with support for Tcl enabled, the statements
570 described above are supplemented with a complete
571 scripting environment, including control structures (conditional
572 expressions and loop constructs), and powerful string manipulation
573 mechanisms for modifying the elements of a record.
580 <section id="grs-internal-representation">
581 <title>GRS Internal Record Representation</title>
584 When records are manipulated by the system, they're represented in a
585 tree-structure, with data elements at the leaf nodes, and tags or
586 variant components at the non-leaf nodes. The root-node identifies the
587 schema that lends context to the tagging and structuring of the
588 record. Imagine a simple record, consisting of a 'title' element and
596 TITLE "Zen and the Art of Motorcycle Maintenance"
597 AUTHOR "Robert Pirsig"
603 A slightly more complex record would have the author element consist
604 of two elements, a surname and a first name:
611 TITLE "Zen and the Art of Motorcycle Maintenance"
620 The root of the record will refer to the record schema that describes
621 the structuring of this particular record. The schema defines the
622 element tags (TITLE, FIRST-NAME, etc.) that may occur in the record, as
623 well as the structuring (SURNAME should appear below AUTHOR, etc.). In
624 addition, the schema establishes element set names that are used by
625 the client to request a subset of the elements of a given record. The
626 schema may also establish rules for converting the record to a
627 different schema, by stating, for each element, a mapping to a
631 <section id="grs-tagged-elements">
632 <title>Tagged Elements</title>
635 A data element is characterized by its tag, and its position in the
636 structure of the record. For instance, while the tag "telephone
637 number" may be used different places in a record, we may need to
638 distinguish between these occurrences, both for searching and
639 presentation purposes. For instance, while the phone numbers for the
640 "customer" and the "service provider" are both
641 representatives for the same type of resource (a telephone number), it
642 is essential that they be kept separate. The record schema provides
643 the structure of the record, and names each data element (defined by
644 the sequence of tags - the tag path - by which the element can be
645 reached from the root of the record).
650 <section id="grs-variant-details">
651 <title>Variants</title>
654 The children of a tag node may be either more tag nodes, a data node
655 (possibly accompanied by tag nodes),
656 or a tree of variant nodes. The children of variant nodes are either
657 more variant nodes or a data node (possibly accompanied by more
658 variant nodes). Each leaf node, which is normally a
659 data node, corresponds to a <emphasis>variant form</emphasis> of the
660 tagged element identified by the tag which parents the variant tree.
661 The following title element occurs in two different languages:
667 VARIANT LANG=ENG "War and Peace"
669 VARIANT LANG=DAN "Krig og Fred"
675 Which of the two elements are transmitted to the client by the server
676 depends on the specifications provided by the client, if any.
680 In practice, each variant node is associated with a triple of class,
681 type, value, corresponding to the variant mechanism of Z39.50.
686 <section id="grs-data-elements">
687 <title>Data Elements</title>
690 Data nodes have no children (they are always leaf nodes in the record
695 FIXME! Documentation needs extension here about types of nodes - numerical,
696 textual, etc., plus the various types of inclusion notes.
704 <section id="grs-conf">
705 <title>GRS Record Model Configuration</title>
708 The following sections describe the configuration files that govern
709 the internal management of <literal>grs</literal> records.
710 The system searches for the files
711 in the directories specified by the <emphasis>profilePath</emphasis>
712 setting in the <literal>zebra.cfg</literal> file.
715 <section id="grs-abstract-syntax">
716 <title>The Abstract Syntax</title>
719 The abstract syntax definition (also known as an Abstract Record
720 Structure, or ARS) is the focal point of the
721 record schema description. For a given schema, the ABS file may state any
722 or all of the following:
726 FIXME - Need a diagram here, or a simple explanation how it all hangs together -H
735 The object identifier of the Z39.50 schema associated
736 with the ARS, so that it can be referred to by the client.
742 The attribute set (which can possibly be a compound of multiple
743 sets) which applies in the profile. This is used when indexing and
744 searching the records belonging to the given profile.
750 The tag set (again, this can consist of several different sets).
751 This is used when reading the records from a file, to recognize the
752 different tags, and when transmitting the record to the client -
753 mapping the tags to their numerical representation, if they are
760 The variant set which is used in the profile. This provides a
761 vocabulary for specifying the <emphasis>forms</emphasis> of
762 data that appear inside the records.
768 Element set names, which are a shorthand way for the client to
769 ask for a subset of the data elements contained in a record. Element
770 set names, in the retrieval module, are mapped to <emphasis>element
771 specifications</emphasis>, which contain information equivalent to the
772 <emphasis>Espec-1</emphasis> syntax of Z39.50.
778 Map tables, which may specify mappings to
779 <emphasis>other</emphasis> database profiles, if desired.
785 Possibly, a set of rules describing the mapping of elements to a
793 A list of element descriptions (this is the actual ARS of the
794 schema, in Z39.50 terms), which lists the ways in which the various
795 tags can be used and organized hierarchically.
804 Several of the entries above simply refer to other files, which
805 describe the given objects.
810 <section id="grs-configuration-files">
811 <title>The Configuration Files</title>
814 This section describes the syntax and use of the various tables which
815 are used by the retrieval module.
819 The number of different file types may appear daunting at first, but
820 each type corresponds fairly clearly to a single aspect of the Z39.50
821 retrieval facilities. Further, the average database administrator,
822 who is simply reusing an existing profile for which tables already
823 exist, shouldn't have to worry too much about the contents of these tables.
827 Generally, the files are simple ASCII files, which can be maintained
828 using any text editor. Blank lines, and lines beginning with a (#) are
829 ignored. Any characters on a line followed by a (#) are also ignored.
830 All other lines contain <emphasis>directives</emphasis>, which provide
831 some setting or value to the system.
832 Generally, settings are characterized by a single
833 keyword, identifying the setting, followed by a number of parameters.
834 Some settings are repeatable (r), while others may occur only once in a
835 file. Some settings are optional (o), while others again are
841 <section id="abs-file">
842 <title>The Abstract Syntax (.abs) Files</title>
845 The name of this file type is slightly misleading in Z39.50 terms,
846 since, apart from the actual abstract syntax of the profile, it also
847 includes most of the other definitions that go into a database
852 When a record in the canonical, SGML-like format is read from a file
853 or from the database, the first tag of the file should reference the
854 profile that governs the layout of the record. If the first tag of the
855 record is, say, <literal><gils></literal>, the system will look
856 for the profile definition in the file <literal>gils.abs</literal>.
857 Profile definitions are cached, so they only have to be read once
858 during the lifespan of the current process.
862 When writing your own input filters, the
863 <emphasis>record-begin</emphasis> command
864 introduces the profile, and should always be called first thing when
865 introducing a new record.
869 The file may contain the following directives:
876 <term>name <replaceable>symbolic-name</replaceable></term>
879 (m) This provides a shorthand name or
880 description for the profile. Mostly useful for diagnostic purposes.
885 <term>reference <replaceable>OID-name</replaceable></term>
888 (m) The reference name of the OID for the profile.
889 The reference names can be found in the <emphasis>util</emphasis>
895 <term>attset <replaceable>filename</replaceable></term>
898 (m) The attribute set that is used for
899 indexing and searching records belonging to this profile.
904 <term>tagset <replaceable>filename</replaceable></term>
907 (o) The tag set (if any) that describe
908 that fields of the records.
913 <term>varset <replaceable>filename</replaceable></term>
916 (o) The variant set used in the profile.
921 <term>maptab <replaceable>filename</replaceable></term>
924 (o,r) This points to a
925 conversion table that might be used if the client asks for the record
926 in a different schema from the native one.
931 <term>marc <replaceable>filename</replaceable></term>
934 (o) Points to a file containing parameters
935 for representing the record contents in the ISO2709 syntax.
936 Read the description of the MARC representation facility below.
941 <term>esetname <replaceable>name filename</replaceable></term>
945 given element set name with an element selection file. If an (@) is
946 given in place of the filename, this corresponds to a null mapping for
947 the given element set name.
952 <term>all <replaceable>tags</replaceable></term>
955 (o) This directive specifies a list of attributes
956 which should be appended to the attribute list given for each
957 element. The effect is to make every single element in the abstract
958 syntax searchable by way of the given attributes. This directive
959 provides an efficient way of supporting free-text searching across all
960 elements. However, it does increase the size of the index
961 significantly. The attributes can be qualified with a structure, as in
962 the <replaceable>elm</replaceable> directive below.
967 <term>elm <replaceable>path name attributes</replaceable></term>
970 (o,r) Adds an element to the abstract record syntax of the schema.
971 The <replaceable>path</replaceable> follows the
972 syntax which is suggested by the Z39.50 document - that is, a sequence
973 of tags separated by slashes (/). Each tag is given as a
974 comma-separated pair of tag type and -value surrounded by parenthesis.
975 The <replaceable>name</replaceable> is the name of the element, and
976 the <replaceable>attributes</replaceable>
977 specifies which attributes to use when indexing the element in a
978 comma-separated list.
979 A <literal>!</literal> in place of the attribute name is equivalent
980 to specifying an attribute name identical to the element name.
981 A <literal>-</literal> in place of the attribute name
982 specifies that no indexing is to take place for the given element.
983 The attributes can be qualified with <replaceable>field
984 types</replaceable> to specify which
985 character set should govern the indexing procedure for that field.
986 The same data element may be indexed into several different
987 fields, using different character set definitions.
988 See the <xref linkend="fields-and-charsets"/>.
989 The default field type is <literal>w</literal> for
990 <emphasis>word</emphasis>.
996 <term>xelm <replaceable>xpath attributes</replaceable></term>
999 Specifies indexing for record nodes given by
1000 <replaceable>xpath</replaceable>. Unlike directive
1001 elm, this directive allows you to index attribute
1002 contents. The <replaceable>xpath</replaceable> uses
1003 a syntax similar to XPath. The <replaceable>attributes</replaceable>
1004 have same syntax and meaning as directive elm, except that operator
1005 ! refers to the nodes selected by <replaceable>xpath</replaceable>.
1007 xelm / !:w default index
1008 xelm // !:w additional index
1009 xelm /gils/title/@att myatt:w index attribute @att in myatt
1010 xelm title/@att myatt:w same meaning.
1016 <term>melm <replaceable>field$subfield attributes</replaceable></term>
1019 This directive is specifically for MARC-formatted records,
1020 ingested either in the form of MARCXML documents, or in the
1021 ISO2709/Z39.2 format using the grs.marcxml input filter. You can
1022 specify indexing rules for any subfield, or you can leave off the
1023 <replaceable>$subfield</replaceable> part and specify default rules
1024 for all subfields of the given field (note: default rules should come
1025 after any subfield-specific rules in the configuration file). The
1026 <replaceable>attributes</replaceable> have the same syntax and meaning
1027 as for the 'elm' directive above.
1032 <term>encoding <replaceable>encodingname</replaceable></term>
1035 This directive specifies character encoding for external records.
1036 For records such as XML that specifies encoding within the
1037 file via a header this directive is ignored.
1038 If neither this directive is given, nor an encoding is set
1039 within external records, ISO-8859-1 encoding is assumed.
1044 <term>xpath <literal>enable</literal>/<literal>disable</literal></term>
1047 If this directive is followed by <literal>enable</literal>,
1048 then extra indexing is performed to allow for XPath-like queries.
1049 If this directive is not specified - equivalent to
1050 <literal>disable</literal> - no extra XPath-indexing is performed.
1057 <term>systag <replaceable>systemtag</replaceable> <replaceable>element</replaceable></term>
1060 This directive maps system information to an element during
1061 retrieval. This information is dynamically created. The
1062 following system tags are defined
1068 Size of record in bytes. By default this
1069 is mapped to element <literal>size</literal>.
1078 Score/rank of record. By default this
1079 is mapped to element <literal>rank</literal>.
1080 If no score was calculated for the record (non-ranked
1081 searched) search this directive is ignored.
1090 Zebra's system number (record ID) for the
1091 record. By default this is mapped to element
1092 <literal>localControlNumber</literal>.
1097 If you do not want a particular system tag to be applied,
1098 then set the resulting element to something undefined in the
1099 abs file (such as <literal>none</literal>).
1105 <!-- Mike's version -->
1109 <replaceable>systemTag</replaceable>
1110 <replaceable>actualTag</replaceable>
1114 Specifies what information, if any, Zebra should
1115 automatically include in retrieval records for the
1116 ``system fields'' that it supports.
1117 <replaceable>systemTag</replaceable> may
1118 be any of the following:
1121 <term><literal>rank</literal></term>
1123 An integer indicating the relevance-ranking score
1124 assigned to the record.
1128 <term><literal>sysno</literal></term>
1130 An automatically generated identifier for the record,
1131 unique within this database. It is represented by the
1132 <literal><localControlNumber></literal> element in
1133 XML and the <literal>(1,14)</literal> tag in GRS-1.
1137 <term><literal>size</literal></term>
1139 The size, in bytes, of the retrieved record.
1145 The <replaceable>actualTag</replaceable> parameter may be
1146 <literal>none</literal> to indicate that the named element
1147 should be omitted from retrieval records.
1156 The mechanism for controlling indexing is not adequate for
1157 complex databases, and will probably be moved into a separate
1158 configuration table eventually.
1163 The following is an excerpt from the abstract syntax file for the GILS
1171 reference GILS-schema
1176 maptab gils-usmarc.map
1180 esetname VARIANT gils-variant.est # for WAIS-compliance
1181 esetname B gils-b.est
1182 esetname G gils-g.est
1187 elm (1,14) localControlNumber Local-number
1188 elm (1,16) dateOfLastModification Date/time-last-modified
1189 elm (2,1) title w:!,p:!
1190 elm (4,1) controlIdentifier Identifier-standard
1191 elm (2,6) abstract Abstract
1192 elm (4,51) purpose !
1193 elm (4,52) originator -
1194 elm (4,53) accessConstraints !
1195 elm (4,54) useConstraints !
1196 elm (4,70) availability -
1197 elm (4,70)/(4,90) distributor -
1198 elm (4,70)/(4,90)/(2,7) distributorName !
1199 elm (4,70)/(4,90)/(2,10) distributorOrganization !
1200 elm (4,70)/(4,90)/(4,2) distributorStreetAddress !
1201 elm (4,70)/(4,90)/(4,3) distributorCity !
1208 <section id="attset-files">
1209 <title>The Attribute Set (.att) Files</title>
1212 This file type describes the <replaceable>Use</replaceable> elements of
1214 It contains the following directives.
1220 <term>name <replaceable>symbolic-name</replaceable></term>
1223 (m) This provides a shorthand name or
1224 description for the attribute set.
1225 Mostly useful for diagnostic purposes.
1227 </listitem></varlistentry>
1229 <term>reference <replaceable>OID-name</replaceable></term>
1232 (m) The reference name of the OID for
1234 The reference names can be found in the <replaceable>util</replaceable>
1235 module of <replaceable>YAZ</replaceable>.
1237 </listitem></varlistentry>
1239 <term>include <replaceable>filename</replaceable></term>
1242 (o,r) This directive is used to
1243 include another attribute set as a part of the current one. This is
1244 used when a new attribute set is defined as an extension to another
1245 set. For instance, many new attribute sets are defined as extensions
1246 to the <replaceable>bib-1</replaceable> set.
1247 This is an important feature of the retrieval
1248 system of Z39.50, as it ensures the highest possible level of
1249 interoperability, as those access points of your database which are
1250 derived from the external set (say, bib-1) can be used even by clients
1251 who are unaware of the new set.
1253 </listitem></varlistentry>
1256 <replaceable>att-value att-name [local-value]</replaceable></term>
1260 repeatable directive introduces a new attribute to the set. The
1261 attribute value is stored in the index (unless a
1262 <replaceable>local-value</replaceable> is
1263 given, in which case this is stored). The name is used to refer to the
1264 attribute from the <replaceable>abstract syntax</replaceable>.
1266 </listitem></varlistentry>
1271 This is an excerpt from the GILS attribute set definition.
1272 Notice how the file describing the <emphasis>bib-1</emphasis>
1273 attribute set is referenced.
1280 reference GILS-attset
1283 att 2001 distributorName
1284 att 2002 indextermsControlled
1286 att 2004 accessConstraints
1287 att 2005 useConstraints
1294 <section id="grs-tag-files">
1295 <title>The Tag Set (.tag) Files</title>
1298 This file type defines the tagset of the profile, possibly by
1299 referencing other tag sets (most tag sets, for instance, will include
1300 tagsetG and tagsetM from the Z39.50 specification. The file may
1301 contain the following directives.
1308 <term>name <emphasis>symbolic-name</emphasis></term>
1311 (m) This provides a shorthand name or
1312 description for the tag set. Mostly useful for diagnostic purposes.
1314 </listitem></varlistentry>
1316 <term>reference <emphasis>OID-name</emphasis></term>
1319 (o) The reference name of the OID for the tag set.
1320 The reference names can be found in the <emphasis>util</emphasis>
1321 module of <emphasis>YAZ</emphasis>.
1322 The directive is optional, since not all tag sets
1323 are registered outside of their schema.
1325 </listitem></varlistentry>
1327 <term>type <emphasis>integer</emphasis></term>
1330 (m) The type number of the tagset within the schema
1331 profile (note: this specification really should belong to the .abs
1332 file. This will be fixed in a future release).
1334 </listitem></varlistentry>
1336 <term>include <emphasis>filename</emphasis></term>
1339 (o,r) This directive is used
1340 to include the definitions of other tag sets into the current one.
1342 </listitem></varlistentry>
1344 <term>tag <emphasis>number names type</emphasis></term>
1347 (o,r) Introduces a new tag to the set.
1348 The <emphasis>number</emphasis> is the tag number as used
1349 in the protocol (there is currently no mechanism for
1350 specifying string tags at this point, but this would be quick
1352 The <emphasis>names</emphasis> parameter is a list of names
1353 by which the tag should be recognized in the input file format.
1354 The names should be separated by slashes (/).
1355 The <emphasis>type</emphasis> is the recommended data type of
1357 It should be one of the following:
1423 </listitem></varlistentry>
1428 The following is an excerpt from the TagsetG definition file.
1439 tag 3 publicationPlace string
1440 tag 4 publicationDate string
1441 tag 5 documentId string
1442 tag 6 abstract string
1444 tag 8 date generalizedtime
1445 tag 9 bodyOfDisplay string
1446 tag 10 organization string
1452 <section id="grs-var-files">
1453 <title>The Variant Set (.var) Files</title>
1456 The variant set file is a straightforward representation of the
1457 variant set definitions associated with the protocol. At present, only
1458 the <emphasis>Variant-1</emphasis> set is known.
1462 These are the directives allowed in the file.
1469 <term>name <emphasis>symbolic-name</emphasis></term>
1472 (m) This provides a shorthand name or
1473 description for the variant set. Mostly useful for diagnostic purposes.
1475 </listitem></varlistentry>
1477 <term>reference <emphasis>OID-name</emphasis></term>
1480 (o) The reference name of the OID for
1481 the variant set, if one is required. The reference names can be found
1482 in the <emphasis>util</emphasis> module of <emphasis>YAZ</emphasis>.
1484 </listitem></varlistentry>
1486 <term>class <emphasis>integer class-name</emphasis></term>
1489 (m,r) Introduces a new
1490 class to the variant set.
1492 </listitem></varlistentry>
1494 <term>type <emphasis>integer type-name datatype</emphasis></term>
1498 new type to the current class (the one introduced by the most recent
1499 <emphasis>class</emphasis> directive).
1500 The type names belong to the same name space as the one used
1501 in the tag set definition file.
1503 </listitem></varlistentry>
1508 The following is an excerpt from the file describing the variant set
1509 <emphasis>Variant-1</emphasis>.
1520 type 1 variantId octetstring
1525 type 2 z39.50 string
1533 <section id="grs-est-files">
1534 <title>The Element Set (.est) Files</title>
1537 The element set specification files describe a selection of a subset
1538 of the elements of a database record. The element selection mechanism
1539 is equivalent to the one supplied by the <emphasis>Espec-1</emphasis>
1540 syntax of the Z39.50 specification.
1541 In fact, the internal representation of an element set
1542 specification is identical to the <emphasis>Espec-1</emphasis> structure,
1543 and we'll refer you to the description of that structure for most of
1544 the detailed semantics of the directives below.
1549 Not all of the Espec-1 functionality has been implemented yet.
1550 The fields that are mentioned below all work as expected, unless
1556 The directives available in the element set file are as follows:
1562 <term>defaultVariantSetId <emphasis>OID-name</emphasis></term>
1565 (o) If variants are used in
1566 the following, this should provide the name of the variantset used
1567 (it's not currently possible to specify a different set in the
1568 individual variant request). In almost all cases (certainly all
1569 profiles known to us), the name
1570 <literal>Variant-1</literal> should be given here.
1572 </listitem></varlistentry>
1574 <term>defaultVariantRequest <emphasis>variant-request</emphasis></term>
1578 provides a default variant request for
1579 use when the individual element requests (see below) do not contain a
1580 variant request. Variant requests consist of a blank-separated list of
1581 variant components. A variant compont is a comma-separated,
1582 parenthesized triple of variant class, type, and value (the two former
1583 values being represented as integers). The value can currently only be
1584 entered as a string (this will change to depend on the definition of
1585 the variant in question). The special value (@) is interpreted as a
1586 null value, however.
1588 </listitem></varlistentry>
1591 <emphasis>path ['variant' variant-request]</emphasis></term>
1594 (o,r) This corresponds to a simple element request
1595 in <emphasis>Espec-1</emphasis>.
1596 The path consists of a sequence of tag-selectors, where each of
1597 these can consist of either:
1604 A simple tag, consisting of a comma-separated type-value pair in
1605 parenthesis, possibly followed by a colon (:) followed by an
1606 occurrences-specification (see below). The tag-value can be a number
1607 or a string. If the first character is an apostrophe ('), this
1608 forces the value to be interpreted as a string, even if it
1609 appears to be numerical.
1615 A WildThing, represented as a question mark (?), possibly
1616 followed by a colon (:) followed by an occurrences
1617 specification (see below).
1623 A WildPath, represented as an asterisk (*). Note that the last
1624 element of the path should not be a wildPath (wildpaths don't
1625 work in this version).
1634 The occurrences-specification can be either the string
1635 <literal>all</literal>, the string <literal>last</literal>, or
1636 an explicit value-range. The value-range is represented as
1637 an integer (the starting point), possibly followed by a
1638 plus (+) and a second integer (the number of elements, default
1643 The variant-request has the same syntax as the defaultVariantRequest
1644 above. Note that it may sometimes be useful to give an empty variant
1645 request, simply to disable the default for a specific set of fields
1646 (we aren't certain if this is proper <emphasis>Espec-1</emphasis>,
1647 but it works in this implementation).
1649 </listitem></varlistentry>
1654 The following is an example of an element specification belonging to
1661 simpleelement (1,10)
1662 simpleelement (1,12)
1664 simpleelement (1,14)
1666 simpleelement (4,52)
1673 <section id="schema-mapping">
1674 <title>The Schema Mapping (.map) Files</title>
1677 Sometimes, the client might want to receive a database record in
1678 a schema that differs from the native schema of the record. For
1679 instance, a client might only know how to process WAIS records, while
1680 the database record is represented in a more specific schema, such as
1681 GILS. In this module, a mapping of data to one of the MARC formats is
1682 also thought of as a schema mapping (mapping the elements of the
1683 record into fields consistent with the given MARC specification, prior
1684 to actually converting the data to the ISO2709). This use of the
1685 object identifier for USMARC as a schema identifier represents an
1686 overloading of the OID which might not be entirely proper. However,
1687 it represents the dual role of schema and record syntax which
1688 is assumed by the MARC family in Z39.50.
1692 <emphasis>NOTE: FIXME! The schema-mapping functions are so far limited to a
1693 straightforward mapping of elements. This should be extended with
1694 mechanisms for conversions of the element contents, and conditional
1695 mappings of elements based on the record contents.</emphasis>
1699 These are the directives of the schema mapping file format:
1706 <term>targetName <emphasis>name</emphasis></term>
1709 (m) A symbolic name for the target schema
1710 of the table. Useful mostly for diagnostic purposes.
1712 </listitem></varlistentry>
1714 <term>targetRef <emphasis>OID-name</emphasis></term>
1717 (m) An OID name for the target schema.
1718 This is used, for instance, by a server receiving a request to present
1719 a record in a different schema from the native one.
1720 The name, again, is found in the <emphasis>oid</emphasis>
1721 module of <emphasis>YAZ</emphasis>.
1723 </listitem></varlistentry>
1725 <term>map <emphasis>element-name target-path</emphasis></term>
1729 an element mapping rule to the table.
1731 </listitem></varlistentry>
1737 <section id="grs-mar-files">
1738 <title>The MARC (ISO2709) Representation (.mar) Files</title>
1741 This file provides rules for representing a record in the ISO2709
1742 format. The rules pertain mostly to the values of the constant-length
1743 header of the record.
1747 NOTE: FIXME! This will be described better. We're in the process of
1748 re-evaluating and most likely changing the way that MARC records are
1749 handled by the system.</emphasis>
1755 <section id="grs-exchange-formats">
1756 <title>GRS Exchange Formats</title>
1759 Converting records from the internal structure to an exchange format
1760 is largely an automatic process. Currently, the following exchange
1761 formats are supported:
1768 GRS-1. The internal representation is based on GRS-1/XML, so the
1769 conversion here is straightforward. The system will create
1770 applied variant and supported variant lists as required, if a record
1771 contains variant information.
1777 XML. The internal representation is based on GRS-1/XML so
1778 the mapping is trivial. Note that XML schemas, preprocessing
1779 instructions and comments are not part of the internal representation
1780 and therefore will never be part of a generated XML record.
1781 Future versions of the Zebra will support that.
1787 SUTRS. Again, the mapping is fairly straightforward. Indentation
1788 is used to show the hierarchical structure of the record. All
1789 "GRS" type records support both the GRS-1 and SUTRS
1791 <!-- FIXME - What is SUTRS - should be expanded here -->
1797 ISO2709-based formats (USMARC, etc.). Only records with a
1798 two-level structure (corresponding to fields and subfields) can be
1799 directly mapped to ISO2709. For records with a different structuring
1800 (eg., GILS), the representation in a structure like USMARC involves a
1801 schema-mapping (see <xref linkend="schema-mapping"/>), to an
1802 "implied" USMARC schema (implied,
1803 because there is no formal schema which specifies the use of the
1804 USMARC fields outside of ISO2709). The resultant, two-level record is
1805 then mapped directly from the internal representation to ISO2709. See
1806 the GILS schema definition files for a detailed example of this
1813 Explain. This representation is only available for records
1814 belonging to the Explain schema.
1820 Summary. This ASN-1 based structure is only available for records
1821 belonging to the Summary schema - or schema which provide a mapping
1822 to this schema (see the description of the schema mapping facility
1827 <!-- FIXME - Is this used anywhere ? -H -->
1830 SOIF. Support for this syntax is experimental, and is currently
1831 keyed to a private Index Data OID (1.2.840.10003.5.1000.81.2). All
1832 abstract syntaxes can be mapped to the SOIF format, although nested
1833 elements are represented by concatenation of the tag names at each
1842 <section id="grs-extended-marc-indexing">
1843 <title>Extended indexing of MARC records</title>
1845 <para>Extended indexing of MARC records will help you if you need index a
1846 combination of subfields, or index only a part of the whole field,
1847 or use during indexing process embedded fields of MARC record.
1850 <para>Extended indexing of MARC records additionally allows:
1854 <para>to index data in LEADER of MARC record</para>
1858 <para>to index data in control fields (with fixed length)</para>
1862 <para>to use during indexing the values of indicators</para>
1866 <para>to index linked fields for UNIMARC based formats</para>
1872 <note><para>In compare with simple indexing process the extended indexing
1873 may increase (about 2-3 times) the time of indexing process for MARC
1874 records.</para></note>
1876 <section id="formula">
1877 <title>The index-formula</title>
1879 <para>At the beginning, we have to define the term
1880 <emphasis>index-formula</emphasis> for MARC records. This term helps
1881 to understand the notation of extended indexing of MARC records by Zebra.
1882 Our definition is based on the document
1883 <ulink url="http://www.rba.ru/rusmarc/soft/Z39-50.htm">"The table
1884 of conformity for Z39.50 use attributes and RUSMARC fields"</ulink>.
1885 The document is available only in russian language.</para>
1888 The <emphasis>index-formula</emphasis> is the combination of
1889 subfields presented in such way:
1893 71-00$a, $g, $h ($c){.$b ($c)} , (1)
1897 We know that Zebra supports a Bib-1 attribute - right truncation.
1898 In this case, the <emphasis>index-formula</emphasis> (1) consists from
1899 forms, defined in the same way as (1)</para>
1908 <para>The original MARC record may be without some elements, which included in <emphasis>index-formula</emphasis>.
1912 <para>This notation includes such operands as:
1917 <listitem><para>It means whitespace character.</para></listitem>
1922 <listitem><para>The position may contain any value, defined by
1924 For example, <emphasis>index-formula</emphasis></para>
1930 <para>includes</para>
1944 <para>The repeatable elements are defined in figure-brackets {}.
1946 <emphasis>index-formula</emphasis></para>
1949 71-00$a, $g, $h ($c){.$b ($c)} , (3)
1952 <para>includes</para>
1955 71-00$a, $g, $h ($c). $b ($c)
1956 71-00$a, $g, $h ($c). $b ($c). $b ($c)
1957 71-00$a, $g, $h ($c). $b ($c). $b ($c). $b ($c)
1966 All another operands are the same as accepted in MARC world.
1972 <section id="notation">
1973 <title>Notation of <emphasis>index-formula</emphasis> for Zebra</title>
1976 <para>Extended indexing overloads <literal>path</literal> of
1977 <literal>elm</literal> definition in abstract syntax file of Zebra
1978 (<literal>.abs</literal> file). It means that names beginning with
1979 <literal>"mc-"</literal> are interpreted by Zebra as
1980 <emphasis>index-formula</emphasis>. The database index is created and
1981 linked with <emphasis>access point</emphasis> (Bib-1 use attribute)
1982 according to this formula.</para>
1984 <para>For example, <emphasis>index-formula</emphasis></para>
1987 71-00$a, $g, $h ($c){.$b ($c)} , (4)
1990 <para>in <literal>.abs</literal> file looks like:</para>
1993 mc-71.00_$a,_$g,_$h_(_$c_){.$b_(_$c_)}
1997 <para>The notation of <emphasis>index-formula</emphasis> uses the operands:
2002 <listitem><para>It means whitespace character.</para></listitem>
2007 <listitem><para>The position may contain any value, defined by
2008 MARC format. For example,
2009 <emphasis>index-formula</emphasis></para>
2015 <para>matches <literal>mc-70._1_$a,_$g_</literal> and includes</para>
2027 <listitem><para>The repeatable elements are defined in
2028 figure-brackets {}. For example,
2029 <emphasis>index-formula</emphasis></para>
2032 71#00$a, $g, $h ($c) {.$b ($c)} , (6)
2036 <literal>mc-71.00_$a,_$g,_$h_(_$c_){.$b_(_$c_)}</literal> and
2040 71.00_$a,_$g,_$h_(_$c_).$b_(_$c_)
2041 71.00_$a,_$g,_$h_(_$c_).$b_(_$c_).$b_(_$c_)
2042 71.00_$a,_$g,_$h_(_$c_).$b_(_$c_).$b_(_$c_).$b_(_$c_)
2048 <term><...></term>
2049 <listitem><para>Embedded <emphasis>index-formula</emphasis> (for
2050 linked fields) is between <>. For example,
2051 <emphasis>index-formula</emphasis>
2055 4--#-$170-#1$a, $g ($c) , (7)
2059 <literal>mc-4.._._$1<70._1_$a,_$g_(_$c_)>_</literal> and
2063 463_._$1<70._1_$a,_$g_(_$c_)>_
2072 <para>All another operands are the same as accepted in MARC world.</para>
2075 <section id="grs-examples">
2076 <title>Examples</title>
2083 <para>indexing LEADER</para>
2085 <para>You need to use keyword "ldr" to index leader. For example,
2086 indexing data from 6th and 7th position of LEADER</para>
2089 elm mc-ldr[6] Record-type !
2090 elm mc-ldr[7] Bib-level !
2097 <para>indexing data from control fields</para>
2099 <para>indexing date (the time added to database)</para>
2102 elm mc-008[0-5] Date/time-added-to-db !
2105 <para>or for RUSMARC (this data included in 100th field)</para>
2108 elm mc-100___$a[0-7]_ Date/time-added-to-db !
2115 <para>using indicators while indexing</para>
2117 <para>For RUSMARC <emphasis>index-formula</emphasis>
2118 <literal>70-#1$a, $g</literal> matches</para>
2121 elm 70._1_$a,_$g_ Author !:w,!:p
2124 <para>When Zebra finds a field according to
2125 <literal>"70."</literal> pattern it checks the indicators. In this
2126 case the value of first indicator doesn't mater, but the value of
2127 second one must be whitespace, in another case a field is not
2133 <para>indexing embedded (linked) fields for UNIMARC based
2136 <para>For RUSMARC <emphasis>index-formula</emphasis>
2137 <literal>4--#-$170-#1$a, $g ($c)</literal> matches</para>
2140 elm mc-4.._._$1<70._1_$a,_$g_(_$c_)>_ Author !:w,!:p
2143 <para>Data are extracted from record if the field matches to
2144 <literal>"4.._."</literal> pattern and data in linked field
2146 <emphasis>index-formula</emphasis>
2147 <literal>70._1_$a,_$g_(_$c_)</literal>.</para>
2161 <!-- Keep this comment at the end of the file
2166 sgml-minimize-attributes:nil
2167 sgml-always-quote-attributes:t
2170 sgml-parent-document: "zebra.xml"
2171 sgml-local-catalogs: nil
2172 sgml-namecase-general:t