1 <chapter id="querymodel">
2 <!-- $Id: querymodel.xml,v 1.15 2006-06-23 13:45:41 marc Exp $ -->
3 <title>Query Model</title>
5 <sect1 id="querymodel-overview">
6 <title>Query Model Overview</title>
9 <sect2 id="querymodel-query-languages">
10 <title>Query Languages</title>
13 Zebra is born as a networking Information Retrieval engine adhering
14 to the international standards
15 <ulink url="&url.z39.50;">Z39.50</ulink> and
16 <ulink url="&url.sru;">SRU</ulink>,
18 <literal>type-1 Reverse Polish Notation (RPN)</literal> query
20 Unfortunately, this model has only defined a binary
21 encoded representation, which is used as transport packaging in
22 the Z39.50 protocol layer. This representation is not human
23 readable, nor defines any convenient way to specify queries.
26 Since the <literal>type-1 (RPN)</literal>
27 query structure has no direct, useful string
28 representation, every origin application needs to provide some
29 form of mapping from a local query notation or representation to it.
33 <sect3 id="querymodel-query-languages-pqf">
34 <title>Prefix Query Format (PQF)</title>
37 Index Data has defined a textual representaion in the
38 <literal>Prefix Query Format</literal>, short
39 <literal>PQF</literal>, which mappes
40 <literal>one-to-one</literal> to binary encoded
41 <literal>type-1 RPN</literal> query packages.
42 It has been adopted by other
43 parties developing Z39.50 software, and is often referred to as
44 <literal>Prefix Query Notation</literal>, or in short
45 <literal>PQN</literal>. See
46 <xref linkend="querymodel-pqf"/> for further explanaitions and
47 descriptions of Zebra's capabilities.
51 <sect3 id="querymodel-query-languages-cql">
52 <title>Common Query Language (CQL)</title>
54 The query model of the <literal>type-1 RPN</literal>,
55 expressed in <literal>PQF/PQN</literal> is natively supported.
56 On the other hand, the default <literal>SRU</literal>
57 webservices <literal>Common Query Language</literal>
58 <ulink url="&url.cql;">CQL</ulink> is not natively supported.
61 Zebra can be configured to understand and map CQL to PQF. See
62 <xref linkend="querymodel-cql-to-pqf"/>.
68 <sect2 id="querymodel-operation-types">
69 <title>Operation types</title>
71 Zebra supports all of the three different
72 <literal>Z39.50/SRU</literal> operations defined in the
73 standards: <literal>explain</literal>, <literal>search</literal>,
74 and <literal>scan</literal>. A short description of the
75 functionality and purpose of each is quite in order here.
78 <sect3 id="querymodel-operation-type-explain">
79 <title>Explain Operation</title>
81 The <emphasis>syntax</emphasis> of Z39.50/SRU queries is
82 well known to any client, but the specific
83 <emphasis>semantics</emphasis> - taking into account a
84 particular servers functionalities and abilities - must be
85 discovered from case to case. Enters the
86 <literal>explain</literal> operation, which provides the means
88 <emphasis>fields</emphasis> (also called
89 <emphasis>indexes</emphasis> or <emphasis>access points</emphasis>
90 are provided, which default parameter the server uses, which
91 retrieve document formats are defined, and which specific parts
92 of the general query model are supported.
95 The Z39.50 embeddes the <literal>explain</literal> operation
97 <literal>search</literal> in the magic
98 <literal>IR-Explain-1</literal> database;
99 see <xref linkend="querymodel-exp1"/>.
102 In SRU, <literal>explain</literal> is an entirely seperate
103 operation, which returns an <literal>Zeerex
104 XML</literal> record according to the
105 structure defined by the protocol.
108 In both cases, the information gathered through
109 <literal>explain</literal> operations can be used to
110 auto-configure a client user interface to the servers
115 <sect3 id="querymodel-operation-type-search">
116 <title>Search Operation</title>
118 Search and retrieve interactions are the raison d'ĂȘtre.
119 They are used to query the remote database and
120 return search result documents. Search queries span from
121 simple free text searches to nested complex boolean queries,
122 targeting specific indexes, and possibly enhanced with many
123 query semantic specifications. Search interactions are the heart
124 and soul of Z39.50/SRU servers.
128 <sect3 id="querymodel-operation-type-scan">
129 <title>Scan Operation</title>
131 The <literal>scan</literal> operation is a helper functionality,
132 which operates on one index or access point a time.
136 the means to investigate the content of specific indexes.
137 Scanning an index returns a handfull of terms actually fond in
138 the indexes, and in addition the <literal>scan</literal>
139 operation returns th enumber of documents indexed by each term.
140 A search client can use this information to propose proper
141 spelling of search terms, to auto-fill search boxes, or to
142 display controlled vocabularies.
151 <sect1 id="querymodel-pqf">
152 <title>Prefix Query Format syntax and semantics</title>
154 The <ulink url="&url.yaz.pqf;">PQF grammer</ulink>
155 is documented in the YAZ manual, and shall not be
156 repeated here. This textual PQF representation
157 is always during search mapped to the equivalent Zebra internal
161 <sect2 id="querymodel-pqf-tree">
162 <title>PQF tree structure</title>
164 The PQF parse tree - or the equivalent textual representation -
165 may start with one specification of the
166 <emphasis>attribute set</emphasis> used. Following is a query
168 consists of <emphasis>atomic query parts (APT)</emphasis> or
169 <emphasis>named result sets</emphasis>, eventually
170 paired by <emphasis>boolean binary operators</emphasis>, and
171 finally <emphasis>recursively combined </emphasis> into
175 <sect3 id="querymodel-attribute-sets">
176 <title>Attribute sets</title>
178 Attribute sets define the exact meaning and semantics of queries
179 issued. Zebra comes with some predefined attribute set
180 definitions, others can easily be defined and added to the
185 <table id="querymodel-attribute-sets-table"
186 frame="all" rowsep="1" colsep="1" align="center">
188 <caption>Attribute sets predefined in Zebra</caption>
192 <td>Attribute set</td>
201 <td><literal>Explain</literal></td>
202 <td><literal>exp-1</literal></td>
203 <td>Special attribute set used on the special automagic
204 <literal>IR-Explain-1</literal> database to gain information on
205 server capabilities, database names, and database
210 <td><literal>Bib1</literal></td>
211 <td><literal>bib-1</literal></td>
212 <td>Standard PQF query language attribute set which defines the
213 semantics of Z39.50 searching. In addition, all of the
214 non-use attributes (type 2-9) define the hard-wired
220 <td><literal>GILS</literal></td>
221 <td><literal>gils</literal></td>
222 <td>Extention to the <literal>Bib1</literal> attribute set.</td>
227 <td><literal>IDXPATH</literal></td>
228 <td><literal>idxpath</literal></td>
229 <td>Hardwired XPATH like attribute set, only available for
230 indexing with the GRS record model</td>
239 The <literal>use attributes (type 1)</literal> mappings the
240 predefined attribute sets are found in the
241 attribute set configuration files <filename>tab/*.att</filename>.
245 The Zebra internal query processing is modeled after
246 the <literal>Bib1</literal> attribute set, and the non-use
247 attributes type 2-6 are hard-wired in. It is therefore essential
248 to be familiar with <xref linkend="querymodel-bib1-nonuse"/>.
252 <sect3 id="querymodel-boolean-operators">
253 <title>Boolean operators</title>
255 A pair of subquery trees, or of atomic queries, is combined
256 using the standard boolean operators into new query trees.
259 <table id="querymodel-boolean-operators-table"
260 frame="all" rowsep="1" colsep="1" align="center">
262 <caption>Boolean operators</caption>
271 <tr><td><literal>@and</literal></td>
272 <td>binary <literal>AND</literal> operator</td>
273 <td>Set intersection of two atomic queries hit sets</td>
275 <tr><td><literal>@or</literal></td>
276 <td>binary <literal>OR</literal> operator</td>
277 <td>Set union of two atomic queries hit sets</td>
279 <tr><td><literal>@not</literal></td>
280 <td>binary <literal>AND NOT</literal> operator</td>
281 <td>Set complement of two atomic queries hit sets</td>
283 <tr><td><literal>@prox</literal></td>
284 <td>binary <literal>PROXIMY</literal> operator</td>
285 <td>Set intersection of two atomic queries hit sets. In
286 addition, the intersection set is purged for all
287 documents which do not satisfy the requested query
288 term proximity. Usually a proper subset of the AND
295 For example, we can combine the terms
296 <emphasis>information</emphasis> and <emphasis>retrieval</emphasis>
297 into different searches in the default index of the default
298 attribute set as follows.
299 Querying for the union of all documents containing the
300 terms <emphasis>information</emphasis> OR
301 <emphasis>retrieval</emphasis>:
303 Z> find @or information retrieval
307 Querying for the intersection of all documents containing the
308 terms <emphasis>information</emphasis> AND
309 <emphasis>retrieval</emphasis>:
310 The hit set is a subset of the coresponding
313 Z> find @and information retrieval
317 Querying for the intersection of all documents containing the
318 terms <emphasis>information</emphasis> AND
319 <emphasis>retrieval</emphasis>, taking proximity into account:
320 The hit set is a subset of the coresponding
323 Z> find @prox 0 3 0 2 k 2 information retrieval
325 See <ulink url="&url.yaz.pqf;">PQF grammer</ulink> for details.
328 Querying for the intersection of all documents containing the
329 terms <emphasis>information</emphasis> AND
330 <emphasis>retrieval</emphasis>, in the same order and near each
331 other as described in the term list
332 The hit set is a subset of the coresponding
335 Z> find "information retrieval"
341 <sect3 id="querymodel-atomic-queries">
342 <title>Atomic queries (APT)</title>
344 Atomic queries are the query parts which work on one acess point
345 only. These consist of <literal>an attribute list</literal>
346 followed by a <literal>single term</literal> or a
347 <literal>quoted term list</literal>, and are often called
348 <emphasis>Attributes-Plus-Terms (APT)</emphasis> queries.
351 Unsupplied non-use attributes type 2-9 are either inherited from
352 higher nodes in the query tree, or are set to Zebra's default values.
353 See <xref linkend="querymodel-bib1"/> for details.
356 <table id="querymodel-atomic-queries-table"
357 frame="all" rowsep="1" colsep="1" align="center">
359 <caption>Atomic queries</caption>
362 <tr><td>one</td><td>two</td></tr>
367 <td><emphasis>attribute list</emphasis></td>
368 <td>List of <literal>orthogonal</literal> attributes</td>
369 <td>Any of the orthogonal attribute types may be omitted,
370 these are inherited from higher query tree nodes, or if not
371 inherited, are set to the default Zebra configuration values.
375 <td><emphasis>term</emphasis></td>
376 <td>single <literal>term</literal>
377 or <literal>quoted term list</literal> </td>
378 <td>Here the search terms or list of search terms is added
384 Querying for the term <emphasis>information</emphasis> in the
385 default index using the default attribite set, the server choice
386 of access point/index, and the default non-use attributes.
392 Equivalent query fully specified including all default values:
394 Z> find @attrset bib-1 @attr 1=1017 @attr 2=3 @attr 3=3 @attr 4=1 @attr 5=100 @attr 6=1 information
399 Finding all documents which have the term
400 <emphasis>debussy</emphasis> in the title field.
402 Z> find @attr 1=4 debussy
409 <sect3 id="querymodel-resultset">
410 <title>Named Result Sets</title>
412 Named result sets are supported in Zebra, and result sets can be
413 used as operands without limitations.
416 After the execution of a search, the result set is available at
417 the server, such that the client can use it for subsequent
418 searches or retrieval requests. The Z30.50 standard actually
419 stresses the fact that result sets are voliatile. It may cease
420 to exist at any time point after search, and the server will
421 send a diagnostic to the effect that the requested
422 result set does not exist any more.
426 Defining a named result set and re-using it in the next query,
427 using <literal>yaz-client</literal>.
429 Z> f @attr 1=4 mozart
431 Number of hits: 43, setno 1
433 Z> f @and @set 1 @attr 1=4 amadeus
435 Number of hits: 14, setno 2
437 Z> f @attr 1=1016 beethoven
439 Number of hits: 26, setno 3
445 Named result sets are only supported by the Z39.50 protocol.
446 The SRU web service is stateless, and therefore the notion of
447 named result sets does not exist when acessing a Zebra server by
453 <sect3 id="querymodel-use-string">
454 <title>Zebra's special access point of type 'string'</title>
456 The numeric <literal>use (type 1)</literal> attribute is usually
457 refered to from a given
458 attribute set. In addition, Zebra let you use
459 <emphasis>any internal index
460 name defined in your configuration</emphasis>
461 as use atribute value. This is a great feature for
462 debugging, and when you do
463 not need the complecity of defined use attribute values. It is
464 the preferred way of accessing Zebra indexes directly.
467 Finding all documents which have the term list "information
468 retrieval" in an Zebra index, using it's internal full string
469 name. Scanning the same index.
471 Z> find @attr 1=sometext "information retrieval"
472 Z> scan @attr 1=sometext aterm
476 Searching or scanning
477 the bib-1 use attribute 54 using it's string name:
479 Z> find @attr 1=Code-language eng
480 Z> scan @attr 1=Code-language ""
484 It is possible to search
485 in any silly string index - if it's defined in your
486 indexation rules and can be parsed by the PQF parser.
487 This is definitely not the recommended use of
488 this facility, as it might confuse your users with some very
491 Z> find @attr 1=silly/xpath/alike[@index]/name "information retrieval"
495 See also <xref linkend="querymodel-pqf-apt-mapping"/> for details, and
496 <xref linkend="server-sru"/>
497 for the SRU PQF query extention using string names as a fast
502 <sect3 id="querymodel-use-xpath">
503 <title>Zebra's special access point of type 'XPath'
504 for GRS filters</title>
506 As we have seen above, it is possible (albeit seldom a great
508 <ulink url="http://www.w3.org/TR/xpath">XPath 1.0</ulink> based
509 search by defining <literal>use (type 1)</literal>
510 <emphasis>string</emphasis> attributes which in appearence
511 <emphasis>resemble XPath queries</emphasis>. There are two
512 problems with this approach: first, the XPath-look-alike has to
513 be defined at indexation time, no new undefined
514 XPath queries can entered at search time, and second, it might
515 confuse users very much that an XPath-alike index name in fact
516 gets populated from a possible entirely different XML element
517 than it pretends to access.
520 When using the <literal>GRS Record Model</literal>
521 (see <xref linkend="record-model-grs"/>), we have the
522 possibility to embed <emphasis>life</emphasis>
524 in the PQF queries, which are here called
525 <literal>use (type 1)</literal> <emphasis>xpath</emphasis>
526 attributes. You must enable the
527 <literal>xpath enable</literal> directive in your
528 <literal>.abs</literal> config files.
531 Only a <emphasis>very</emphasis> restricted subset of the
532 <ulink url="http://www.w3.org/TR/xpath">XPath 1.0</ulink>
533 standard is supported as the GRS record model is simpler than
534 a full XML DOM structure. See the following examples for
538 Finding all documents which have the term "content"
539 inside a text node found in a specific XML DOM
540 <emphasis>subtree</emphasis>, whose starting element is
543 Z> find @attr 1=/root content
544 Z> find @attr 1=/root/first content
546 <emphasis>Notice that the
547 XPath must be absolute, i.e., must start with '/', and that the
548 XPath <literal>decendant-or-self</literal> axis followed by a
549 text node selection <literal>text()</literal> is implicitly
550 appended to the stated XPath.
552 It follows that the above searches are interpreted as:
554 Z> find @attr 1=/root//text() content
555 Z> find @attr 1=/root/first//text() content
560 Searching inside attribute strings is possible:
562 Z> find @attr 1=/link/@creator morten
567 Filter the adressing XPath by a predicate working on exact
569 attributes (in the XML sense) can be done: return all those docs which
570 have the term "english" contained in one of all text subnodes of
571 the subtree defined by the XPath
572 <literal>/record/title[@lang='en']</literal>. And similar
575 Z> find @attr 1=/record/title[@lang='en'] english
576 Z> find @attr 1=/link[@creator='sisse'] sibelius
577 Z> find @attr 1=/link[@creator='sisse']/description[@xml:lang='da'] sibelius
582 Combining numeric indexes, boolean expressions,
583 and xpath based searches is possible:
585 Z> find @attr 1=/record/title @and foo bar
586 Z> find @and @attr 1=/record/title foo @attr 1=4 bar
590 Escaping PQF keywords and other non-parseable XPath constructs
591 with <literal>'{ }'</literal> to prevent syntax errors:
593 Z> find @attr {1=/root/first[@attr='danish']} content
594 Z> find @attr {1=/record/@set} oai
598 It is worth mentioning that these dynamic performed XPath
599 queries are a performance bottelneck, as no optimized
600 specialized indexes can be used. Therefore, avoid the use of
601 this facility when speed is essential, and the database content
602 size is medium to large.
609 <sect2 id="querymodel-exp1">
610 <title>Explain Attribute Set</title>
612 The Z39.50 standard defines the
613 <ulink url="&url.z39.50.explain;">Explain</ulink> attribute set
614 <literal>Exp-1</literal>, which is used to discover information
615 about a server's search semantics and functional capabilities
616 Zebra exposes a "classic"
617 Explain database by base name <literal>IR-Explain-1</literal>, which
618 is populated with system internal information.
621 The attribute-set <literal>exp-1</literal> consists of a single
622 <literal>use attribute (type 1)</literal>.
625 In addition, the non-Use
626 <literal>bib-1</literal> attributes, that is, the types
627 <literal>Relation</literal>, <literal>Position</literal>,
628 <literal>Structure</literal>, <literal>Truncation</literal>,
629 and <literal>Completeness</literal> are imported from
630 the <literal>bib-1</literal> attribute set, and may be used
631 within any explain query.
634 <sect3 id="querymodel-exp1-use">
635 <title>Use Attributes (type = 1)</title>
637 The following Explain search atributes are supported:
638 <literal>ExplainCategory</literal> (@attr 1=1),
639 <literal>DatabaseName</literal> (@attr 1=3),
640 <literal>DateAdded</literal> (@attr 1=9),
641 <literal>DateChanged</literal>(@attr 1=10).
644 A search in the use attribute <literal>ExplainCategory</literal>
645 supports only these predefined values:
646 <literal>CategoryList</literal>, <literal>TargetInfo</literal>,
647 <literal>DatabaseInfo</literal>, <literal>AttributeDetails</literal>.
650 See <filename>tab/explain.att</filename> and the
651 <ulink url="&url.z39.50;">Z39.50</ulink> standard
652 for more information.
657 <title>Explain searches with yaz-client</title>
659 Classic Explain only defines retrieval of Explain information
660 via ASN.1. Pratically no Z39.50 clients supports this. Fortunately
661 they don't have to - Zebra allows retrieval of this information
663 <literal>SUTRS</literal>, <literal>XML</literal>,
664 <literal>GRS-1</literal> and <literal>ASN.1</literal> Explain.
668 List supported categories to find out which explain commands are
672 Z> find @attr exp1 1=1 categorylist
679 Get target info, that is, investigate which databases exist at
680 this server endpoint:
683 Z> find @attr exp1 1=1 targetinfo
694 List all supported databases, the number of hits
695 is the number of databases found, which most commonly are the
697 the <literal>Default</literal> and the
698 <literal>IR-Explain-1</literal> databases.
701 Z> find @attr exp1 1=1 databaseinfo
708 Get database info record for database <literal>Default</literal>.
711 Z> find @and @attr exp1 1=1 databaseinfo @attr exp1 1=3 Default
713 Identical query with explicitly specified attribute set:
716 Z> find @attrset exp1 @and @attr 1=1 databaseinfo @attr 1=3 Default
721 Get attribute details record for database
722 <literal>Default</literal>.
723 This query is very useful to study the internal Zebra indexes.
724 If records have been indexed using the <literal>alvis</literal>
725 XSLT filter, the string representation names of the known indexes can be
729 Z> find @and @attr exp1 1=1 attributedetails @attr exp1 1=3 Default
731 Identical query with explicitly specified attribute set:
734 Z> find @attrset exp1 @and @attr 1=1 attributedetails @attr 1=3 Default
741 <sect2 id="querymodel-bib1">
742 <title>Bib1 Attribute Set</title>
744 Most of the information contained in this section is an excerpt of
745 the <literal>ATTRIBUTE SET BIB-1 (Z39.50-1995)
747 found at <ulink url="&url.z39.50.attset.bib1.1995;">. The BIB-1
748 Attribute Set Semantics</ulink> from 1995, also in an updated
749 <ulink url="&url.z39.50.attset.bib1;">Bib-1
750 Attribute Set</ulink>
751 version from 2003. Index Data is not the copyright holder of this
752 information, except for the configuration details, the listing of
753 Zebra's capabilities, and the example queries.
757 <sect3 id="querymodel-bib1-use">
758 <title>Use Attributes (type 1)</title>
761 A use attribute specifies an access point for any atomic query.
762 These acess points are highly dependent on the attribute set used
763 in the query, and are user configurable using the following
764 default configuration files:
765 <filename>tab/bib1.att</filename>,
766 <filename>tab/dan1.att</filename>,
767 <filename>tab/explain.att</filename>, and
768 <filename>tab/gils.att</filename>.
769 New attribute sets can be added by adding new
770 <filename>tab/*.att</filename> configuration files, which need to
771 be sourced in the main configuration <filename>zebra.cfg</filename>.
775 In addition, Zebra allows the acess of
776 <emphasis>internal index names</emphasis> and <emphasis>dynamic
777 XPath</emphasis> as use attributes; see
778 <xref linkend="querymodel-use-string"/> and
779 <xref linkend="querymodel-use-xpath"/>.
783 Phrase search for <emphasis>information retrieval</emphasis> in
784 the title-register, scanning the same register afterwards:
786 Z> find @attr 1=4 "information retrieval"
787 Z> scan @attr 1=4 information
795 <sect2 id="querymodel-bib1-nonuse">
796 <title>Zebra general Bib1 Non-Use Attributes (type 2-6)</title>
798 <sect3 id="querymodel-bib1-relation">
799 <title>Relation Attributes (type 2)</title>
802 Relation attributes describe the relationship of the access
804 of the relation) to the search term as qualified by the attributes (right
805 side of the relation), e.g., Date-publication <= 1975.
808 <table id="querymodel-bib1-relation-table"
809 frame="all" rowsep="1" colsep="1" align="center">
811 <caption>Relation Attributes (type 2)</caption>
826 <td>Less than or equal</td>
836 <td>Greater or equal</td>
841 <td>Greater than</td>
866 <td>AlwaysMatches</td>
874 The relation attributes
875 <literal>1-5</literal> are supported and work exactly as
877 All ordering operations are based on a lexicographical ordering,
878 <emphasis>expect</emphasis> when the
879 <literal>structure attribute numeric (109)</literal> is used. In
880 this case, ordering is numerical. See
881 <xref linkend="querymodel-bib1-structure"/>.
883 Z> find @attr 1=Title @attr 2=1 music
885 Number of hits: 11745, setno 1
887 Z> find @attr 1=Title @attr 2=2 music
889 Number of hits: 11771, setno 2
891 Z> find @attr 1=Title @attr 2=3 music
893 Number of hits: 532, setno 3
895 Z> find @attr 1=Title @attr 2=4 music
897 Number of hits: 11463, setno 4
899 Z> find @attr 1=Title @attr 2=5 music
901 Number of hits: 11419, setno 5
906 The relation attribute
907 <literal>Relevance (102)</literal> is supported, see
908 <xref linkend="administration-ranking"/> for full information.
912 Ranked search for <emphasis>information retrieval</emphasis> in
915 Z> find @attr 1=4 @attr 2=102 "information retrieval"
920 The relation attribute
921 <literal>AlwaysMatches (103)</literal> is in the default
923 supported in conjecture with structure attribute
924 <literal>Phrase (1)</literal> (which may be omitted by
926 It can be configured to work with other structure attributes,
927 see the configuration file
928 <filename>tab/default.idx</filename> and
929 <xref linkend="querymodel-pqf-apt-mapping"/>.
932 <literal>AlwaysMatches (103)</literal> is a
933 great way to discover how many documents have been indexed in a
934 given field. The search term is ignored, but needed for correct
935 PQF syntax. An empty search term may be supplied.
937 Z> find @attr 1=Title @attr 2=103 ""
938 Z> find @attr 1=Title @attr 2=103 @attr 4=1 ""
945 <sect3 id="querymodel-bib1-position">
946 <title>Position Attributes (type 3)</title>
949 The position attribute specifies the location of the search term
950 within the field or subfield in which it appears.
953 <table id="querymodel-bib1-position-table"
954 frame="all" rowsep="1" colsep="1" align="center">
956 <caption>Position Attributes (type 3)</caption>
966 <td>First in field </td>
971 <td>First in subfield</td>
976 <td>Any position in field</td>
984 The position attribute values <literal>first in field (1)</literal>,
985 and <literal>first in subfield(2)</literal> are unsupported.
986 Using them does not trigger an error, but silent defaults to
987 <literal>any position in field (3)</literal>.
992 <sect3 id="querymodel-bib1-structure">
993 <title>Structure Attributes (type 4)</title>
996 The structure attribute specifies the type of search
997 term. This causes the search to be mapped on
998 different Zebra internal indexes, which must have been defined
1003 The possible values of the
1004 <literal>structure attribute (type 4)</literal> can be defined
1005 using the configuration file <filename>
1006 tab/default.idx</filename>.
1007 The default configuration is summerized in this table.
1010 <table id="querymodel-bib1-structure-table"
1011 frame="all" rowsep="1" colsep="1" align="center">
1013 <caption>Structure Attributes (type 4)</caption>
1043 <td>Date (normalized)</td>
1053 <td>Date (un-normalized)</td>
1055 <td>unsupported</td>
1058 <td>Name (normalized) </td>
1060 <td>unsupported</td>
1063 <td>Name (un-normalized) </td>
1065 <td>unsupported</td>
1070 <td>unsupported</td>
1078 <td>Free-form-text</td>
1083 <td>Document-text</td>
1088 <td>Local-number</td>
1095 <td>unsupported</td>
1098 <td>Numeric string</td>
1107 The structure attribute values
1108 <literal>Word list (6)</literal>
1109 is supported, and maps to the boolean <literal>AND</literal>
1110 combination of words supplied. The word list is useful when
1111 google-like bag-of-word queries need to be translated from a GUI
1112 query language to PQF. For example, the following queries
1115 Z> find @attr 1=Title @attr 4=6 "mozart amadeus"
1116 Z> find @attr 1=Title @and mozart amadeus
1121 The structure attribute value
1122 <literal>Free-form-text (105)</literal> and
1123 <literal>Document-text (106)</literal>
1124 are supported, and map both to the boolean <literal>OR</literal>
1125 combination of words supplied. The following queries
1128 Z> find @attr 1=Body-of-text @attr 4=105 "bach salieri teleman"
1129 Z> find @attr 1=Body-of-text @attr 4=106 "bach salieri teleman"
1130 Z> find @attr 1=Body-of-text @or bach @or salieri teleman
1132 This <literal>OR</literal> list of terms is very usefull in
1133 combination with relevance ranking:
1135 Z> find @attr 1=Body-of-text @attr 2=102 @attr 4=105 "bach salieri teleman"
1140 The structure attribute value
1141 <literal>Local number (107)</literal>
1142 is supported, and maps always to the Zebra internal document ID,
1143 irrespectively which use attribute is specified. The following queries
1144 have exactly the same unique record in the hit set:
1146 Z> find @attr 4=107 10
1147 Z> find @attr 1=4 @attr 4=107 10
1148 Z> find @attr 1=1010 @attr 4=107 10
1154 the GILS schema (<literal>gils.abs</literal>), the
1155 west-bounding-coordinate is indexed as type <literal>n</literal>,
1156 and is therefore searched by specifying
1157 <emphasis>structure</emphasis>=<emphasis>Numeric String</emphasis>.
1158 To match all those records with west-bounding-coordinate greater
1159 than -114 we use the following query:
1161 Z> find @attr 4=109 @attr 2=5 @attr gils 1=2038 -114
1165 The exact mapping between PQF queries and Zebra internal indexes
1166 and index types is explained in
1167 <xref linkend="querymodel-pqf-apt-mapping"/>.
1172 <sect3 id="querymodel-bib1-truncation">
1173 <title>Truncation Attributes (type = 5)</title>
1176 The truncation attribute specifies whether variations of one or
1177 more characters are allowed between serch term and hit terms, or
1178 not. Using non-default truncation attributes will broaden the
1179 document hit set of a search query.
1182 <table id="querymodel-bib1-truncation-table"
1183 frame="all" rowsep="1" colsep="1" align="center">
1185 <caption>Truncation Attributes (type 5)</caption>
1195 <td>Right truncation </td>
1200 <td>Left truncation</td>
1205 <td>Left and right truncation</td>
1210 <td>Do not truncate</td>
1215 <td>Process # in search term</td>
1233 The truncation attribute values 1-3 perform the obvious way:
1235 Z> scan @attr 1=Body-of-text schnittke
1241 Z> find @attr 1=Body-of-text @attr 5=1 schnittke
1243 Number of hits: 95, setno 7
1245 Z> find @attr 1=Body-of-text @attr 5=2 schnittke
1247 Number of hits: 81, setno 6
1249 Z> find @attr 1=Body-of-text @attr 5=3 schnittke
1251 Number of hits: 95, setno 8
1256 The truncation attribute value
1257 <literal>Process # in search term (101)</literal> is a
1258 poor-man's regular expression search. It maps
1259 each <literal>#</literal> to <literal>.*</literal>, and
1260 performes then a <literal>Regexp-1 (102)</literal> regular
1261 expression search. The following two queries are equivalent:
1263 Z> find @attr 1=Body-of-text @attr 5=101 schnit#ke
1264 Z> find @attr 1=Body-of-text @attr 5=102 schnit.*ke
1266 Number of hits: 89, setno 10
1271 The truncation attribute value
1272 <literal>Regexp-1 (102)</literal> is a normal regular search,
1273 see <xref linkend="querymodel-regular"/> for details.
1275 Z> find @attr 1=Body-of-text @attr 5=102 schnit+ke
1276 Z> find @attr 1=Body-of-text @attr 5=102 schni[a-t]+ke
1281 The truncation attribute value
1282 <literal>Regexp-2 (103) </literal> is a Zebra specific extention
1283 which allows <emphasis>fuzzy</emphasis> matches. One single
1284 error in spelling of search terms is allowed, i.e., a document
1285 is hit if it includes a term which can be mapped to the used
1286 search term by one character substitution, addition, deletion or
1289 Z> find @attr 1=Body-of-text @attr 5=100 schnittke
1291 Number of hits: 81, setno 14
1293 Z> find @attr 1=Body-of-text @attr 5=103 schnittke
1295 Number of hits: 103, setno 15
1301 <sect3 id="querymodel-bib1-completeness">
1302 <title>Completeness Attributes (type = 6)</title>
1306 The <literal>Completeness Attributes (type = 6)</literal>
1307 is used to specify that a given search term or term list is either
1308 part of the terms of a given index/field
1309 (<literal>Incomplete subfield (1)</literal>), or is
1310 what literally is found in the entire field's index
1311 (<literal>Complete field (3)</literal>).
1314 <table id="querymodel-bib1-completeness-table"
1315 frame="all" rowsep="1" colsep="1" align="center">
1316 <caption>Completeness Attributes (type = 6)</caption>
1319 <td>Completeness</td>
1326 <td>Incomplete subfield</td>
1331 <td>Complete subfield</td>
1333 <td>depreciated</td>
1336 <td>Complete field</td>
1344 The <literal>Completeness Attributes (type = 6)</literal>
1345 is only partially and conditionally
1346 supported in the sense that it is ignored if the hit index is
1347 not of structure <literal>type="w"</literal> or
1348 <literal>type="p"</literal>.
1351 <literal>Incomplete subfield (1)</literal> is the default, and
1353 register <literal>type="w"</literal>, whereas
1354 <literal>Complete field (3)</literal> triggers
1355 search and scan in index <literal>type="p"</literal>.
1358 The <literal>Complete subfield (2)</literal> is a reminiscens
1359 from the happy <literal>MARC</literal>
1360 binary format days. Zebra does not support it, but maps silently
1361 to <literal>Complete field (3)</literal>.
1365 The exact mapping between PQF queries and Zebra internal indexes
1366 and index types is explained in
1367 <xref linkend="querymodel-pqf-apt-mapping"/>.
1375 <sect1 id="querymodel-zebra">
1376 <title>Advanced Zebra PQF Features</title>
1378 The Zebra internal query engine has been extended to specific needs
1379 not covered by the <literal>bib-1</literal> attribute set query
1380 model. These extentions are <emphasis>non-standard</emphasis>
1381 and <emphasis>non-portable</emphasis>: most functional extentions
1382 are modeled over the <literal>bib-1</literal> attribute set,
1383 defining type 7-9 attributes.
1384 There are also the speciel
1385 <literal>string</literal> type index names for the
1386 <literal>idxpath</literal> attribute set.
1389 <sect2 id="querymodel-zebra-attr-allrecords">
1390 <title>Zebra specific retrieval of all records</title>
1392 Zebra defines a hardwired <literal>string</literal> index name
1393 called <literal>_ALLRECORDS</literal>. It matches any record
1394 contained in the database, if used in conjunction with
1395 the relation attribute
1396 <literal>AlwaysMatches (103)</literal>.
1399 The <literal>_ALLRECORDS</literal> index name is used for total database
1400 export. The search term is ignored, it may be empty.
1402 Z> find @attr 1=_ALLRECORDS @attr 2=103 ""
1406 Combination with other index types can be made. For example, to
1407 find all records which are <emphasis>not</emphasis> indexed in
1408 the <literal>Title</literal> register, issue one of the two
1411 Z> find @not @attr 1=_ALLRECORDS @attr 2=103 "" @attr 1=Title @attr 2=103 ""
1412 Z> find @not @attr 1=_ALLRECORDS @attr 2=103 "" @attr 1=4 @attr 2=103 ""
1416 The special string index <literal>_ALLRECORDS</literal> is
1417 experimental, and the provided functionality and syntax may very
1418 well change in future releases of Zebra.
1423 <sect2 id="querymodel-zebra-attr-search">
1424 <title>Zebra specific Search Extentions to all Attribute Sets</title>
1426 Zebra extends the Bib1 attribute types, and these extentions are
1427 recognized regardless of attribute
1428 set used in a <literal>search</literal> operation query.
1431 <table id="querymodel-zebra-attr-search-table"
1432 frame="all" rowsep="1" colsep="1" align="center">
1434 <caption>Zebra Search Attribute Extentions</caption>
1440 <td>Zebra version</td>
1445 <td>Embedded Sort</td>
1457 <td>Rank Weight</td>
1463 <td>Approx Limit</td>
1469 <td>Term Reference</td>
1477 <sect3 id="querymodel-zebra-attr-sorting">
1478 <title>Zebra Extention Embedded Sort Attribute (type 7)</title>
1481 The embedded sort is a way to specify sort within a query - thus
1482 removing the need to send a Sort Request separately. It is both
1483 faster and does not require clients to deal with the Sort
1488 All ordering operations are based on a lexicographical ordering,
1489 <emphasis>expect</emphasis> when the
1490 <literal>structure attribute numeric (109)</literal> is used. In
1491 this case, ordering is numerical. See
1492 <xref linkend="querymodel-bib1-structure"/>.
1496 The possible values after attribute <literal>type 7</literal> are
1497 <literal>1</literal> ascending and
1498 <literal>2</literal> descending.
1499 The attributes+term (APT) node is separate from the
1500 rest and must be <literal>@or</literal>'ed.
1501 The term associated with APT is the sorting level in integers,
1502 where <literal>0</literal> means primary sort,
1503 <literal>1</literal> means secondary sort, and so forth.
1504 See also <xref linkend="administration-ranking"/>.
1507 For example, searching for water, sort by title (ascending)
1509 Z> find @or @attr 1=1016 water @attr 7=1 @attr 1=4 0
1513 Or, searching for water, sort by title ascending, then date descending
1515 Z> find @or @or @attr 1=1016 water @attr 7=1 @attr 1=4 0 @attr 7=2 @attr 1=30 1
1519 <sect3 id="querymodel-zebra-attr-estimation">
1520 <title>Zebra Extention Term Set Attribute (type 8)</title>
1523 The Term Set feature is a facility that allows a search to store
1524 hitting terms in a "pseudo" resultset; thus a search (as usual) +
1525 a scan-like facility. Requires a client that can do named result
1526 sets since the search generates two result sets. The value for
1527 attribute 8 is the name of a result set (string). The terms in
1528 the named term set are returned as SUTRS records.
1531 For example, searching for u in title, right truncated, and
1532 storing the result in term set named 'aset'
1534 Z> find @attr 5=1 @attr 1=4 @attr 8=aset u
1538 The model has one serious flaw: we don't know the size of term
1539 set. Experimental. Do not use in production code.
1542 <sect3 id="querymodel-zebra-attr-weight">
1543 <title>Zebra Extention Rank Weight Attribute (type 9)</title>
1546 Rank weight is a way to pass a value to a ranking algorithm - so
1547 that one APT has one value - while another as a different one.
1548 See also <xref linkend="administration-ranking"/>.
1551 For example, searching for utah in title with weight 30 as well
1552 as any with weight 20:
1554 Z> find @attr 2=102 @or @attr 9=30 @attr 1=4 utah @attr 9=20 utah
1558 <sect3 id="querymodel-zebra-attr-limit">
1559 <title>Zebra Extention Approximative Limit Attribute (type 9)</title>
1562 Newer Zebra versions normally estemiates hit count for every APT
1563 (leaf) in the query tree. These hit counts are returned as part of
1564 the searchResult-1 facility in the binary encoded Z39.50 search
1568 By setting a limit for the APT we can make Zebra turn into
1569 approximate hit count when a certain hit count limit is
1570 reached. A value of zero means exact hit count.
1573 For example, we might be intersted in exact hit count for a, but
1574 for b we allow hit count estimates for 1000 and higher.
1576 Z> find @and a @attr 9=1000 b
1580 The estimated hit count fascility makes searches faster, as one
1581 only needs to process large hit lists partially.
1584 This facility clashes with rank weight, because there all
1585 documents in the hit lists need to be examined for scoring and
1587 It is an experimental
1588 extention. Do not use in production code.
1591 <sect3 id="querymodel-zebra-attr-termref">
1592 <title>Zebra Extention Term Reference Attribute (type 10)</title>
1595 Zebra supports the <literal>searchResult-1</literal> facility.
1596 If the <literal>Term Reference Attribute (type 10)</literal> is
1597 given, that specifies a subqueryId value returned as part of the
1598 search result. It is a way for a client to name an APT part of a
1608 Experimental. Do not use in production code.
1615 <sect2 id="querymodel-zebra-attr-scan">
1616 <title>Zebra specific Scan Extentions to all Attribute Sets</title>
1618 Zebra extends the Bib1 attribute types, and these extentions are
1619 recognized regardless of attribute
1620 set used in a <literal>scan</literal> operation query.
1622 <table id="querymodel-zebra-attr-scan-table"
1623 frame="all" rowsep="1" colsep="1" align="center">
1625 <caption>Zebra Scan Attribute Extentions</caption>
1631 <td>Zebra version</td>
1636 <td>Result Set Narrow</td>
1642 <td>Approximative Limit</td>
1650 <sect3 id="querymodel-zebra-attr-narrow">
1651 <title>Zebra Extention Result Set Narrow (type 8)</title>
1654 If attribute <literal>Result Set Narrow (type 8)</literal>
1655 is given for <literal>scan</literal>, the value is the name of a
1656 result set. Each hit count in <literal>scan</literal> is
1657 <literal>@and</literal>'ed with the result set given.
1660 Consider for example
1661 the case of scanning all title fields around the
1662 scanterm <emphasis>mozart</emphasis>, then refining the scan by
1663 issuing a filtering query for <emphasis>amadeus</emphasis> to
1664 restric the scan to the result set of the query:
1666 Z> scan @attr 1=4 mozart
1669 mozartforskningen (1)
1673 Z> f @attr 1=4 amadeus
1675 Number of hits: 15, setno 2
1677 Z> scan @attr 1=4 @attr 8=2 mozart
1680 mozartforskningen (0)
1688 Experimental. Do not use in production code.
1691 <sect3 id="querymodel-zebra-attr-approx">
1692 <title>Zebra Extention Approximative Limit (type 9)</title>
1695 The <literal>Zebra Extention Approximative Limit (type
1696 9)</literal> is a way to enable approx
1697 hit counts for <literal>scan</literal> hit counts, in the same
1698 way as for <literal>search</literal> hit counts.
1707 Experimental and buggy. Definitely not to be used in production code.
1714 <sect2 id="querymodel-idxpath">
1715 <title>Zebra special IDXPATH Attribute Set for GRS indexing</title>
1717 The attribute-set <literal>idxpath</literal> consists of a single
1718 <literal>Use (type 1)</literal> attribute. All non-use attributes
1722 This feature is enabled when defining the
1723 <literal>xpath enable</literal> option in the GRS filter
1724 <filename>*.abs</filename> configuration files. If one wants to use
1725 the special <literal>idxpath</literal> numeric attribute set, the
1726 main Zebra configuraiton file <filename>zebra.cfg</filename>
1727 directive <literal>attset: idxpath.att</literal> must be enabled.
1729 <warning>The <literal>idxpath</literal> is depreciated, may not be
1730 supported in future Zebra versions, and should definitely
1731 not be used in production code.
1734 <sect3 id="querymodel-idxpath-use">
1735 <title>IDXPATH Use Attributes (type = 1)</title>
1737 This attribute set allows one to search GRS filter indexed
1738 records by XPATH like structured index names.
1741 <warning>The <literal>idxpath</literal> option defines hard-coded
1742 index names, which might clash with your own index names.
1745 <table id="querymodel-idxpath-use-table"
1746 frame="all" rowsep="1" colsep="1" align="center">
1748 <caption>Zebra specific IDXPATH Use Attributes (type 1)</caption>
1753 <td>String Index</td>
1759 <td>XPATH Begin</td>
1761 <td>_XPATH_BEGIN</td>
1762 <td>depreciated</td>
1768 <td>depreciated</td>
1771 <td>XPATH CData</td>
1773 <td>_XPATH_CDATA</td>
1774 <td>depreciated</td>
1777 <td>XPATH Attribute Name</td>
1779 <td>_XPATH_ATTR_NAME</td>
1780 <td>depreciated</td>
1783 <td>XPATH Attribute CData</td>
1785 <td>_XPATH_ATTR_CDATA</td>
1786 <td>depreciated</td>
1793 See <filename>tab/idxpath.att</filename> for more information.
1796 Search for all documents starting with root element
1797 <literal>/root</literal> (either using the numeric or the string
1800 Z> find @attrset idxpath @attr 1=1 @attr 4=3 root/
1801 Z> find @attr idxpath 1=1 @attr 4=3 root/
1802 Z> find @attr 1=_XPATH_BEGIN @attr 4=3 root/
1806 Search for all documents where specific nested XPATH
1807 <literal>/c1/c2/../cn</literal> exists. Notice the very
1808 counter-intuitive <emphasis>reverse</emphasis> notation!
1810 Z> find @attrset idxpath @attr 1=1 @attr 4=3 cn/cn-1/../c1/
1811 Z> find @attr 1=_XPATH_BEGIN @attr 4=3 cn/cn-1/../c1/
1815 Search for CDATA string <emphasis>text</emphasis> in any element
1817 Z> find @attrset idxpath @attr 1=1016 text
1818 Z> find @attr 1=_XPATH_CDATA text
1822 Search for CDATA string <emphasis>anothertext</emphasis> in any
1825 Z> find @attrset idxpath @attr 1=1015 anothertext
1826 Z> find @attr 1=_XPATH_ATTR_CDATA anothertext
1830 Search for all documents with have an XML element node
1831 including an XML attribute named <emphasis>creator</emphasis>
1833 Z> find @attrset idxpath @attr 1=3 @attr 4=3 creator
1834 Z> find @attr 1=_XPATH_ATTR_NAME @attr 4=3 creator
1838 Combining usual <literal>bib-1</literal> attribut set searches
1839 with <literal>idxpath</literal> attribute set searches:
1841 Z> find @and @attr idxpath 1=1 @attr 4=3 link/ @attr 1=4 mozart
1842 Z> find @and @attr 1=_XPATH_BEGIN @attr 4=3 link/ @attr 1=_XPATH_CDATA mozart
1846 Scanning is supportet on all <literal>idxpath</literal>
1847 indexes, both specified as numeric use attributes, or as string
1850 Z> scan @attrset idxpath @attr 1=1016 text
1851 Z> scan @attr 1=_XPATH_ATTR_CDATA anothertext
1852 Z> scan @attrset idxpath @attr 1=3 @attr 4=3 ''
1860 <sect2 id="querymodel-pqf-apt-mapping">
1861 <title>Mapping from PQF atomic APT queries to Zebra internal
1862 register indexes</title>
1864 The rules for PQF APT mapping are rather tricky to grasp in the
1865 first place. We deal first with the rules for deciding which
1866 internal register or string index to use, according to the use
1867 attribute or access point specified in the query. Thereafter we
1868 deal with the rules for determining the correct structure type of
1872 <sect3 id="querymodel-pqf-apt-mapping-accesspoint">
1873 <title>Mapping of PQF APT access points</title>
1875 Zebra understands four fundamental different types of access
1876 points, of which only the
1877 <emphasis>numeric use attribute</emphasis> type access points
1878 are defined by the <ulink url="&url.z39.50;">Z39.50</ulink>
1880 All other access point types are Zebra specific, and non-portable.
1883 <table id="querymodel-zebra-mapping-accesspoint-types"
1884 frame="all" rowsep="1" colsep="1" align="center">
1886 <caption>Acces point name mapping</caption>
1889 <td>Acess Point</td>
1897 <td>Use attibute</td>
1899 <td>[1-9][1-9]*</td>
1900 <td>directly mapped to string index name</td>
1903 <td>String index name</td>
1905 <td>[a-zA-Z](\-?[a-zA-Z0-9])*</td>
1906 <td>normalized name is used as internal string index name</td>
1909 <td>Zebra internal index name</td>
1911 <td>_[a-zA-Z](_?[a-zA-Z0-9])*</td>
1912 <td>hardwired internal string index name</td>
1915 <td>XPATH special index</td>
1918 <td>special xpath search for GRS indexed records</td>
1924 <literal>Attribute set names</literal> and
1925 <literal>string index names</literal> are normalizes
1926 according to the following rules: all <emphasis>single</emphasis>
1927 hyphens <literal>'-'</literal> are stripped, and all upper case
1928 letters are folded to lower case.
1932 <emphasis>Numeric use attributes</emphasis> are mapped
1933 to the Zebra internal
1934 string index according to the attribute set defintion in use.
1935 The default attribute set is <literal>Bib-1</literal>, and may be
1936 omitted in the PQF query.
1940 According to normalization and numeric
1941 use attribute mapping, it follows that the following
1942 PQF queries are considered equivalent (assuming the default
1943 configuration has not been altered):
1945 Z> find @attr 1=Body-of-text serenade
1946 Z> find @attr 1=bodyoftext serenade
1947 Z> find @attr 1=BodyOfText serenade
1948 Z> find @attr 1=bO-d-Y-of-tE-x-t serenade
1949 Z> find @attr 1=1010 serenade
1950 Z> find @attrset Bib-1 @attr 1=1010 serenade
1951 Z> find @attrset bib1 @attr 1=1010 serenade
1952 Z> find @attrset Bib1 @attr 1=1010 serenade
1953 Z> find @attrset b-I-b-1 @attr 1=1010 serenade
1958 The <emphasis>numerical</emphasis>
1959 <literal>use attributes (type 1)</literal>
1960 are interpreted according to the
1961 attribute sets which have been loaded in the
1962 <literal>zebra.cfg</literal> file, and are matched against specific
1963 fields as specified in the <literal>.abs</literal> file which
1964 describes the profile of the records which have been loaded.
1965 If no use attribute is provided, a default of
1966 <literal>Bib-1 Use Any (1016)</literal> is
1968 The predefined <literal>use attribute sets</literal>
1969 can be reconfigured by tweaking the configuration files
1970 <filename>tab/*.att</filename>, and
1971 new attribute sets can be defined by adding similar files in the
1972 configuration path <literal>profilePath</literal> of the server.
1976 <literal>String indexes</literal> can be acessed directly,
1977 independently which attribute set is in use. These are just
1978 ignored. The above mentioned name normalization applies.
1979 <literal>String index names</literal> are defined in the
1980 used indexing filter configuration files, for example in the
1981 <literal>GRS</literal>
1982 <filename>*.abs</filename> configuration files, or in the
1983 <literal>alvis</literal> filter XSLT indexing stylesheets.
1987 <literal>Zebra internal indexes</literal> can be acessed directly,
1988 according to the same rules as the user defined
1989 <literal>string indexes</literal>. The only difference is that
1990 <literal>Zebra internal indexe names</literal> are hardwired,
1992 must start with the character <literal>'_'</literal>.
1996 Finally, <literal>XPATH</literal> access points are only
1997 available using the <literal>GRS</literal> filter for indexing.
1998 These acees point names must start with the character
1999 <literal>'/'</literal>, they are <emphasis>not
2000 normalized</emphasis>, but passed unaltered to the Zebra internal
2001 XPATH engine. See <xref linkend="querymodel-use-xpath"/>.
2009 <sect3 id="querymodel-pqf-apt-mapping-structuretype">
2010 <title>Mapping of PQF APT structure and completeness to
2011 register type</title>
2013 Internally Zebra has in it's default configuration several
2014 different types of registers or indexes, whose tokenization and
2015 character normalization rules differ. This reflects the fact that
2016 serching fundamental different tokens like dates, numbers,
2017 bitfields and string based text needs different rulesets.
2020 <table id="querymodel-zebra-mapping-structure-types"
2021 frame="all" rowsep="1" colsep="1" align="center">
2023 <caption>Structure and completeness mapping to register types</caption>
2027 <td>Completeness</td>
2028 <td>Register type</td>
2035 phrase (@attr 4=1), word (@attr 4=2),
2036 word-list (@attr 4=6),
2037 free-form-text (@attr 4=105), or document-text (@attr 4=106)
2039 <td>Incomplete field (@attr 6=1)</td>
2041 <td>Traditional tokenized and character normalized word index</td>
2045 phrase (@attr 4=1), word (@attr 4=2),
2046 word-list (@attr 4=6),
2047 free-form-text (@attr 4=105), or document-text (@attr 4=106)
2049 <td>complete field' (@attr 6=3)</td>
2050 <td>Phrase ('p')</td>
2051 <td>Character normalized, but not tokenized index for phrase
2056 <td>urx (@attr 4=104)</td>
2058 <td>URX/URL ('u')</td>
2059 <td>Special index for URL web adresses</td>
2062 <td>numeric (@attr 4=109)</td>
2064 <td>Numeric ('u')</td>
2065 <td>Special index for digital numbers</td>
2068 <td>key (@attr 4=3)</td>
2070 <td>Null bitmap ('0')</td>
2071 <td>Used for non-tokenizated and non-normalized bit sequences</td>
2074 <td>year (@attr 4=4)</td>
2077 <td>Non-tokenizated and non-normalized 4 digit numbers</td>
2080 <td>date (@attr 4=5)</td>
2083 <td>Non-tokenizated and non-normalized ISO date strings</td>
2089 <td>Used with special sort attribute set (@attr 7=1, @attr 7=2)</td>
2095 <td>Internal record ID register, used whenever
2096 Relation Always Matches (@attr 2=103) is specified</td>
2101 <!-- see in util/zebramap.c -->
2104 If a <emphasis>Structure</emphasis> attribute of
2105 <emphasis>Phrase</emphasis> is used in conjunction with a
2106 <emphasis>Completeness</emphasis> attribute of
2107 <emphasis>Complete (Sub)field</emphasis>, the term is matched
2108 against the contents of the phrase (long word) register, if one
2109 exists for the given <emphasis>Use</emphasis> attribute.
2110 A phrase register is created for those fields in the
2111 GRS <filename>*.abs</filename> file that contains a
2112 <literal>p</literal>-specifier.
2114 Z> scan @attr 1=Title @attr 4=1 @attr 6=3 beethoven
2116 bayreuther festspiele (1)
2117 * beethoven bibliography database (1)
2120 Z> find @attr 1=Title @attr 4=1 @attr 6=3 "beethoven bibliography"
2122 Number of hits: 0, setno 5
2124 Z> find @attr 1=Title @attr 4=1 @attr 6=3 "beethoven bibliography database"
2126 Number of hits: 1, setno 6
2131 If <emphasis>Structure</emphasis>=<emphasis>Phrase</emphasis> is
2132 used in conjunction with <emphasis>Incomplete Field</emphasis> - the
2133 default value for <emphasis>Completeness</emphasis>, the
2134 search is directed against the normal word registers, but if the term
2135 contains multiple words, the term will only match if all of the words
2136 are found immediately adjacent, and in the given order.
2137 The word search is performed on those fields that are indexed as
2138 type <literal>w</literal> in the GRS <filename>*.abs</filename> file.
2140 Z> scan @attr 1=Title @attr 4=1 @attr 6=1 beethoven
2146 Z> find @attr 1=Title @attr 4=1 @attr 6=1 beethoven
2148 Number of hits: 18, setno 1
2150 Z> find @attr 1=Title @attr 4=1 @attr 6=1 "beethoven bibliography"
2152 Number of hits: 2, setno 2
2158 If the <emphasis>Structure</emphasis> attribute is
2159 <emphasis>Word List</emphasis>,
2160 <emphasis>Free-form Text</emphasis>, or
2161 <emphasis>Document Text</emphasis>, the term is treated as a
2162 natural-language, relevance-ranked query.
2163 This search type uses the word register, i.e. those fields
2164 that are indexed as type <literal>w</literal> in the
2165 GRS <filename>*.abs</filename> file.
2169 If the <emphasis>Structure</emphasis> attribute is
2170 <emphasis>Numeric String</emphasis> the term is treated as an integer.
2171 The search is performed on those fields that are indexed
2172 as type <literal>n</literal> in the GRS
2173 <filename>*.abs</filename> file.
2177 If the <emphasis>Structure</emphasis> attribute is
2178 <emphasis>URx</emphasis> the term is treated as a URX (URL) entity.
2179 The search is performed on those fields that are indexed as type
2180 <literal>u</literal> in the <filename>*.abs</filename> file.
2184 If the <emphasis>Structure</emphasis> attribute is
2185 <emphasis>Local Number</emphasis> the term is treated as
2186 native Zebra Record Identifier.
2190 If the <emphasis>Relation</emphasis> attribute is
2191 <emphasis>Equals</emphasis> (default), the term is matched
2192 in a normal fashion (modulo truncation and processing of
2193 individual words, if required).
2194 If <emphasis>Relation</emphasis> is <emphasis>Less Than</emphasis>,
2195 <emphasis>Less Than or Equal</emphasis>,
2196 <emphasis>Greater than</emphasis>, or <emphasis>Greater than or
2197 Equal</emphasis>, the term is assumed to be numerical, and a
2198 standard regular expression is constructed to match the given
2200 If <emphasis>Relation</emphasis> is <emphasis>Relevance</emphasis>,
2201 the standard natural-language query processor is invoked.
2205 For the <emphasis>Truncation</emphasis> attribute,
2206 <emphasis>No Truncation</emphasis> is the default.
2207 <emphasis>Left Truncation</emphasis> is not supported.
2208 <emphasis>Process # in search term</emphasis> is supported, as is
2209 <emphasis>Regxp-1</emphasis>.
2210 <emphasis>Regxp-2</emphasis> enables the fault-tolerant (fuzzy)
2211 search. As a default, a single error (deletion, insertion,
2212 replacement) is accepted when terms are matched against the register
2219 <sect2 id="querymodel-regular">
2220 <title>Zebra Regular Expressions in Truncation Attribute (type = 5)</title>
2223 Each term in a query is interpreted as a regular expression if
2224 the truncation value is either <emphasis>Regxp-1 (@attr 5=102)</emphasis>
2225 or <emphasis>Regxp-2 (@attr 5=103)</emphasis>.
2226 Both query types follow the same syntax with the operands:
2229 <table id="querymodel-regular-operands-table"
2230 frame="all" rowsep="1" colsep="1" align="center">
2232 <caption>Regular Expression Operands</caption>
2235 <tr><td>one</td><td>two</td></tr>
2240 <td><literal>x</literal></td>
2241 <td>Matches the character <literal>x</literal>.</td>
2244 <td><literal>.</literal></td>
2245 <td>Matches any character.</td>
2248 <td><literal>[ .. ]</literal></td>
2249 <td>Matches the set of characters specified;
2250 such as <literal>[abc]</literal> or <literal>[a-c]</literal>.</td>
2256 The above operands can be combined with the following operators:
2259 <table id="querymodel-regular-operators-table"
2260 frame="all" rowsep="1" colsep="1" align="center">
2261 <caption>Regular Expression Operators</caption>
2264 <tr><td>one</td><td>two</td></tr>
2269 <td><literal>x*</literal></td>
2270 <td>Matches <literal>x</literal> zero or more times.
2271 Priority: high.</td>
2274 <td><literal>x+</literal></td>
2275 <td>Matches <literal>x</literal> one or more times.
2276 Priority: high.</td>
2279 <td><literal>x?</literal></td>
2280 <td> Matches <literal>x</literal> zero or once.
2281 Priority: high.</td>
2284 <td><literal>xy</literal></td>
2285 <td> Matches <literal>x</literal>, then <literal>y</literal>.
2286 Priority: medium.</td>
2289 <td><literal>x|y</literal></td>
2290 <td> Matches either <literal>x</literal> or <literal>y</literal>.
2294 <td><literal>( )</literal></td>
2295 <td>The order of evaluation may be changed by using parentheses.</td>
2301 If the first character of the <literal>Regxp-2</literal> query
2302 is a plus character (<literal>+</literal>) it marks the
2303 beginning of a section with non-standard specifiers.
2304 The next plus character marks the end of the section.
2305 Currently Zebra only supports one specifier, the error tolerance,
2306 which consists one digit.
2310 Since the plus operator is normally a suffix operator the addition to
2311 the query syntax doesn't violate the syntax for standard regular
2316 For example, a phrase search with regular expressions in
2317 the title-register is performed like this:
2319 Z> find @attr 1=4 @attr 5=102 "informat.* retrieval"
2324 Combinations with other attributes are possible. For example, a
2325 ranked search with a regular expression:
2327 Z> find @attr 1=4 @attr 5=102 @attr 2=102 "informat.* retrieval"
2335 The RecordType parameter in the <literal>zebra.cfg</literal> file, or
2336 the <literal>-t</literal> option to the indexer tells Zebra how to
2337 process input records.
2338 Two basic types of processing are available - raw text and structured
2339 data. Raw text is just that, and it is selected by providing the
2340 argument <literal>text</literal> to Zebra. Structured records are
2341 all handled internally using the basic mechanisms described in the
2342 subsequent sections.
2343 Zebra can read structured records in many different formats.
2349 <sect1 id="querymodel-cql-to-pqf">
2350 <title>Server Side CQL to PQF Query Translation</title>
2353 <literal><cql2rpn>l2rpn.txt</cql2rpn></literal>
2354 YAZ Frontend Virtual
2355 Hosts option, one can configure
2356 the YAZ Frontend CQL-to-PQF
2357 converter, specifying the interpretation of various
2358 <ulink url="&url.cql;">CQL</ulink>
2359 indexes, relations, etc. in terms of Type-1 query attributes.
2360 <!-- The yaz-client config file -->
2363 For example, using server-side CQL-to-PQF conversion, one might
2364 query a zebra server like this:
2367 yaz-client localhost:9999
2369 Z> find text=(plant and soil)
2372 and - if properly configured - even static relevance ranking can
2373 be performed using CQL query syntax:
2376 Z> find text = /relevant (plant and soil)
2382 By the way, the same configuration can be used to
2383 search using client-side CQL-to-PQF conversion:
2384 (the only difference is <literal>querytype cql2rpn</literal>
2386 <literal>querytype cql</literal>, and the call specifying a local
2390 yaz-client -q local/cql2pqf.txt localhost:9999
2391 Z> querytype cql2rpn
2392 Z> find text=(plant and soil)
2398 Exhaustive information can be found in the
2399 Section "Specification of CQL to RPN mappings" in the YAZ manual.
2400 <ulink url="http://www.indexdata.dk/yaz/doc/tools.tkl#tools.cql.map">
2401 http://www.indexdata.dk/yaz/doc/tools.tkl#tools.cql.map</ulink>,
2402 and shall therefore not be repeated here.
2407 <ulink url="http://www.loc.gov/z3950/agency/zing/cql/dc-indexes.html">
2408 http://www.loc.gov/z3950/agency/zing/cql/dc-indexes.html</ulink>
2409 for the Maintenance Agency's work-in-progress mapping of Dublin Core
2410 indexes to Attribute Architecture (util, XD and BIB-2)
2420 <!-- Keep this comment at the end of the file
2425 sgml-minimize-attributes:nil
2426 sgml-always-quote-attributes:t
2429 sgml-parent-document: "zebra.xml"
2430 sgml-local-catalogs: nil
2431 sgml-namecase-general:t