1 <chapter id="administration">
2 <!-- $Id: administration.xml,v 1.30 2006-05-01 12:59:33 marc Exp $ -->
3 <title>Administrating Zebra</title>
4 <!-- ### It's a bit daft that this chapter (which describes half of
5 the configuration-file formats) is separated from
6 "recordmodel-grs.xml" (which describes the other half) by the
7 instructions on running zebraidx and zebrasrv. Some careful
8 re-ordering is required here.
12 Unlike many simpler retrieval systems, Zebra supports safe, incremental
13 updates to an existing index.
17 Normally, when Zebra modifies the index it reads a number of records
19 Depending on your specifications and on the contents of each record
20 one the following events take place for each record:
27 The record is indexed as if it never occurred before.
28 Either the Zebra system doesn't know how to identify the record or
29 Zebra can identify the record but didn't find it to be already indexed.
37 The record has already been indexed.
38 In this case either the contents of the record or the location
39 (file) of the record indicates that it has been indexed before.
47 The record is deleted from the index. As in the
48 update-case it must be able to identify the record.
56 Please note that in both the modify- and delete- case the Zebra
57 indexer must be able to generate a unique key that identifies the record
58 in question (more on this below).
62 To administrate the Zebra retrieval system, you run the
63 <literal>zebraidx</literal> program.
64 This program supports a number of options which are preceded by a dash,
65 and a few commands (not preceded by dash).
69 Both the Zebra administrative tool and the Z39.50 server share a
70 set of index files and a global configuration file.
71 The name of the configuration file defaults to
72 <literal>zebra.cfg</literal>.
73 The configuration file includes specifications on how to index
74 various kinds of records and where the other configuration files
75 are located. <literal>zebrasrv</literal> and <literal>zebraidx</literal>
76 <emphasis>must</emphasis> be run in the directory where the
77 configuration file lives unless you indicate the location of the
78 configuration file by option <literal>-c</literal>.
81 <sect1 id="record-types">
82 <title>Record Types</title>
85 Indexing is a per-record process, in which either insert/modify/delete
86 will occur. Before a record is indexed search keys are extracted from
87 whatever might be the layout the original record (sgml,html,text, etc..).
88 The Zebra system currently supports two fundamental types of records:
89 structured and simple text.
90 To specify a particular extraction process, use either the
91 command line option <literal>-t</literal> or specify a
92 <literal>recordType</literal> setting in the configuration file.
97 <sect1 id="configuration-file">
98 <title>The Zebra Configuration File</title>
101 The Zebra configuration file, read by <literal>zebraidx</literal> and
102 <literal>zebrasrv</literal> defaults to <literal>zebra.cfg</literal>
103 unless specified by <literal>-c</literal> option.
107 You can edit the configuration file with a normal text editor.
108 parameter names and values are separated by colons in the file. Lines
109 starting with a hash sign (<literal>#</literal>) are
114 If you manage different sets of records that share common
115 characteristics, you can organize the configuration settings for each
117 When <literal>zebraidx</literal> is run and you wish to address a
118 given group you specify the group name with the <literal>-g</literal>
120 In this case settings that have the group name as their prefix
121 will be used by <literal>zebraidx</literal>.
122 If no <literal>-g</literal> option is specified, the settings
123 without prefix are used.
127 In the configuration file, the group name is placed before the option
128 name itself, separated by a dot (.). For instance, to set the record type
129 for group <literal>public</literal> to <literal>grs.sgml</literal>
130 (the SGML-like format for structured records) you would write:
135 public.recordType: grs.sgml
140 To set the default value of the record type to <literal>text</literal>
151 The available configuration settings are summarized below. They will be
152 explained further in the following sections.
156 FIXME - Didn't Adam make something to have multiple databases in multiple dirs...
164 <emphasis>group</emphasis>
165 .recordType[<emphasis>.name</emphasis>]:
166 <replaceable>type</replaceable>
170 Specifies how records with the file extension
171 <emphasis>name</emphasis> should be handled by the indexer.
172 This option may also be specified as a command line option
173 (<literal>-t</literal>). Note that if you do not specify a
174 <emphasis>name</emphasis>, the setting applies to all files.
175 In general, the record type specifier consists of the elements (each
176 element separated by dot), <emphasis>fundamental-type</emphasis>,
177 <emphasis>file-read-type</emphasis> and arguments. Currently, two
178 fundamental types exist, <literal>text</literal> and
179 <literal>grs</literal>.
184 <term><emphasis>group</emphasis>.recordId:
185 <replaceable>record-id-spec</replaceable></term>
188 Specifies how the records are to be identified when updated. See
189 <xref linkend="locating-records"/>.
194 <term><emphasis>group</emphasis>.database:
195 <replaceable>database</replaceable></term>
198 Specifies the Z39.50 database name.
199 <!-- FIXME - now we can have multiple databases in one server. -H -->
204 <term><emphasis>group</emphasis>.storeKeys:
205 <replaceable>boolean</replaceable></term>
208 Specifies whether key information should be saved for a given
209 group of records. If you plan to update/delete this type of
210 records later this should be specified as 1; otherwise it
211 should be 0 (default), to save register space.
212 <!-- ### this is the first mention of "register" -->
213 See <xref linkend="file-ids"/>.
218 <term><emphasis>group</emphasis>.storeData:
219 <replaceable>boolean</replaceable></term>
222 Specifies whether the records should be stored internally
223 in the Zebra system files.
224 If you want to maintain the raw records yourself,
225 this option should be false (0).
226 If you want Zebra to take care of the records for you, it
232 <!-- ### probably a better place to define "register" -->
233 <term>register: <replaceable>register-location</replaceable></term>
236 Specifies the location of the various register files that Zebra uses
237 to represent your databases.
238 See <xref linkend="register-location"/>.
243 <term>shadow: <replaceable>register-location</replaceable></term>
246 Enables the <emphasis>safe update</emphasis> facility of Zebra, and
247 tells the system where to place the required, temporary files.
248 See <xref linkend="shadow-registers"/>.
253 <term>lockDir: <replaceable>directory</replaceable></term>
256 Directory in which various lock files are stored.
261 <term>keyTmpDir: <replaceable>directory</replaceable></term>
264 Directory in which temporary files used during zebraidx's update
270 <term>setTmpDir: <replaceable>directory</replaceable></term>
273 Specifies the directory that the server uses for temporary result sets.
274 If not specified <literal>/tmp</literal> will be used.
279 <term>profilePath: <replaceable>path</replaceable></term>
282 Specifies a path of profile specification files.
283 The path is composed of one or more directories separated by
284 colon. Similar to PATH for UNIX systems.
289 <term>attset: <replaceable>filename</replaceable></term>
292 Specifies the filename(s) of attribute set files for use in
293 searching. At least the Bib-1 set should be loaded
294 (<literal>bib1.att</literal>).
295 The <literal>profilePath</literal> setting is used to look for
297 See <xref linkend="attset-files"/>
302 <term>memMax: <replaceable>size</replaceable></term>
305 Specifies <replaceable>size</replaceable> of internal memory
306 to use for the zebraidx program.
307 The amount is given in megabytes - default is 4 (4 MB).
308 The more memory, the faster large updates happen, up to about
309 half the free memory available on the computer.
314 <term>tempfiles: <replaceable>Yes/Auto/No</replaceable></term>
317 Tells zebra if it should use temporary files when indexing. The
318 default is Auto, in which case zebra uses temporary files only
319 if it would need more that <replaceable>memMax</replaceable>
320 megabytes of memory. This should be good for most uses.
326 <term>root: <replaceable>dir</replaceable></term>
329 Specifies a directory base for Zebra. All relative paths
330 given (in profilePath, register, shadow) are based on this
331 directory. This setting is useful if your Zebra server
332 is running in a different directory from where
333 <literal>zebra.cfg</literal> is located.
339 <term>passwd: <replaceable>file</replaceable></term>
342 Specifies a file with description of user accounts for Zebra.
343 The format is similar to that known to Apache's htpasswd files
344 and UNIX' passwd files. Non-empty lines not beginning with
345 # are considered account lines. There is one account per-line.
346 A line consists of fields separate by a single colon character.
347 First field is username, second is password.
353 <term>passwd.c: <replaceable>file</replaceable></term>
356 Specifies a file with description of user accounts for Zebra.
357 File format is similar to that used by the passwd directive except
358 that the password are encrypted. Use Apache's htpasswd or similar
365 <term>perm.<replaceable>user</replaceable>:
366 <replaceable>permstring</replaceable></term>
369 Specifies permissions (priviledge) for a user that are allowed
370 to access Zebra via the passwd system. There are two kinds
371 of permissions currently: read (r) and write(w). By default
372 users not listed in a permission directive are given the read
373 priviledge. To specify permissions for a user with no
374 username, or Z39.50 anonymous style use
375 <literal>anonymous</literal>. The permstring consists of
376 a sequence of characters. Include character <literal>w</literal>
377 for write/update access, <literal>r</literal> for read access.
383 <term>dbaccess <replaceable>accessfile</replaceable></term>
386 Names a file which lists database subscriptions for individual users.
387 The access file should consists of lines of the form <literal>username:
388 dbnames</literal>, where dbnames is a list of database names, seprated by
389 '+'. No whitespace is allowed in the database list.
399 <sect1 id="locating-records">
400 <title>Locating Records</title>
403 The default behavior of the Zebra system is to reference the
404 records from their original location, i.e. where they were found when you
405 ran <literal>zebraidx</literal>.
406 That is, when a client wishes to retrieve a record
407 following a search operation, the files are accessed from the place
408 where you originally put them - if you remove the files (without
409 running <literal>zebraidx</literal> again, the server will return
410 diagnostic number 14 (``System error in presenting records'') to
415 If your input files are not permanent - for example if you retrieve
416 your records from an outside source, or if they were temporarily
417 mounted on a CD-ROM drive,
418 you may want Zebra to make an internal copy of them. To do this,
419 you specify 1 (true) in the <literal>storeData</literal> setting. When
420 the Z39.50 server retrieves the records they will be read from the
421 internal file structures of the system.
426 <sect1 id="simple-indexing">
427 <title>Indexing with no Record IDs (Simple Indexing)</title>
430 If you have a set of records that are not expected to change over time
431 you may can build your database without record IDs.
432 This indexing method uses less space than the other methods and
437 To use this method, you simply omit the <literal>recordId</literal> entry
438 for the group of files that you index. To add a set of records you use
439 <literal>zebraidx</literal> with the <literal>update</literal> command. The
440 <literal>update</literal> command will always add all of the records that it
441 encounters to the index - whether they have already been indexed or
442 not. If the set of indexed files change, you should delete all of the
443 index files, and build a new index from scratch.
447 Consider a system in which you have a group of text files called
448 <literal>simple</literal>.
449 That group of records should belong to a Z39.50 database called
450 <literal>textbase</literal>.
451 The following <literal>zebra.cfg</literal> file will suffice:
456 profilePath: /usr/local/idzebra/tab
458 simple.recordType: text
459 simple.database: textbase
465 Since the existing records in an index can not be addressed by their
466 IDs, it is impossible to delete or modify records when using this method.
471 <sect1 id="file-ids">
472 <title>Indexing with File Record IDs</title>
475 If you have a set of files that regularly change over time: Old files
476 are deleted, new ones are added, or existing files are modified, you
477 can benefit from using the <emphasis>file ID</emphasis>
478 indexing methodology.
479 Examples of this type of database might include an index of WWW
480 resources, or a USENET news spool area.
481 Briefly speaking, the file key methodology uses the directory paths
482 of the individual records as a unique identifier for each record.
483 To perform indexing of a directory with file keys, again, you specify
484 the top-level directory after the <literal>update</literal> command.
485 The command will recursively traverse the directories and compare
486 each one with whatever have been indexed before in that same directory.
487 If a file is new (not in the previous version of the directory) it
488 is inserted into the registers; if a file was already indexed and
489 it has been modified since the last update, the index is also
490 modified; if a file has been removed since the last
491 visit, it is deleted from the index.
495 The resulting system is easy to administrate. To delete a record you
496 simply have to delete the corresponding file (say, with the
497 <literal>rm</literal> command). And to add records you create new
498 files (or directories with files). For your changes to take effect
499 in the register you must run <literal>zebraidx update</literal> with
500 the same directory root again. This mode of operation requires more
501 disk space than simpler indexing methods, but it makes it easier for
502 you to keep the index in sync with a frequently changing set of data.
503 If you combine this system with the <emphasis>safe update</emphasis>
504 facility (see below), you never have to take your server off-line for
505 maintenance or register updating purposes.
509 To enable indexing with pathname IDs, you must specify
510 <literal>file</literal> as the value of <literal>recordId</literal>
511 in the configuration file. In addition, you should set
512 <literal>storeKeys</literal> to <literal>1</literal>, since the Zebra
513 indexer must save additional information about the contents of each record
514 in order to modify the indexes correctly at a later time.
518 FIXME - There must be a simpler way to do this with Adams string tags -H
522 For example, to update records of group <literal>esdd</literal>
524 <literal>/data1/records/</literal> you should type:
526 $ zebraidx -g esdd update /data1/records
531 The corresponding configuration file includes:
534 esdd.recordType: grs.sgml
540 <para>You cannot start out with a group of records with simple
541 indexing (no record IDs as in the previous section) and then later
542 enable file record Ids. Zebra must know from the first time that you
544 the files should be indexed with file record IDs.
549 You cannot explicitly delete records when using this method (using the
550 <literal>delete</literal> command to <literal>zebraidx</literal>. Instead
551 you have to delete the files from the file system (or move them to a
553 and then run <literal>zebraidx</literal> with the
554 <literal>update</literal> command.
556 <!-- ### what happens if a file contains multiple records? -->
559 <sect1 id="generic-ids">
560 <title>Indexing with General Record IDs</title>
563 When using this method you construct an (almost) arbitrary, internal
564 record key based on the contents of the record itself and other system
565 information. If you have a group of records that explicitly associates
566 an ID with each record, this method is convenient. For example, the
567 record format may contain a title or a ID-number - unique within the group.
568 In either case you specify the Z39.50 attribute set and use-attribute
569 location in which this information is stored, and the system looks at
570 that field to determine the identity of the record.
574 As before, the record ID is defined by the <literal>recordId</literal>
575 setting in the configuration file. The value of the record ID specification
576 consists of one or more tokens separated by whitespace. The resulting
577 ID is represented in the index by concatenating the tokens and
578 separating them by ASCII value (1).
582 There are three kinds of tokens:
586 <term>Internal record info</term>
589 The token refers to a key that is
590 extracted from the record. The syntax of this token is
591 <literal>(</literal> <emphasis>set</emphasis> <literal>,</literal>
592 <emphasis>use</emphasis> <literal>)</literal>,
593 where <emphasis>set</emphasis> is the
594 attribute set name <emphasis>use</emphasis> is the
595 name or value of the attribute.
600 <term>System variable</term>
603 The system variables are preceded by
608 and immediately followed by the system variable name, which
621 <term>database</term>
624 Current database specified.
641 <term>Constant string</term>
644 A string used as part of the ID — surrounded
645 by single- or double quotes.
653 For instance, the sample GILS records that come with the Zebra
654 distribution contain a unique ID in the data tagged Control-Identifier.
655 The data is mapped to the Bib-1 use attribute Identifier-standard
656 (code 1007). To use this field as a record id, specify
657 <literal>(bib1,Identifier-standard)</literal> as the value of the
658 <literal>recordId</literal> in the configuration file.
659 If you have other record types that uses the same field for a
660 different purpose, you might add the record type
661 (or group or database name) to the record id of the gils
662 records as well, to prevent matches with other types of records.
663 In this case the recordId might be set like this:
666 gils.recordId: $type (bib1,Identifier-standard)
672 (see <xref linkend="record-model-grs"/>
673 for details of how the mapping between elements of your records and
674 searchable attributes is established).
678 As for the file record ID case described in the previous section,
679 updating your system is simply a matter of running
680 <literal>zebraidx</literal>
681 with the <literal>update</literal> command. However, the update with general
682 keys is considerably slower than with file record IDs, since all files
683 visited must be (re)read to discover their IDs.
687 As you might expect, when using the general record IDs
688 method, you can only add or modify existing records with the
689 <literal>update</literal> command.
690 If you wish to delete records, you must use the,
691 <literal>delete</literal> command, with a directory as a parameter.
692 This will remove all records that match the files below that root
698 <sect1 id="register-location">
699 <title>Register Location</title>
702 Normally, the index files that form dictionaries, inverted
703 files, record info, etc., are stored in the directory where you run
704 <literal>zebraidx</literal>. If you wish to store these, possibly large,
705 files somewhere else, you must add the <literal>register</literal>
706 entry to the <literal>zebra.cfg</literal> file.
707 Furthermore, the Zebra system allows its file
708 structures to span multiple file systems, which is useful for
709 managing very large databases.
713 The value of the <literal>register</literal> setting is a sequence
714 of tokens. Each token takes the form:
717 <emphasis>dir</emphasis><literal>:</literal><emphasis>size</emphasis>.
720 The <emphasis>dir</emphasis> specifies a directory in which index files
721 will be stored and the <emphasis>size</emphasis> specifies the maximum
722 size of all files in that directory. The Zebra indexer system fills
723 each directory in the order specified and use the next specified
724 directories as needed.
725 The <emphasis>size</emphasis> is an integer followed by a qualifier
727 <literal>b</literal> for bytes,
728 <literal>k</literal> for kilobytes.
729 <literal>M</literal> for megabytes,
730 <literal>G</literal> for gigabytes.
734 For instance, if you have allocated two disks for your register, and
735 the first disk is mounted
736 on <literal>/d1</literal> and has 2GB of free space and the
737 second, mounted on <literal>/d2</literal> has 3.6 GB, you could
738 put this entry in your configuration file:
741 register: /d1:2G /d2:3600M
747 Note that Zebra does not verify that the amount of space specified is
748 actually available on the directory (file system) specified - it is
749 your responsibility to ensure that enough space is available, and that
750 other applications do not attempt to use the free space. In a large
751 production system, it is recommended that you allocate one or more
752 file system exclusively to the Zebra register files.
757 <sect1 id="shadow-registers">
758 <title>Safe Updating - Using Shadow Registers</title>
761 <title>Description</title>
764 The Zebra server supports <emphasis>updating</emphasis> of the index
765 structures. That is, you can add, modify, or remove records from
766 databases managed by Zebra without rebuilding the entire index.
767 Since this process involves modifying structured files with various
768 references between blocks of data in the files, the update process
769 is inherently sensitive to system crashes, or to process interruptions:
770 Anything but a successfully completed update process will leave the
771 register files in an unknown state, and you will essentially have no
772 recourse but to re-index everything, or to restore the register files
773 from a backup medium.
774 Further, while the update process is active, users cannot be
775 allowed to access the system, as the contents of the register files
776 may change unpredictably.
780 You can solve these problems by enabling the shadow register system in
782 During the updating procedure, <literal>zebraidx</literal> will temporarily
783 write changes to the involved files in a set of "shadow
784 files", without modifying the files that are accessed by the
785 active server processes. If the update procedure is interrupted by a
786 system crash or a signal, you simply repeat the procedure - the
787 register files have not been changed or damaged, and the partially
788 written shadow files are automatically deleted before the new updating
793 At the end of the updating procedure (or in a separate operation, if
794 you so desire), the system enters a "commit mode". First,
795 any active server processes are forced to access those blocks that
796 have been changed from the shadow files rather than from the main
797 register files; the unmodified blocks are still accessed at their
798 normal location (the shadow files are not a complete copy of the
799 register files - they only contain those parts that have actually been
800 modified). If the commit process is interrupted at any point during the
801 commit process, the server processes will continue to access the
802 shadow files until you can repeat the commit procedure and complete
803 the writing of data to the main register files. You can perform
804 multiple update operations to the registers before you commit the
805 changes to the system files, or you can execute the commit operation
806 at the end of each update operation. When the commit phase has
807 completed successfully, any running server processes are instructed to
808 switch their operations to the new, operational register, and the
809 temporary shadow files are deleted.
815 <title>How to Use Shadow Register Files</title>
818 The first step is to allocate space on your system for the shadow
820 You do this by adding a <literal>shadow</literal> entry to the
821 <literal>zebra.cfg</literal> file.
822 The syntax of the <literal>shadow</literal> entry is exactly the
823 same as for the <literal>register</literal> entry
824 (see <xref linkend="register-location"/>).
825 The location of the shadow area should be
826 <emphasis>different</emphasis> from the location of the main register
827 area (if you have specified one - remember that if you provide no
828 <literal>register</literal> setting, the default register area is the
829 working directory of the server and indexing processes).
833 The following excerpt from a <literal>zebra.cfg</literal> file shows
834 one example of a setup that configures both the main register
835 location and the shadow file area.
836 Note that two directories or partitions have been set aside
837 for the shadow file area. You can specify any number of directories
838 for each of the file areas, but remember that there should be no
839 overlaps between the directories used for the main registers and the
840 shadow files, respectively.
846 shadow: /scratch1:100M /scratch2:200M
852 When shadow files are enabled, an extra command is available at the
853 <literal>zebraidx</literal> command line.
854 In order to make changes to the system take effect for the
855 users, you'll have to submit a "commit" command after a
856 (sequence of) update operation(s).
862 $ zebraidx update /d1/records
869 Or you can execute multiple updates before committing the changes:
875 $ zebraidx -g books update /d1/records /d2/more-records
876 $ zebraidx -g fun update /d3/fun-records
883 If one of the update operations above had been interrupted, the commit
884 operation on the last line would fail: <literal>zebraidx</literal>
885 will not let you commit changes that would destroy the running register.
886 You'll have to rerun all of the update operations since your last
887 commit operation, before you can commit the new changes.
891 Similarly, if the commit operation fails, <literal>zebraidx</literal>
892 will not let you start a new update operation before you have
893 successfully repeated the commit operation.
894 The server processes will keep accessing the shadow files rather
895 than the (possibly damaged) blocks of the main register files
896 until the commit operation has successfully completed.
900 You should be aware that update operations may take slightly longer
901 when the shadow register system is enabled, since more file access
902 operations are involved. Further, while the disk space required for
903 the shadow register data is modest for a small update operation, you
904 may prefer to disable the system if you are adding a very large number
905 of records to an already very large database (we use the terms
906 <emphasis>large</emphasis> and <emphasis>modest</emphasis>
907 very loosely here, since every application will have a
908 different perception of size).
909 To update the system without the use of the the shadow files,
910 simply run <literal>zebraidx</literal> with the <literal>-n</literal>
911 option (note that you do not have to execute the
912 <emphasis>commit</emphasis> command of <literal>zebraidx</literal>
913 when you temporarily disable the use of the shadow registers in
915 Note also that, just as when the shadow registers are not enabled,
916 server processes will be barred from accessing the main register
917 while the update procedure takes place.
925 <sect1 id="administration-ranking">
926 <title>Relevance Ranking and Sorting of Result Sets</title>
929 The default ordering of a result set is left up to the server,
930 which inside Zebra means sorting in ascending document ID order.
931 This is not always the order humans want to browse the sometimes
932 quite large hit sets. Ranking and sorting comes to the rescue.
936 In case a good presentation ordering can be computed at
937 indexing time, we can use a fixed <literal>static ranking</literal>
938 scheme, which is provided for the <literal>alvis</literal>
939 indexing filter. This defines a fixed ordering of hit lists,
940 independently of the query issued.
944 There are cases, however, where relevance of hit set documents is
945 hghly dependent on the query processed.
946 Simply put, <literal>dynamic relevance ranking</literal>
947 sortes a set of retrieved
949 that those most likely to be relevant to your request are
951 Internally, Zebra retrieves all documents ID's that satisfy your
952 search query, and re-orders the hit list to arrange them based on
953 a measurement of similarity between your query and the content of
958 Finally, there are situations where hit sets of documents should be
959 <literal>sorted</literal> during query time according to the
960 lexicographical ordering of certain sort indexes created at
966 <sect2 id="administration-ranking-static">
967 <title>Static Ranking</title>
970 Zebra uses internally inverted indexes to look up term occurencies
971 in documents. Multiple queries from different indexes can be
972 combined by the binary boolean operations <literal>AND</literal>,
973 <literal>OR</literal> and/or <literal>NOT</literal> (which
974 is in fact a binary <literal>AND NOT</literal> operation).
975 To ensure fast query execution
976 speed, all indexes have to be sorted in the same order.
979 The indexes are normally sorted according to document
980 <literal>ID</literal> in
981 ascending order, and any query which does not invoke a special
982 re-ranking function will therefore retrieve the result set in
984 <literal>ID</literal>
992 directive in the main core Zebra config file, the internal document
993 keys used for ordering are augmented by a preceeding integer, which
994 contains the static rank of a given document, and the index lists
996 first by ascending static rank,
997 then by ascending document <literal>ID</literal>.
1000 This implies that the default rank <literal>0</literal>
1001 is the best rank at the
1002 beginning of the list, and <literal>max int</literal>
1003 is the worst static rank.
1006 The experimental <literal>alvis</literal> filter provides a
1007 directive to fetch static rank information out of the indexed XML
1008 records, thus making <emphasis>all</emphasis> hit sets orderd
1009 after <emphasis>ascending</emphasis> static
1010 rank, and for those doc's which have the same static rank, ordered
1011 after <emphasis>ascending</emphasis> doc <literal>ID</literal>.
1012 See <xref linkend="record-model-alvisxslt"/> for the glory details.
1017 <sect2 id="administration-ranking-dynamic">
1018 <title>Dynamic Ranking</title>
1020 If one wants to do a little fiddeling with the static rank order,
1021 one has to invoke additional re-ranking/re-ordering using dynamic
1022 reranking or score functions. These functions return positive
1023 interger scores, where <emphasis>highest</emphasis> score is
1024 <emphasis>best</emphasis>, which means that the
1025 hit sets will be sorted according to
1026 <emphasis>decending</emphasis>
1028 to the index lists which are sorted according to
1029 <emphasis>ascending</emphasis> rank number and document ID).
1032 Those are in the zebra config file enabled by a directive like (use
1033 only one of these a time!):
1035 rank: rank-1 # default TDF-IDF like
1036 rank: rank-static # dummy do-nothing
1037 rank: zvrank # configurable, experimental TDF-IDF like
1039 Notice that the <literal>rank-1</literal> and
1040 <literal>zvrank</literal> do not use the static rank
1041 information in the list keys, and will produce the same ordering
1042 with our without static ranking enabled.
1045 The dummy <literal>rank-static</literal> reranking/scoring
1046 function returns just
1047 <literal>score = max int - staticrank</literal>
1048 in order to preserve the ordering of hit sets with and without it's
1050 Obviously, to combine static and dynamic ranking usefully, one wants
1051 to make a new ranking
1052 function, which is left
1053 as an exercise for the reader.
1058 Invoking dynamic ranking is done in query time (this is why we
1059 call it 'dynamic ranking' in the first place ..). One has to add
1060 the Bib-1 relation attribute with
1061 value "relevance" to the PQF query (that is, <literal>@attr
1062 2=102</literal>, see also
1063 <ulink url="ftp://ftp.loc.gov/pub/z3950/defs/bib1.txt">
1064 The BIB-1 Attribute Set Semantics</ulink>).
1065 To find all articles with the word 'Eoraptor' in
1066 the title, and present them relevance ranked, one issues the PQF query:
1068 Z> f @attr 2=102 @attr 1=4 Eoraptor
1073 The default <literal>rank-1</literal> ranking module implements a
1074 TF-IDF (Term Frequecy over Inverse Document Frequency) like algorithm.
1078 It is possible to apply dynamic ranking on parts of the PQF query
1081 Z> f @and @attr 2=102 @attr 1=1010 Utah @attr 1=1018 Springer
1083 searches for all documents which have the term 'Utah' on the
1084 body of text, and which have the term 'Springer' in the publisher
1085 field, and sort them in the order of the relvance ranking made on
1086 the body of text index only.
1089 Rank weight is a way to pass a value to a ranking algorithm - so that
1090 one APT has one value - while another as a different one. For
1092 search for 'utah' in use attribute set 'title' with weight 30, as
1093 well as in use attribute set 'any' with weight 20.
1095 Z> f @attr 2=102 @or @attr 9=30 @attr 1=4 utah @attr 9=20 utah
1100 The rank weight feature is experimental. It may change in future
1101 releases of zebra, and is not production mature.
1106 Notice that <literal>dynamic ranking</literal> can be enabled in
1107 sever side CQL query expansion by adding <literal>@attr
1108 2=102</literal> to the CQL config file. For example
1110 relationModifier.relevant = 2=102
1112 invokes dynamik ranking each time a CQL query of the form
1115 Z> f alvis.text =/relevant house
1117 is issued. Dynamic ranking can be enabled on specific CQL indexes
1118 by (for example) setting
1120 index.alvis.text = 1=text 2=102
1122 which then invokes dynamik ranking each time a CQL query of the form
1125 Z> f alvis.text = house
1133 <sect2 id="administration-ranking-sorting">
1134 <title>Sorting</title>
1136 Sorting is enabled in the configuration of record indexing. For
1137 example, to enable sorting according to the BIB-1
1138 <literal>Date/time-added-to-db</literal> field, one could add the line
1140 xelm /*/@created Date/time-added-to-db:s
1142 to any <literal>.abs</literal> record indexing config file, or
1143 similarily, one could add an indexing element of the form
1145 <z:index name="date-modified" type="s">
1146 <xsl:value-of select="some/xpath"/>
1149 to any <literal>alvis</literal> indexing rule.
1152 To trigger a sorting on a pre-defined sorting index of type
1153 <literal>s</literal>, we can issue a sort with BIB-1
1154 embedded sort attribute set <literal>7</literal>.
1155 The embedded sort is a way to specify sort within a query - thus
1156 removing the need to send a Z39.50 <literal>Sort
1157 Request</literal> separately.
1160 The value after attribute type <literal>7</literal> is
1161 <literal>1</literal> (=ascending), or <literal>2</literal>
1163 The attributes+term (APT) node is separate from the rest of the
1164 PQF query, and must be <literal>@or</literal>'ed.
1165 The term associated with this attribute is the sorting level,
1167 <literal>0</literal> specifies the primary sort key,
1168 <literal>1</literal> the secondary sort key, and so on.
1170 <para>For example, a search for water, sort by title (ascending),
1171 is expressed by the PQF query
1173 Z> f @or @attr 1=1016 water @attr 7=1 @attr 1=4 0
1175 whereas a search for water, sort by title ascending,
1176 then date descending would be
1178 Z> f @or @or @attr 1=1016 water @attr 7=1 @attr 1=4 0 @attr 7=2 @attr 1=30 1
1182 Notice the fundamental differences between <literal>dynamic
1183 ranking</literal> and <literal>sorting</literal>: there can only
1184 be one ranking function defined and configured, but there can be
1185 specified multiple sorting indexes dynamically at search
1186 time. Ranking does not need to use specific indexes, which means,
1187 dynamic ranking can be enabled and disabled without
1188 re-indexing. On the other hand, sorting indexes need to be
1189 defined before indexing.
1197 <sect1 id="administration-extended-services">
1198 <title>Extended Services: Remote Insert, Update and Delete</title>
1201 The extended services are not enabled by default in zebra - due to the
1202 fact that they modify the system.
1203 In order to allow anybody to update, use
1207 in the main zebra configuration file <filename>zebra.cfg</filename>.
1208 Or, even better, allow only updates for a particular admin user. For
1209 user <literal>admin</literal>, you could use:
1212 passwd: passwordfile
1214 And in <filename>passwordfile</filename>, specify users and
1215 passwords as colon seperated strings:
1221 We can now start a yaz-client admin session and create a database:
1224 $ yaz-client localhost:9999 -u admin/secret
1228 Now the <literal>Default</literal> database was created,
1229 we can insert an XML file (esdd0006.grs
1230 from example/gils/records) and index it:
1233 Z> update insert 1 esdd0006.grs
1236 The 3rd parameter - <literal>1</literal> here -
1237 is the opaque record ID from <literal>Ext update</literal>.
1238 It a record ID that <emphasis>we</emphasis> assign to the record
1239 in question. If we do not
1240 assign one, the usual rules for match apply (recordId: from zebra.cfg).
1243 Actually, we should have a way to specify "no opaque record id" for
1244 yaz-client's update command.. We'll fix that.
1247 The newly inserted record can be searched as usual:
1252 Received SearchResponse.
1253 Search was a success.
1254 Number of hits: 1, setno 1
1255 SearchResult-1: term=utah cnt=1
1262 Let's delete the beast:
1266 No last record (update ignored)
1267 Z> update delete 1 esdd0006.grs
1268 Got extended services response
1273 Received SearchResponse.
1274 Search was a success.
1275 Number of hits: 0, setno 2
1276 SearchResult-1: term=utah cnt=0
1283 If shadow register is enabled in your
1284 <filename>zebra.cfg</filename>,
1285 you must run the adm-commit command
1291 after each update session in order write your changes from the
1292 shadow to the life register space.
1295 Extended services are also available from the YAZ client layer. An
1296 example of an YAZ-PHP extended service transaction is given here:
1299 $record = '<record><title>A fine specimen of a record</title></record>';
1301 $options = array('action' => 'recordInsert',
1303 'record' => $record,
1304 'databaseName' => 'mydatabase'
1307 yaz_es($yaz, 'update', $options);
1308 yaz_es($yaz, 'commit', array());
1311 if ($error = yaz_error($yaz))
1315 The <literal>action</literal> parameter can be any of
1316 <literal>recordInsert</literal> (will fail if the record already exists),
1317 <literal>recordReplace</literal> (will fail if the record does not exist),
1318 <literal>recordDelete</literal> (will fail if the record does not
1320 <literal>specialUpdate</literal> (will insert or update the record
1324 If a record is inserted
1325 using the action <literal>recordInsert</literal>
1326 one can specify the optional
1327 <literal>recordIdOpaque</literal> parameter, which is a
1328 client-supplied, opaque record identifier. This identifier will
1329 replace zebra's own automagic identifier generation.
1332 When using the action <literal>recordReplace</literal> or
1333 <literal>recordDelete</literal>, one must specify the additional
1334 <literal>recordIdNumber</literal> parameter, which must be an
1335 existing Zebra internal system ID number. When retrieving existing
1336 records, the ID number is returned in the field
1337 <literal>/*/id:idzebra/localnumber</literal> in the namespace
1338 <literal>xmlns:id="http://www.indexdata.dk/zebra/"</literal>,
1339 where it can be picked up for later record updates or deletes.
1344 <sect1 id="gfs-config">
1345 <title>YAZ Frontend Virtual Hosts</title>
1347 <command>zebrasrv</command> uses the YAZ server frontend and does
1348 support multiple virtual servers behind multiple listening sockets.
1353 Section "Virtual Hosts" in the YAZ manual.
1354 <filename>http://www.indexdata.dk/yaz/doc/server.vhosts.tkl</filename>
1359 <sect1 id="administration-cql-to-pqf">
1360 <title>Server Side CQL to PQF Query Translation</title>
1363 <literal><cql2rpn>l2rpn.txt</cql2rpn></literal>
1364 YAZ Frontend Virtual
1365 Hosts option, one can configure
1366 the YAZ Frontend CQL-to-PQF
1367 converter, specifying the interpretation of various
1368 <ulink url="http://www.loc.gov/standards/sru/cql/">CQL</ulink>
1369 indexes, relations, etc. in terms of Type-1 query attributes.
1370 <!-- The yaz-client config file -->
1373 For example, using server-side CQL-to-PQF conversion, one might
1374 query a zebra server like this:
1377 yaz-client localhost:9999
1379 Z> find text=(plant and soil)
1382 and - if properly configured - even static relevance ranking can
1383 be performed using CQL query syntax:
1386 Z> find text = /relevant (plant and soil)
1392 By the way, the same configuration can be used to
1393 search using client-side CQL-to-PQF conversion:
1394 (the only difference is <literal>querytype cql2rpn</literal>
1396 <literal>querytype cql</literal>, and the call specifying a local
1400 yaz-client -q local/cql2pqf.txt localhost:9999
1401 Z> querytype cql2rpn
1402 Z> find text=(plant and soil)
1408 Exhaustive information can be found in the
1409 Section "Specification of CQL to RPN mappings" in the YAZ manual.
1410 <ulink url="http://www.indexdata.dk/yaz/doc/tools.tkl#tools.cql.map">
1411 http://www.indexdata.dk/yaz/doc/tools.tkl#tools.cql.map</ulink>,
1412 and shall therefore not be repeated here.
1417 <ulink url="http://www.loc.gov/z3950/agency/zing/cql/dc-indexes.html">
1418 http://www.loc.gov/z3950/agency/zing/cql/dc-indexes.html</ulink>
1419 for the Maintenance Agency's work-in-progress mapping of Dublin Core
1420 indexes to Attribute Architecture (util, XD and BIB-2)
1430 <!-- Keep this comment at the end of the file
1435 sgml-minimize-attributes:nil
1436 sgml-always-quote-attributes:t
1439 sgml-parent-document: "zebra.xml"
1440 sgml-local-catalogs: nil
1441 sgml-namecase-general:t