1 <chapter id="administration">
2 <title>Administrating &zebra;</title>
3 <!-- ### It's a bit daft that this chapter (which describes half of
4 the configuration-file formats) is separated from
5 "recordmodel-grs.xml" (which describes the other half) by the
6 instructions on running zebraidx and zebrasrv. Some careful
7 re-ordering is required here.
11 Unlike many simpler retrieval systems, &zebra; supports safe, incremental
12 updates to an existing index.
16 Normally, when &zebra; modifies the index it reads a number of records
18 Depending on your specifications and on the contents of each record
19 one the following events take place for each record:
26 The record is indexed as if it never occurred before.
27 Either the &zebra; system doesn't know how to identify the record or
28 &zebra; can identify the record but didn't find it to be already indexed.
36 The record has already been indexed.
37 In this case either the contents of the record or the location
38 (file) of the record indicates that it has been indexed before.
46 The record is deleted from the index. As in the
47 update-case it must be able to identify the record.
55 Please note that in both the modify- and delete- case the &zebra;
56 indexer must be able to generate a unique key that identifies the record
57 in question (more on this below).
61 To administrate the &zebra; retrieval system, you run the
62 <literal>zebraidx</literal> program.
63 This program supports a number of options which are preceded by a dash,
64 and a few commands (not preceded by dash).
68 Both the &zebra; administrative tool and the &acro.z3950; server share a
69 set of index files and a global configuration file.
70 The name of the configuration file defaults to
71 <literal>zebra.cfg</literal>.
72 The configuration file includes specifications on how to index
73 various kinds of records and where the other configuration files
74 are located. <literal>zebrasrv</literal> and <literal>zebraidx</literal>
75 <emphasis>must</emphasis> be run in the directory where the
76 configuration file lives unless you indicate the location of the
77 configuration file by option <literal>-c</literal>.
80 <sect1 id="record-types">
81 <title>Record Types</title>
84 Indexing is a per-record process, in which either insert/modify/delete
85 will occur. Before a record is indexed search keys are extracted from
86 whatever might be the layout the original record (sgml,html,text, etc..).
87 The &zebra; system currently supports two fundamental types of records:
88 structured and simple text.
89 To specify a particular extraction process, use either the
90 command line option <literal>-t</literal> or specify a
91 <literal>recordType</literal> setting in the configuration file.
96 <sect1 id="zebra-cfg">
97 <title>The &zebra; Configuration File</title>
100 The &zebra; configuration file, read by <literal>zebraidx</literal> and
101 <literal>zebrasrv</literal> defaults to <literal>zebra.cfg</literal>
102 unless specified by <literal>-c</literal> option.
106 You can edit the configuration file with a normal text editor.
107 parameter names and values are separated by colons in the file. Lines
108 starting with a hash sign (<literal>#</literal>) are
113 If you manage different sets of records that share common
114 characteristics, you can organize the configuration settings for each
116 When <literal>zebraidx</literal> is run and you wish to address a
117 given group you specify the group name with the <literal>-g</literal>
119 In this case settings that have the group name as their prefix
120 will be used by <literal>zebraidx</literal>.
121 If no <literal>-g</literal> option is specified, the settings
122 without prefix are used.
126 In the configuration file, the group name is placed before the option
127 name itself, separated by a dot (.). For instance, to set the record type
128 for group <literal>public</literal> to <literal>grs.sgml</literal>
129 (the &acro.sgml;-like format for structured records) you would write:
134 public.recordType: grs.sgml
139 To set the default value of the record type to <literal>text</literal>
150 The available configuration settings are summarized below. They will be
151 explained further in the following sections.
155 FIXME - Didn't Adam make something to have multiple databases in multiple dirs...
163 <emphasis>group</emphasis>
164 .recordType[<emphasis>.name</emphasis>]:
165 <replaceable>type</replaceable>
169 Specifies how records with the file extension
170 <emphasis>name</emphasis> should be handled by the indexer.
171 This option may also be specified as a command line option
172 (<literal>-t</literal>). Note that if you do not specify a
173 <emphasis>name</emphasis>, the setting applies to all files.
174 In general, the record type specifier consists of the elements (each
175 element separated by dot), <emphasis>fundamental-type</emphasis>,
176 <emphasis>file-read-type</emphasis> and arguments. Currently, two
177 fundamental types exist, <literal>text</literal> and
178 <literal>grs</literal>.
183 <term><emphasis>group</emphasis>.recordId:
184 <replaceable>record-id-spec</replaceable></term>
187 Specifies how the records are to be identified when updated. See
188 <xref linkend="locating-records"/>.
193 <term><emphasis>group</emphasis>.database:
194 <replaceable>database</replaceable></term>
197 Specifies the &acro.z3950; database name.
198 <!-- FIXME - now we can have multiple databases in one server. -H -->
203 <term><emphasis>group</emphasis>.storeKeys:
204 <replaceable>boolean</replaceable></term>
207 Specifies whether key information should be saved for a given
208 group of records. If you plan to update/delete this type of
209 records later this should be specified as 1; otherwise it
210 should be 0 (default), to save register space.
211 <!-- ### this is the first mention of "register" -->
212 See <xref linkend="file-ids"/>.
217 <term><emphasis>group</emphasis>.storeData:
218 <replaceable>boolean</replaceable></term>
221 Specifies whether the records should be stored internally
222 in the &zebra; system files.
223 If you want to maintain the raw records yourself,
224 this option should be false (0).
225 If you want &zebra; to take care of the records for you, it
231 <!-- ### probably a better place to define "register" -->
232 <term>register: <replaceable>register-location</replaceable></term>
235 Specifies the location of the various register files that &zebra; uses
236 to represent your databases.
237 See <xref linkend="register-location"/>.
242 <term>shadow: <replaceable>register-location</replaceable></term>
245 Enables the <emphasis>safe update</emphasis> facility of &zebra;, and
246 tells the system where to place the required, temporary files.
247 See <xref linkend="shadow-registers"/>.
252 <term>lockDir: <replaceable>directory</replaceable></term>
255 Directory in which various lock files are stored.
260 <term>keyTmpDir: <replaceable>directory</replaceable></term>
263 Directory in which temporary files used during zebraidx's update
269 <term>setTmpDir: <replaceable>directory</replaceable></term>
272 Specifies the directory that the server uses for temporary result sets.
273 If not specified <literal>/tmp</literal> will be used.
278 <term>profilePath: <replaceable>path</replaceable></term>
281 Specifies a path of profile specification files.
282 The path is composed of one or more directories separated by
283 colon. Similar to <literal>PATH</literal> for UNIX systems.
289 <term>modulePath: <replaceable>path</replaceable></term>
292 Specifies a path of record filter modules.
293 The path is composed of one or more directories separated by
294 colon. Similar to <literal>PATH</literal> for UNIX systems.
295 The 'make install' procedure typically puts modules in
296 <filename>/usr/local/lib/idzebra-2.0/modules</filename>.
302 <term>index: <replaceable>filename</replaceable></term>
305 Defines the filename which holds fields structure
306 definitions. If omitted, the file <filename>default.idx</filename>
308 Refer to <xref linkend="default-idx-file"/> for
315 <term>sortmax: <replaceable>integer</replaceable></term>
318 Specifies the maximum number of records that will be sorted
319 in a result set. If the result set contains more than
320 <replaceable>integer</replaceable> records, records after the
321 limit will not be sorted. If omitted, the default value is
328 <term>staticrank: <replaceable>integer</replaceable></term>
331 Enables whether static ranking is to be enabled (1) or
332 disabled (0). If omitted, it is disabled - corresponding
334 Refer to <xref linkend="administration-ranking-static"/> .
341 <term>estimatehits: <replaceable>integer</replaceable></term>
344 Controls whether &zebra; should calculate approximate hit counts and
345 at which hit count it is to be enabled.
346 A value of 0 disables approximate hit counts.
347 For a positive value approximate hit count is enabled
348 if it is known to be larger than <replaceable>integer</replaceable>.
351 Approximate hit counts can also be triggered by a particular
352 attribute in a query.
353 Refer to <xref linkend="querymodel-zebra-global-attr-limit"/>.
359 <term>attset: <replaceable>filename</replaceable></term>
362 Specifies the filename(s) of attribute set files for use in
363 searching. In many configurations <filename>bib1.att</filename>
364 is used, but that is not required. If Classic Explain
365 attributes is to be used for searching,
366 <filename>explain.att</filename> must be given.
367 The path to att-files in general can be given using
368 <literal>profilePath</literal> setting.
369 See also <xref linkend="attset-files"/>.
374 <term>memMax: <replaceable>size</replaceable></term>
377 Specifies <replaceable>size</replaceable> of internal memory
378 to use for the zebraidx program.
379 The amount is given in megabytes - default is 4 (4 MB).
380 The more memory, the faster large updates happen, up to about
381 half the free memory available on the computer.
386 <term>tempfiles: <replaceable>Yes/Auto/No</replaceable></term>
389 Tells zebra if it should use temporary files when indexing. The
390 default is Auto, in which case zebra uses temporary files only
391 if it would need more that <replaceable>memMax</replaceable>
392 megabytes of memory. This should be good for most uses.
398 <term>root: <replaceable>dir</replaceable></term>
401 Specifies a directory base for &zebra;. All relative paths
402 given (in profilePath, register, shadow) are based on this
403 directory. This setting is useful if your &zebra; server
404 is running in a different directory from where
405 <literal>zebra.cfg</literal> is located.
411 <term>passwd: <replaceable>file</replaceable></term>
414 Specifies a file with description of user accounts for &zebra;.
415 The format is similar to that known to Apache's htpasswd files
416 and UNIX' passwd files. Non-empty lines not beginning with
417 # are considered account lines. There is one account per-line.
418 A line consists of fields separate by a single colon character.
419 First field is username, second is password.
425 <term>passwd.c: <replaceable>file</replaceable></term>
428 Specifies a file with description of user accounts for &zebra;.
429 File format is similar to that used by the passwd directive except
430 that the password are encrypted. Use Apache's htpasswd or similar
437 <term>perm.<replaceable>user</replaceable>:
438 <replaceable>permstring</replaceable></term>
441 Specifies permissions (privilege) for a user that are allowed
442 to access &zebra; via the passwd system. There are two kinds
443 of permissions currently: read (r) and write(w). By default
444 users not listed in a permission directive are given the read
445 privilege. To specify permissions for a user with no
446 username, or &acro.z3950; anonymous style use
447 <literal>anonymous</literal>. The permstring consists of
448 a sequence of characters. Include character <literal>w</literal>
449 for write/update access, <literal>r</literal> for read access and
450 <literal>a</literal> to allow anonymous access through this account.
456 <term>dbaccess: <replaceable>accessfile</replaceable></term>
459 Names a file which lists database subscriptions for individual users.
460 The access file should consists of lines of the form
461 <literal>username: dbnames</literal>, where dbnames is a list of
462 database names, separated by '+'. No whitespace is allowed in the
469 <term>encoding: <replaceable>charsetname</replaceable></term>
472 Tells &zebra; to interpret the terms in Z39.50 queries as
473 having been encoded using the specified character
474 encoding. The default is <literal>ISO-8859-1</literal>; one
475 useful alternative is <literal>UTF-8</literal>.
481 <term>storeKeys: <replaceable>value</replaceable></term>
484 Specifies whether &zebra; keeps a copy of indexed keys.
485 Use a value of 1 to enable; 0 to disable. If storeKeys setting is
486 omitted, it is enabled. Enabled storeKeys
487 are required for updating and deleting records. Disable only
488 storeKeys to save space and only plan to index data once.
494 <term>storeData: <replaceable>value</replaceable></term>
497 Specifies whether &zebra; keeps a copy of indexed records.
498 Use a value of 1 to enable; 0 to disable. If storeData setting is
499 omitted, it is enabled. A storeData setting of 0 (disabled) makes
500 Zebra fetch records from the original locaction in the file
501 system using filename, file offset and file length. For the
502 DOM and ALVIS filter, the storeData setting is ignored.
512 <sect1 id="locating-records">
513 <title>Locating Records</title>
516 The default behavior of the &zebra; system is to reference the
517 records from their original location, i.e. where they were found when you
518 run <literal>zebraidx</literal>.
519 That is, when a client wishes to retrieve a record
520 following a search operation, the files are accessed from the place
521 where you originally put them - if you remove the files (without
522 running <literal>zebraidx</literal> again, the server will return
523 diagnostic number 14 (``System error in presenting records'') to
528 If your input files are not permanent - for example if you retrieve
529 your records from an outside source, or if they were temporarily
530 mounted on a CD-ROM drive,
531 you may want &zebra; to make an internal copy of them. To do this,
532 you specify 1 (true) in the <literal>storeData</literal> setting. When
533 the &acro.z3950; server retrieves the records they will be read from the
534 internal file structures of the system.
539 <sect1 id="simple-indexing">
540 <title>Indexing with no Record IDs (Simple Indexing)</title>
543 If you have a set of records that are not expected to change over time
544 you may can build your database without record IDs.
545 This indexing method uses less space than the other methods and
550 To use this method, you simply omit the <literal>recordId</literal> entry
551 for the group of files that you index. To add a set of records you use
552 <literal>zebraidx</literal> with the <literal>update</literal> command. The
553 <literal>update</literal> command will always add all of the records that it
554 encounters to the index - whether they have already been indexed or
555 not. If the set of indexed files change, you should delete all of the
556 index files, and build a new index from scratch.
560 Consider a system in which you have a group of text files called
561 <literal>simple</literal>.
562 That group of records should belong to a &acro.z3950; database called
563 <literal>textbase</literal>.
564 The following <literal>zebra.cfg</literal> file will suffice:
569 profilePath: /usr/local/idzebra/tab
571 simple.recordType: text
572 simple.database: textbase
578 Since the existing records in an index can not be addressed by their
579 IDs, it is impossible to delete or modify records when using this method.
584 <sect1 id="file-ids">
585 <title>Indexing with File Record IDs</title>
588 If you have a set of files that regularly change over time: Old files
589 are deleted, new ones are added, or existing files are modified, you
590 can benefit from using the <emphasis>file ID</emphasis>
591 indexing methodology.
592 Examples of this type of database might include an index of WWW
593 resources, or a USENET news spool area.
594 Briefly speaking, the file key methodology uses the directory paths
595 of the individual records as a unique identifier for each record.
596 To perform indexing of a directory with file keys, again, you specify
597 the top-level directory after the <literal>update</literal> command.
598 The command will recursively traverse the directories and compare
599 each one with whatever have been indexed before in that same directory.
600 If a file is new (not in the previous version of the directory) it
601 is inserted into the registers; if a file was already indexed and
602 it has been modified since the last update, the index is also
603 modified; if a file has been removed since the last
604 visit, it is deleted from the index.
608 The resulting system is easy to administrate. To delete a record you
609 simply have to delete the corresponding file (say, with the
610 <literal>rm</literal> command). And to add records you create new
611 files (or directories with files). For your changes to take effect
612 in the register you must run <literal>zebraidx update</literal> with
613 the same directory root again. This mode of operation requires more
614 disk space than simpler indexing methods, but it makes it easier for
615 you to keep the index in sync with a frequently changing set of data.
616 If you combine this system with the <emphasis>safe update</emphasis>
617 facility (see below), you never have to take your server off-line for
618 maintenance or register updating purposes.
622 To enable indexing with pathname IDs, you must specify
623 <literal>file</literal> as the value of <literal>recordId</literal>
624 in the configuration file. In addition, you should set
625 <literal>storeKeys</literal> to <literal>1</literal>, since the &zebra;
626 indexer must save additional information about the contents of each record
627 in order to modify the indexes correctly at a later time.
631 FIXME - There must be a simpler way to do this with Adams string tags -H
635 For example, to update records of group <literal>esdd</literal>
637 <literal>/data1/records/</literal> you should type:
639 $ zebraidx -g esdd update /data1/records
644 The corresponding configuration file includes:
647 esdd.recordType: grs.sgml
653 <para>You cannot start out with a group of records with simple
654 indexing (no record IDs as in the previous section) and then later
655 enable file record Ids. &zebra; must know from the first time that you
657 the files should be indexed with file record IDs.
662 You cannot explicitly delete records when using this method (using the
663 <literal>delete</literal> command to <literal>zebraidx</literal>. Instead
664 you have to delete the files from the file system (or move them to a
666 and then run <literal>zebraidx</literal> with the
667 <literal>update</literal> command.
669 <!-- ### what happens if a file contains multiple records? -->
672 <sect1 id="generic-ids">
673 <title>Indexing with General Record IDs</title>
676 When using this method you construct an (almost) arbitrary, internal
677 record key based on the contents of the record itself and other system
678 information. If you have a group of records that explicitly associates
679 an ID with each record, this method is convenient. For example, the
680 record format may contain a title or a ID-number - unique within the group.
681 In either case you specify the &acro.z3950; attribute set and use-attribute
682 location in which this information is stored, and the system looks at
683 that field to determine the identity of the record.
687 As before, the record ID is defined by the <literal>recordId</literal>
688 setting in the configuration file. The value of the record ID specification
689 consists of one or more tokens separated by whitespace. The resulting
690 ID is represented in the index by concatenating the tokens and
691 separating them by ASCII value (1).
695 There are three kinds of tokens:
699 <term>Internal record info</term>
702 The token refers to a key that is
703 extracted from the record. The syntax of this token is
704 <literal>(</literal> <emphasis>set</emphasis> <literal>,</literal>
705 <emphasis>use</emphasis> <literal>)</literal>,
706 where <emphasis>set</emphasis> is the
707 attribute set name <emphasis>use</emphasis> is the
708 name or value of the attribute.
713 <term>System variable</term>
716 The system variables are preceded by
721 and immediately followed by the system variable name, which
734 <term>database</term>
737 Current database specified.
754 <term>Constant string</term>
757 A string used as part of the ID — surrounded
758 by single- or double quotes.
766 For instance, the sample GILS records that come with the &zebra;
767 distribution contain a unique ID in the data tagged Control-Identifier.
768 The data is mapped to the &acro.bib1; use attribute Identifier-standard
769 (code 1007). To use this field as a record id, specify
770 <literal>(bib1,Identifier-standard)</literal> as the value of the
771 <literal>recordId</literal> in the configuration file.
772 If you have other record types that uses the same field for a
773 different purpose, you might add the record type
774 (or group or database name) to the record id of the gils
775 records as well, to prevent matches with other types of records.
776 In this case the recordId might be set like this:
779 gils.recordId: $type (bib1,Identifier-standard)
785 (see <xref linkend="grs"/>
786 for details of how the mapping between elements of your records and
787 searchable attributes is established).
791 As for the file record ID case described in the previous section,
792 updating your system is simply a matter of running
793 <literal>zebraidx</literal>
794 with the <literal>update</literal> command. However, the update with general
795 keys is considerably slower than with file record IDs, since all files
796 visited must be (re)read to discover their IDs.
800 As you might expect, when using the general record IDs
801 method, you can only add or modify existing records with the
802 <literal>update</literal> command.
803 If you wish to delete records, you must use the,
804 <literal>delete</literal> command, with a directory as a parameter.
805 This will remove all records that match the files below that root
811 <sect1 id="register-location">
812 <title>Register Location</title>
815 Normally, the index files that form dictionaries, inverted
816 files, record info, etc., are stored in the directory where you run
817 <literal>zebraidx</literal>. If you wish to store these, possibly large,
818 files somewhere else, you must add the <literal>register</literal>
819 entry to the <literal>zebra.cfg</literal> file.
820 Furthermore, the &zebra; system allows its file
821 structures to span multiple file systems, which is useful for
822 managing very large databases.
826 The value of the <literal>register</literal> setting is a sequence
827 of tokens. Each token takes the form:
829 <emphasis>dir</emphasis><literal>:</literal><emphasis>size</emphasis>
831 The <emphasis>dir</emphasis> specifies a directory in which index files
832 will be stored and the <emphasis>size</emphasis> specifies the maximum
833 size of all files in that directory. The &zebra; indexer system fills
834 each directory in the order specified and use the next specified
835 directories as needed.
836 The <emphasis>size</emphasis> is an integer followed by a qualifier
838 <literal>b</literal> for bytes,
839 <literal>k</literal> for kilobytes.
840 <literal>M</literal> for megabytes,
841 <literal>G</literal> for gigabytes.
842 Specifying a negative value disables the checking (it still needs the unit,
843 use <literal>-1b</literal>).
847 For instance, if you have allocated three disks for your register, and
848 the first disk is mounted
849 on <literal>/d1</literal> and has 2GB of free space, the
850 second, mounted on <literal>/d2</literal> has 3.6 GB, and the third,
851 on which you have more space than you bother to worry about, mounted on
852 <literal>/d3</literal> you could put this entry in your configuration file:
855 register: /d1:2G /d2:3600M /d3:-1b
860 Note that &zebra; does not verify that the amount of space specified is
861 actually available on the directory (file system) specified - it is
862 your responsibility to ensure that enough space is available, and that
863 other applications do not attempt to use the free space. In a large
864 production system, it is recommended that you allocate one or more
865 file system exclusively to the &zebra; register files.
870 <sect1 id="shadow-registers">
871 <title>Safe Updating - Using Shadow Registers</title>
873 <sect2 id="shadow-registers-description">
874 <title>Description</title>
877 The &zebra; server supports <emphasis>updating</emphasis> of the index
878 structures. That is, you can add, modify, or remove records from
879 databases managed by &zebra; without rebuilding the entire index.
880 Since this process involves modifying structured files with various
881 references between blocks of data in the files, the update process
882 is inherently sensitive to system crashes, or to process interruptions:
883 Anything but a successfully completed update process will leave the
884 register files in an unknown state, and you will essentially have no
885 recourse but to re-index everything, or to restore the register files
886 from a backup medium.
887 Further, while the update process is active, users cannot be
888 allowed to access the system, as the contents of the register files
889 may change unpredictably.
893 You can solve these problems by enabling the shadow register system in
895 During the updating procedure, <literal>zebraidx</literal> will temporarily
896 write changes to the involved files in a set of "shadow
897 files", without modifying the files that are accessed by the
898 active server processes. If the update procedure is interrupted by a
899 system crash or a signal, you simply repeat the procedure - the
900 register files have not been changed or damaged, and the partially
901 written shadow files are automatically deleted before the new updating
906 At the end of the updating procedure (or in a separate operation, if
907 you so desire), the system enters a "commit mode". First,
908 any active server processes are forced to access those blocks that
909 have been changed from the shadow files rather than from the main
910 register files; the unmodified blocks are still accessed at their
911 normal location (the shadow files are not a complete copy of the
912 register files - they only contain those parts that have actually been
913 modified). If the commit process is interrupted at any point during the
914 commit process, the server processes will continue to access the
915 shadow files until you can repeat the commit procedure and complete
916 the writing of data to the main register files. You can perform
917 multiple update operations to the registers before you commit the
918 changes to the system files, or you can execute the commit operation
919 at the end of each update operation. When the commit phase has
920 completed successfully, any running server processes are instructed to
921 switch their operations to the new, operational register, and the
922 temporary shadow files are deleted.
927 <sect2 id="shadow-registers-how-to-use">
928 <title>How to Use Shadow Register Files</title>
931 The first step is to allocate space on your system for the shadow
933 You do this by adding a <literal>shadow</literal> entry to the
934 <literal>zebra.cfg</literal> file.
935 The syntax of the <literal>shadow</literal> entry is exactly the
936 same as for the <literal>register</literal> entry
937 (see <xref linkend="register-location"/>).
938 The location of the shadow area should be
939 <emphasis>different</emphasis> from the location of the main register
940 area (if you have specified one - remember that if you provide no
941 <literal>register</literal> setting, the default register area is the
942 working directory of the server and indexing processes).
946 The following excerpt from a <literal>zebra.cfg</literal> file shows
947 one example of a setup that configures both the main register
948 location and the shadow file area.
949 Note that two directories or partitions have been set aside
950 for the shadow file area. You can specify any number of directories
951 for each of the file areas, but remember that there should be no
952 overlaps between the directories used for the main registers and the
953 shadow files, respectively.
959 shadow: /scratch1:100M /scratch2:200M
965 When shadow files are enabled, an extra command is available at the
966 <literal>zebraidx</literal> command line.
967 In order to make changes to the system take effect for the
968 users, you'll have to submit a "commit" command after a
969 (sequence of) update operation(s).
975 $ zebraidx update /d1/records
982 Or you can execute multiple updates before committing the changes:
988 $ zebraidx -g books update /d1/records /d2/more-records
989 $ zebraidx -g fun update /d3/fun-records
996 If one of the update operations above had been interrupted, the commit
997 operation on the last line would fail: <literal>zebraidx</literal>
998 will not let you commit changes that would destroy the running register.
999 You'll have to rerun all of the update operations since your last
1000 commit operation, before you can commit the new changes.
1004 Similarly, if the commit operation fails, <literal>zebraidx</literal>
1005 will not let you start a new update operation before you have
1006 successfully repeated the commit operation.
1007 The server processes will keep accessing the shadow files rather
1008 than the (possibly damaged) blocks of the main register files
1009 until the commit operation has successfully completed.
1013 You should be aware that update operations may take slightly longer
1014 when the shadow register system is enabled, since more file access
1015 operations are involved. Further, while the disk space required for
1016 the shadow register data is modest for a small update operation, you
1017 may prefer to disable the system if you are adding a very large number
1018 of records to an already very large database (we use the terms
1019 <emphasis>large</emphasis> and <emphasis>modest</emphasis>
1020 very loosely here, since every application will have a
1021 different perception of size).
1022 To update the system without the use of the the shadow files,
1023 simply run <literal>zebraidx</literal> with the <literal>-n</literal>
1024 option (note that you do not have to execute the
1025 <emphasis>commit</emphasis> command of <literal>zebraidx</literal>
1026 when you temporarily disable the use of the shadow registers in
1028 Note also that, just as when the shadow registers are not enabled,
1029 server processes will be barred from accessing the main register
1030 while the update procedure takes place.
1038 <sect1 id="administration-ranking">
1039 <title>Relevance Ranking and Sorting of Result Sets</title>
1041 <sect2 id="administration-overview">
1042 <title>Overview</title>
1044 The default ordering of a result set is left up to the server,
1045 which inside &zebra; means sorting in ascending document ID order.
1046 This is not always the order humans want to browse the sometimes
1047 quite large hit sets. Ranking and sorting comes to the rescue.
1051 In cases where a good presentation ordering can be computed at
1052 indexing time, we can use a fixed <literal>static ranking</literal>
1053 scheme, which is provided for the <literal>alvis</literal>
1054 indexing filter. This defines a fixed ordering of hit lists,
1055 independently of the query issued.
1059 There are cases, however, where relevance of hit set documents is
1060 highly dependent on the query processed.
1061 Simply put, <literal>dynamic relevance ranking</literal>
1062 sorts a set of retrieved records such that those most likely to be
1063 relevant to your request are retrieved first.
1064 Internally, &zebra; retrieves all documents that satisfy your
1065 query, and re-orders the hit list to arrange them based on
1066 a measurement of similarity between your query and the content of
1071 Finally, there are situations where hit sets of documents should be
1072 <literal>sorted</literal> during query time according to the
1073 lexicographical ordering of certain sort indexes created at
1079 <sect2 id="administration-ranking-static">
1080 <title>Static Ranking</title>
1083 &zebra; uses internally inverted indexes to look up term frequencies
1084 in documents. Multiple queries from different indexes can be
1085 combined by the binary boolean operations <literal>AND</literal>,
1086 <literal>OR</literal> and/or <literal>NOT</literal> (which
1087 is in fact a binary <literal>AND NOT</literal> operation).
1088 To ensure fast query execution
1089 speed, all indexes have to be sorted in the same order.
1092 The indexes are normally sorted according to document
1093 <literal>ID</literal> in
1094 ascending order, and any query which does not invoke a special
1095 re-ranking function will therefore retrieve the result set in
1097 <literal>ID</literal>
1105 directive in the main core &zebra; configuration file, the internal document
1106 keys used for ordering are augmented by a preceding integer, which
1107 contains the static rank of a given document, and the index lists
1109 first by ascending static rank,
1110 then by ascending document <literal>ID</literal>.
1112 is the ``best'' rank, as it occurs at the
1113 beginning of the list; higher numbers represent worse scores.
1116 The experimental <literal>alvis</literal> filter provides a
1117 directive to fetch static rank information out of the indexed &acro.xml;
1118 records, thus making <emphasis>all</emphasis> hit sets ordered
1119 after <emphasis>ascending</emphasis> static
1120 rank, and for those doc's which have the same static rank, ordered
1121 after <emphasis>ascending</emphasis> doc <literal>ID</literal>.
1122 See <xref linkend="record-model-alvisxslt"/> for the gory details.
1127 <sect2 id="administration-ranking-dynamic">
1128 <title>Dynamic Ranking</title>
1130 In order to fiddle with the static rank order, it is necessary to
1131 invoke additional re-ranking/re-ordering using dynamic
1132 ranking or score functions. These functions return positive
1133 integer scores, where <emphasis>highest</emphasis> score is
1135 hit sets are sorted according to <emphasis>descending</emphasis>
1137 to the index lists which are sorted according to
1138 ascending rank number and document ID).
1141 Dynamic ranking is enabled by a directive like one of the
1142 following in the zebra configuration file (use only one of these a time!):
1144 rank: rank-1 # default TDF-IDF like
1145 rank: rank-static # dummy do-nothing
1150 Dynamic ranking is done at query time rather than
1151 indexing time (this is why we
1152 call it ``dynamic ranking'' in the first place ...)
1153 It is invoked by adding
1154 the &acro.bib1; relation attribute with
1155 value ``relevance'' to the &acro.pqf; query (that is,
1156 <literal>@attr 2=102</literal>, see also
1157 <ulink url="&url.z39.50;bib1.html">
1158 The &acro.bib1; Attribute Set Semantics</ulink>, also in
1159 <ulink url="&url.z39.50.attset.bib1;">HTML</ulink>).
1160 To find all articles with the word <literal>Eoraptor</literal> in
1161 the title, and present them relevance ranked, issue the &acro.pqf; query:
1163 @attr 2=102 @attr 1=4 Eoraptor
1167 <sect3 id="administration-ranking-dynamic-rank1">
1168 <title>Dynamically ranking using &acro.pqf; queries with the 'rank-1'
1172 The default <literal>rank-1</literal> ranking module implements a
1173 TF/IDF (Term Frequecy over Inverse Document Frequency) like
1174 algorithm. In contrast to the usual definition of TF/IDF
1175 algorithms, which only considers searching in one full-text
1176 index, this one works on multiple indexes at the same time.
1178 &zebra; does boolean queries and searches in specific addressed
1179 indexes (there are inverted indexes pointing from terms in the
1180 dictionary to documents and term positions inside documents).
1184 <term>Query Components</term>
1187 First, the boolean query is dismantled into its principal components,
1188 i.e. atomic queries where one term is looked up in one index.
1189 For example, the query
1191 @attr 2=102 @and @attr 1=1010 Utah @attr 1=1018 Springer
1193 is a boolean AND between the atomic parts
1195 @attr 2=102 @attr 1=1010 Utah
1199 @attr 2=102 @attr 1=1018 Springer
1201 which gets processed each for itself.
1207 <term>Atomic hit lists</term>
1210 Second, for each atomic query, the hit list of documents is
1214 In this example, two hit lists for each index
1215 <literal>@attr 1=1010</literal> and
1216 <literal>@attr 1=1018</literal> are computed.
1222 <term>Atomic scores</term>
1225 Third, each document in the hit list is assigned a score (_if_ ranking
1226 is enabled and requested in the query) using a TF/IDF scheme.
1229 In this example, both atomic parts of the query assign the magic
1230 <literal>@attr 2=102</literal> relevance attribute, and are
1231 to be used in the relevance ranking functions.
1234 It is possible to apply dynamic ranking on only parts of the
1237 @and @attr 2=102 @attr 1=1010 Utah @attr 1=1018 Springer
1239 searches for all documents which have the term 'Utah' on the
1240 body of text, and which have the term 'Springer' in the publisher
1241 field, and sort them in the order of the relevance ranking made on
1242 the body-of-text index only.
1248 <term>Hit list merging</term>
1251 Fourth, the atomic hit lists are merged according to the boolean
1252 conditions to a final hit list of documents to be returned.
1255 This step is always performed, independently of the fact that
1256 dynamic ranking is enabled or not.
1262 <term>Document score computation</term>
1265 Fifth, the total score of a document is computed as a linear
1266 combination of the atomic scores of the atomic hit lists
1269 Ranking weights may be used to pass a value to a ranking
1270 algorithm, using the non-standard &acro.bib1; attribute type 9.
1271 This allows one branch of a query to use one value while
1272 another branch uses a different one. For example, we can search
1273 for <literal>utah</literal> in the
1274 <literal>@attr 1=4</literal> index with weight 30, as
1275 well as in the <literal>@attr 1=1010</literal> index with weight 20:
1277 @attr 2=102 @or @attr 9=30 @attr 1=4 utah @attr 9=20 @attr 1=1010 city
1281 The default weight is
1282 sqrt(1000) ~ 34 , as the &acro.z3950; standard prescribes that the top score
1283 is 1000 and the bottom score is 0, encoded in integers.
1287 The ranking-weight feature is experimental. It may change in future
1295 <term>Re-sorting of hit list</term>
1298 Finally, the final hit list is re-ordered according to scores.
1306 Still need to describe the exact TF/IDF formula. Here's the info, need -->
1307 <!--to extract it in human readable form .. MC
1309 static int calc (void *set_handle, zint sysno, zint staticrank,
1312 int i, lo, divisor, score = 0;
1313 struct rank_set_info *si = (struct rank_set_info *) set_handle;
1315 if (!si->no_rank_entries)
1316 return -1; /* ranking not enabled for any terms */
1318 for (i = 0; i < si->no_entries; i++)
1320 yaz_log(log_level, "calc: i=%d rank_flag=%d lo=%d",
1321 i, si->entries[i].rank_flag, si->entries[i].local_occur);
1322 if (si->entries[i].rank_flag && (lo = si->entries[i].local_occur))
1323 score += (8+log2_int (lo)) * si->entries[i].global_inv *
1324 si->entries[i].rank_weight;
1326 divisor = si->no_rank_entries * (8+log2_int (si->last_pos/si->no_entries));
1327 score = score / divisor;
1328 yaz_log(log_level, "calc sysno=" ZINT_FORMAT " score=%d", sysno, score);
1331 /* reset the counts for the next term */
1332 for (i = 0; i < si->no_entries; i++)
1333 si->entries[i].local_occur = 0;
1338 where lo = si->entries[i].local_occur is the local documents term-within-index frequency, si->entries[i].global_inv represents the IDF part (computed in static void *begin()), and
1339 si->entries[i].rank_weight is the weight assigner per index (default 34, or set in the @attr 9=xyz magic)
1341 Finally, the IDF part is computed as:
1343 static void *begin (struct zebra_register *reg,
1344 void *class_handle, RSET rset, NMEM nmem,
1345 TERMID *terms, int numterms)
1347 struct rank_set_info *si =
1348 (struct rank_set_info *) nmem_malloc (nmem,sizeof(*si));
1351 yaz_log(log_level, "rank-1 begin");
1352 si->no_entries = numterms;
1353 si->no_rank_entries = 0;
1355 si->entries = (struct rank_term_info *)
1356 nmem_malloc (si->nmem, sizeof(*si->entries)*numterms);
1357 for (i = 0; i < numterms; i++)
1359 zint g = rset_count(terms[i]->rset);
1360 yaz_log(log_level, "i=%d flags=%s '%s'", i,
1361 terms[i]->flags, terms[i]->name );
1362 if (!strncmp (terms[i]->flags, "rank,", 5))
1364 const char *cp = strstr(terms[i]->flags+4, ",w=");
1365 si->entries[i].rank_flag = 1;
1367 si->entries[i].rank_weight = atoi (cp+3);
1369 si->entries[i].rank_weight = 34; /* sqrroot of 1000 */
1370 yaz_log(log_level, " i=%d weight=%d g="ZINT_FORMAT, i,
1371 si->entries[i].rank_weight, g);
1372 (si->no_rank_entries)++;
1375 si->entries[i].rank_flag = 0;
1376 si->entries[i].local_occur = 0; /* FIXME */
1377 si->entries[i].global_occur = g;
1378 si->entries[i].global_inv = 32 - log2_int (g);
1379 yaz_log(log_level, " global_inv = %d g = " ZINT_FORMAT,
1380 (int) (32-log2_int (g)), g);
1381 si->entries[i].term = terms[i];
1382 si->entries[i].term_index=i;
1383 terms[i]->rankpriv = &(si->entries[i]);
1389 where g = rset_count(terms[i]->rset) is the count of all documents in this specific index hit list, and the IDF part then is
1391 si->entries[i].global_inv = 32 - log2_int (g);
1398 The <literal>rank-1</literal> algorithm
1399 does not use the static rank
1400 information in the list keys, and will produce the same ordering
1401 with or without static ranking enabled.
1406 <sect3 id="administration-ranking-dynamic-rank1">
1407 <title>Dynamically ranking &acro.pqf; queries with the 'rank-static'
1410 The dummy <literal>rank-static</literal> reranking/scoring
1411 function returns just
1412 <literal>score = max int - staticrank</literal>
1413 in order to preserve the static ordering of hit sets that would
1414 have been produced had it not been invoked.
1415 Obviously, to combine static and dynamic ranking usefully,
1417 to make a new ranking
1418 function; this is left
1419 as an exercise for the reader.
1426 <literal>Dynamic ranking</literal> is not compatible
1427 with <literal>estimated hit sizes</literal>, as all documents in
1428 a hit set must be accessed to compute the correct placing in a
1429 ranking sorted list. Therefore the use attribute setting
1430 <literal>@attr 2=102</literal> clashes with
1431 <literal>@attr 9=integer</literal>.
1436 we might want to add ranking like this:
1438 Simple BM25 Extension to Multiple Weighted Fields
1439 Stephen Robertson, Hugo Zaragoza and Michael Taylor
1443 mitaylor2microsoft.com
1448 <sect3 id="administration-ranking-dynamic-cql">
1449 <title>Dynamically ranking &acro.cql; queries</title>
1451 Dynamic ranking can be enabled during sever side &acro.cql;
1452 query expansion by adding <literal>@attr 2=102</literal>
1453 chunks to the &acro.cql; config file. For example
1455 relationModifier.relevant = 2=102
1457 invokes dynamic ranking each time a &acro.cql; query of the form
1460 Z> f alvis.text =/relevant house
1462 is issued. Dynamic ranking can also be automatically used on
1463 specific &acro.cql; indexes by (for example) setting
1465 index.alvis.text = 1=text 2=102
1467 which then invokes dynamic ranking each time a &acro.cql; query of the form
1470 Z> f alvis.text = house
1480 <sect2 id="administration-ranking-sorting">
1481 <title>Sorting</title>
1483 &zebra; sorts efficiently using special sorting indexes
1484 (type=<literal>s</literal>; so each sortable index must be known
1485 at indexing time, specified in the configuration of record
1486 indexing. For example, to enable sorting according to the &acro.bib1;
1487 <literal>Date/time-added-to-db</literal> field, one could add the line
1489 xelm /*/@created Date/time-added-to-db:s
1491 to any <literal>.abs</literal> record-indexing configuration file.
1492 Similarly, one could add an indexing element of the form
1494 <z:index name="date-modified" type="s">
1495 <xsl:value-of select="some/xpath"/>
1498 to any <literal>alvis</literal>-filter indexing stylesheet.
1501 Indexing can be specified at searching time using a query term
1502 carrying the non-standard
1503 &acro.bib1; attribute-type <literal>7</literal>. This removes the
1504 need to send a &acro.z3950; <literal>Sort Request</literal>
1505 separately, and can dramatically improve latency when the client
1506 and server are on separate networks.
1507 The sorting part of the query is separate from the rest of the
1508 query - the actual search specification - and must be combined
1512 A sorting subquery needs two attributes: an index (such as a
1513 &acro.bib1; type-1 attribute) specifying which index to sort on, and a
1514 type-7 attribute whose value is be <literal>1</literal> for
1515 ascending sorting, or <literal>2</literal> for descending. The
1516 term associated with the sorting attribute is the priority of
1517 the sort key, where <literal>0</literal> specifies the primary
1518 sort key, <literal>1</literal> the secondary sort key, and so
1521 <para>For example, a search for water, sort by title (ascending),
1522 is expressed by the &acro.pqf; query
1524 @or @attr 1=1016 water @attr 7=1 @attr 1=4 0
1526 whereas a search for water, sort by title ascending,
1527 then date descending would be
1529 @or @or @attr 1=1016 water @attr 7=1 @attr 1=4 0 @attr 7=2 @attr 1=30 1
1533 Notice the fundamental differences between <literal>dynamic
1534 ranking</literal> and <literal>sorting</literal>: there can be
1535 only one ranking function defined and configured; but multiple
1536 sorting indexes can be specified dynamically at search
1537 time. Ranking does not need to use specific indexes, so
1538 dynamic ranking can be enabled and disabled without
1539 re-indexing; whereas, sorting indexes need to be
1540 defined before indexing.
1548 <sect1 id="administration-extended-services">
1549 <title>Extended Services: Remote Insert, Update and Delete</title>
1553 Extended services are only supported when accessing the &zebra;
1554 server using the <ulink url="&url.z39.50;">&acro.z3950;</ulink>
1555 protocol. The <ulink url="&url.sru;">&acro.sru;</ulink> protocol does
1556 not support extended services.
1561 The extended services are not enabled by default in zebra - due to the
1562 fact that they modify the system. &zebra; can be configured
1564 search, and to allow only updates for a particular admin user
1565 in the main zebra configuration file <filename>zebra.cfg</filename>.
1566 For user <literal>admin</literal>, you could use:
1570 passwd: passwordfile
1572 And in the password file
1573 <filename>passwordfile</filename>, you have to specify users and
1574 encrypted passwords as colon separated strings.
1575 Use a tool like <filename>htpasswd</filename>
1576 to maintain the encrypted passwords.
1580 It is essential to configure &zebra; to store records internally,
1582 modifications and deletion of records:
1587 The general record type should be set to any record filter which
1588 is able to parse &acro.xml; records, you may use any of the two
1589 declarations (but not both simultaneously!)
1591 recordType: dom.filter_dom_conf.xml
1592 # recordType: grs.xml
1594 Notice the difference to the specific instructions
1596 recordType.xml: dom.filter_dom_conf.xml
1597 # recordType.xml: grs.xml
1599 which only work when indexing XML files from the filesystem using
1600 the <literal>*.xml</literal> naming convention.
1603 To enable transaction safe shadow indexing,
1604 which is extra important for this kind of operation, set
1606 shadow: directoryname: size (e.g. 1000M)
1608 See <xref linkend="zebra-cfg"/> for additional information on
1609 these configuration options.
1613 It is not possible to carry information about record types or
1614 similar to &zebra; when using extended services, due to
1615 limitations of the <ulink url="&url.z39.50;">&acro.z3950;</ulink>
1616 protocol. Therefore, indexing filters can not be chosen on a
1617 per-record basis. One and only one general &acro.xml; indexing filter
1619 <!-- but because it is represented as an OID, we would need some
1620 form of proprietary mapping scheme between record type strings and
1623 However, as a minimum, it would be extremely useful to enable
1624 people to use &acro.marc21;, assuming grs.marcxml.marc21 as a record
1631 <sect2 id="administration-extended-services-z3950">
1632 <title>Extended services in the &acro.z3950; protocol</title>
1635 The <ulink url="&url.z39.50;">&acro.z3950;</ulink> standard allows
1636 servers to accept special binary <emphasis>extended services</emphasis>
1637 protocol packages, which may be used to insert, update and delete
1638 records into servers. These carry control and update
1639 information to the servers, which are encoded in seven package fields:
1642 <table id="administration-extended-services-z3950-table" frame="top">
1643 <title>Extended services &acro.z3950; Package Fields</title>
1647 <entry>Parameter</entry>
1648 <entry>Value</entry>
1649 <entry>Notes</entry>
1654 <entry><literal>type</literal></entry>
1655 <entry><literal>'update'</literal></entry>
1656 <entry>Must be set to trigger extended services</entry>
1659 <entry><literal>action</literal></entry>
1660 <entry><literal>string</literal></entry>
1662 Extended service action type with
1663 one of four possible values: <literal>recordInsert</literal>,
1664 <literal>recordReplace</literal>,
1665 <literal>recordDelete</literal>,
1666 and <literal>specialUpdate</literal>
1670 <entry><literal>record</literal></entry>
1671 <entry><literal>&acro.xml; string</literal></entry>
1672 <entry>An &acro.xml; formatted string containing the record</entry>
1675 <entry><literal>syntax</literal></entry>
1676 <entry><literal>'xml'</literal></entry>
1677 <entry>XML/SUTRS/MARC. GRS-1 not supported.
1678 The default filter (record type) as given by recordType in
1679 zebra.cfg is used to parse the record.</entry>
1682 <entry><literal>recordIdOpaque</literal></entry>
1683 <entry><literal>string</literal></entry>
1685 Optional client-supplied, opaque record
1686 identifier used under insert operations.
1690 <entry><literal>recordIdNumber </literal></entry>
1691 <entry><literal>positive number</literal></entry>
1692 <entry>&zebra;'s internal system number,
1693 not allowed for <literal>recordInsert</literal> or
1694 <literal>specialUpdate</literal> actions which result in fresh
1699 <entry><literal>databaseName</literal></entry>
1700 <entry><literal>database identifier</literal></entry>
1702 The name of the database to which the extended services should be
1712 The <literal>action</literal> parameter can be any of
1713 <literal>recordInsert</literal> (will fail if the record already exists),
1714 <literal>recordReplace</literal> (will fail if the record does not exist),
1715 <literal>recordDelete</literal> (will fail if the record does not
1717 <literal>specialUpdate</literal> (will insert or update the record
1718 as needed, record deletion is not possible).
1722 During all actions, the
1723 usual rules for internal record ID generation apply, unless an
1724 optional <literal>recordIdNumber</literal> &zebra; internal ID or a
1725 <literal>recordIdOpaque</literal> string identifier is assigned.
1726 The default ID generation is
1727 configured using the <literal>recordId:</literal> from
1728 <filename>zebra.cfg</filename>.
1729 See <xref linkend="zebra-cfg"/>.
1733 Setting of the <literal>recordIdNumber</literal> parameter,
1734 which must be an existing &zebra; internal system ID number, is not
1735 allowed during any <literal>recordInsert</literal> or
1736 <literal>specialUpdate</literal> action resulting in fresh record
1741 When retrieving existing
1742 records indexed with &acro.grs1; indexing filters, the &zebra; internal
1743 ID number is returned in the field
1744 <literal>/*/id:idzebra/localnumber</literal> in the namespace
1745 <literal>xmlns:id="http://www.indexdata.dk/zebra/"</literal>,
1746 where it can be picked up for later record updates or deletes.
1750 A new element set for retrieval of internal record
1751 data has been added, which can be used to access minimal records
1752 containing only the <literal>recordIdNumber</literal> &zebra;
1753 internal ID, or the <literal>recordIdOpaque</literal> string
1754 identifier. This works for any indexing filter used.
1755 See <xref linkend="special-retrieval"/>.
1759 The <literal>recordIdOpaque</literal> string parameter
1760 is an client-supplied, opaque record
1761 identifier, which may be used under
1762 insert, update and delete operations. The
1763 client software is responsible for assigning these to
1764 records. This identifier will
1765 replace zebra's own automagic identifier generation with a unique
1766 mapping from <literal>recordIdOpaque</literal> to the
1767 &zebra; internal <literal>recordIdNumber</literal>.
1768 <emphasis>The opaque <literal>recordIdOpaque</literal> string
1770 are not visible in retrieval records, nor are
1771 searchable, so the value of this parameter is
1772 questionable. It serves mostly as a convenient mapping from
1773 application domain string identifiers to &zebra; internal ID's.
1779 <sect2 id="administration-extended-services-yaz-client">
1780 <title>Extended services from yaz-client</title>
1783 We can now start a yaz-client admin session and create a database:
1786 $ yaz-client localhost:9999 -u admin/secret
1790 Now the <literal>Default</literal> database was created,
1791 we can insert an &acro.xml; file (esdd0006.grs
1792 from example/gils/records) and index it:
1795 Z> update insert id1234 esdd0006.grs
1798 The 3rd parameter - <literal>id1234</literal> here -
1799 is the <literal>recordIdOpaque</literal> package field.
1802 Actually, we should have a way to specify "no opaque record id" for
1803 yaz-client's update command.. We'll fix that.
1806 The newly inserted record can be searched as usual:
1811 Received SearchResponse.
1812 Search was a success.
1813 Number of hits: 1, setno 1
1814 SearchResult-1: term=utah cnt=1
1821 Let's delete the beast, using the same
1822 <literal>recordIdOpaque</literal> string parameter:
1825 Z> update delete id1234
1826 No last record (update ignored)
1827 Z> update delete 1 esdd0006.grs
1828 Got extended services response
1833 Received SearchResponse.
1834 Search was a success.
1835 Number of hits: 0, setno 2
1836 SearchResult-1: term=utah cnt=0
1843 If shadow register is enabled in your
1844 <filename>zebra.cfg</filename>,
1845 you must run the adm-commit command
1851 after each update session in order write your changes from the
1852 shadow to the life register space.
1857 <sect2 id="administration-extended-services-yaz-php">
1858 <title>Extended services from yaz-php</title>
1861 Extended services are also available from the &yaz; &acro.php; client layer. An
1862 example of an &yaz;-&acro.php; extended service transaction is given here:
1865 $record = '<record><title>A fine specimen of a record</title></record>';
1867 $options = array('action' => 'recordInsert',
1869 'record' => $record,
1870 'databaseName' => 'mydatabase'
1873 yaz_es($yaz, 'update', $options);
1874 yaz_es($yaz, 'commit', array());
1877 if ($error = yaz_error($yaz))
1884 <sect2 id="administration-extended-services-debugging">
1885 <title>Extended services debugging guide</title>
1887 When debugging ES over PHP we recommend the following order of tests:
1893 Make sure you have a nice record on your filesystem, which you can
1894 index from the filesystem by use of the zebraidx command.
1895 Do it exactly as you planned, using one of the GRS-1 filters,
1896 or the DOMXML filter.
1897 When this works, proceed.
1902 Check that your server setup is OK before you even coded one single
1904 Take the same record form the file system, and send as ES via
1905 <literal>yaz-client</literal> like described in
1906 <xref linkend="administration-extended-services-yaz-client"/>,
1908 remember the <literal>-a</literal> option which tells you what
1909 goes over the wire! Notice also the section on permissions:
1914 in <literal>zebra.cfg</literal> to make sure you do not run into
1915 permission problems (but never expose such an insecure setup on the
1916 internet!!!). Then, make sure to set the general
1917 <literal>recordType</literal> instruction, pointing correctly
1918 to the GRS-1 filters,
1919 or the DOMXML filters.
1924 If you insist on using the <literal>sysno</literal> in the
1925 <literal>recordIdNumber</literal> setting,
1926 please make sure you do only updates and deletes. Zebra's internal
1927 system number is not allowed for
1928 <literal>recordInsert</literal> or
1929 <literal>specialUpdate</literal> actions
1930 which result in fresh record inserts.
1935 If <literal>shadow register</literal> is enabled in your
1936 <literal>zebra.cfg</literal>, you must remember running the
1945 If this works, then proceed to do the same thing in your PHP script.
1957 <!-- Keep this comment at the end of the file
1962 sgml-minimize-attributes:nil
1963 sgml-always-quote-attributes:t
1966 sgml-parent-document: "zebra.xml"
1967 sgml-local-catalogs: nil
1968 sgml-namecase-general:t