+
+ <!--
+ <sect3 id="administration-ranking-dynamic-rank1">
+ <title>Dynamically ranking &acro.pqf; queries with the 'rank-static'
+ algorithm</title>
+ <para>
+ The dummy <literal>rank-static</literal> reranking/scoring
+ function returns just
+ <literal>score = max int - staticrank</literal>
+ in order to preserve the static ordering of hit sets that would
+ have been produced had it not been invoked.
+ Obviously, to combine static and dynamic ranking usefully,
+ it is necessary
+ to make a new ranking
+ function; this is left
+ as an exercise for the reader.
+ </para>
+ </sect3>
+ -->
+
+ <warning>
+ <para>
+ <literal>Dynamic ranking</literal> is not compatible
+ with <literal>estimated hit sizes</literal>, as all documents in
+ a hit set must be accessed to compute the correct placing in a
+ ranking sorted list. Therefore the use attribute setting
+ <literal>@attr 2=102</literal> clashes with
+ <literal>@attr 9=integer</literal>.
+ </para>
+ </warning>
+
+ <!--
+ we might want to add ranking like this:
+ UNPUBLISHED:
+ Simple BM25 Extension to Multiple Weighted Fields
+ Stephen Robertson, Hugo Zaragoza and Michael Taylor
+ Microsoft Research
+ ser@microsoft.com
+ hugoz@microsoft.com
+ mitaylor2microsoft.com
+ -->
+
+ </sect3>
+
+ <sect3 id="administration-ranking-dynamic-cql">
+ <title>Dynamically ranking &acro.cql; queries</title>
+ <para>
+ Dynamic ranking can be enabled during sever side &acro.cql;
+ query expansion by adding <literal>@attr 2=102</literal>
+ chunks to the &acro.cql; config file. For example
+ <screen>
+ relationModifier.relevant = 2=102
+ </screen>
+ invokes dynamic ranking each time a &acro.cql; query of the form
+ <screen>
+ Z> querytype cql
+ Z> f alvis.text =/relevant house
+ </screen>
+ is issued. Dynamic ranking can also be automatically used on
+ specific &acro.cql; indexes by (for example) setting
+ <screen>
+ index.alvis.text = 1=text 2=102
+ </screen>
+ which then invokes dynamic ranking each time a &acro.cql; query of the form
+ <screen>
+ Z> querytype cql
+ Z> f alvis.text = house
+ </screen>
+ is issued.
+ </para>
+
+ </sect3>
+
+ </sect2>
+
+
+ <sect2 id="administration-ranking-sorting">
+ <title>Sorting</title>
+ <para>
+ &zebra; sorts efficiently using special sorting indexes
+ (type=<literal>s</literal>; so each sortable index must be known
+ at indexing time, specified in the configuration of record
+ indexing. For example, to enable sorting according to the &acro.bib1;
+ <literal>Date/time-added-to-db</literal> field, one could add the line
+ <screen>
+ xelm /*/@created Date/time-added-to-db:s
+ </screen>
+ to any <literal>.abs</literal> record-indexing configuration file.
+ Similarly, one could add an indexing element of the form
+ <screen><![CDATA[
+ <z:index name="date-modified" type="s">
+ <xsl:value-of select="some/xpath"/>
+ </z:index>
+ ]]></screen>
+ to any <literal>alvis</literal>-filter indexing stylesheet.
+ </para>
+ <para>
+ Indexing can be specified at searching time using a query term
+ carrying the non-standard
+ &acro.bib1; attribute-type <literal>7</literal>. This removes the
+ need to send a &acro.z3950; <literal>Sort Request</literal>
+ separately, and can dramatically improve latency when the client
+ and server are on separate networks.
+ The sorting part of the query is separate from the rest of the
+ query - the actual search specification - and must be combined
+ with it using OR.
+ </para>
+ <para>
+ A sorting subquery needs two attributes: an index (such as a
+ &acro.bib1; type-1 attribute) specifying which index to sort on, and a
+ type-7 attribute whose value is be <literal>1</literal> for
+ ascending sorting, or <literal>2</literal> for descending. The
+ term associated with the sorting attribute is the priority of
+ the sort key, where <literal>0</literal> specifies the primary
+ sort key, <literal>1</literal> the secondary sort key, and so
+ on.
+ </para>
+ <para>For example, a search for water, sort by title (ascending),
+ is expressed by the &acro.pqf; query
+ <screen>
+ @or @attr 1=1016 water @attr 7=1 @attr 1=4 0
+ </screen>
+ whereas a search for water, sort by title ascending,
+ then date descending would be
+ <screen>
+ @or @or @attr 1=1016 water @attr 7=1 @attr 1=4 0 @attr 7=2 @attr 1=30 1
+ </screen>
+ </para>
+ <para>
+ Notice the fundamental differences between <literal>dynamic
+ ranking</literal> and <literal>sorting</literal>: there can be
+ only one ranking function defined and configured; but multiple
+ sorting indexes can be specified dynamically at search
+ time. Ranking does not need to use specific indexes, so
+ dynamic ranking can be enabled and disabled without
+ re-indexing; whereas, sorting indexes need to be
+ defined before indexing.
+ </para>
+
+ </sect2>
+
+
+ </sect1>
+
+ <sect1 id="administration-extended-services">
+ <title>Extended Services: Remote Insert, Update and Delete</title>
+
+ <note>
+ <para>
+ Extended services are only supported when accessing the &zebra;
+ server using the <ulink url="&url.z39.50;">&acro.z3950;</ulink>
+ protocol. The <ulink url="&url.sru;">&acro.sru;</ulink> protocol does
+ not support extended services.
+ </para>
+ </note>
+
+ <para>
+ The extended services are not enabled by default in zebra - due to the
+ fact that they modify the system. &zebra; can be configured
+ to allow anybody to
+ search, and to allow only updates for a particular admin user
+ in the main zebra configuration file <filename>zebra.cfg</filename>.
+ For user <literal>admin</literal>, you could use:
+ <screen>
+ perm.anonymous: r
+ perm.admin: rw
+ passwd: passwordfile
+ </screen>
+ And in the password file
+ <filename>passwordfile</filename>, you have to specify users and
+ encrypted passwords as colon separated strings.
+ Use a tool like <filename>htpasswd</filename>
+ to maintain the encrypted passwords.
+ <screen>
+ admin:secret
+ </screen>
+ It is essential to configure &zebra; to store records internally,
+ and to support
+ modifications and deletion of records:
+ <screen>
+ storeData: 1
+ storeKeys: 1
+ </screen>
+ The general record type should be set to any record filter which
+ is able to parse &acro.xml; records, you may use any of the two
+ declarations (but not both simultaneously!)
+ <screen>
+ recordType: dom.filter_dom_conf.xml
+ # recordType: grs.xml
+ </screen>
+ Notice the difference to the specific instructions
+ <screen>
+ recordType.xml: dom.filter_dom_conf.xml
+ # recordType.xml: grs.xml
+ </screen>
+ which only work when indexing XML files from the filesystem using
+ the <literal>*.xml</literal> naming convention.
+ </para>
+ <para>
+ To enable transaction safe shadow indexing,
+ which is extra important for this kind of operation, set
+ <screen>
+ shadow: directoryname: size (e.g. 1000M)
+ </screen>
+ See <xref linkend="zebra-cfg"/> for additional information on
+ these configuration options.
+ </para>
+ <note>
+ <para>
+ It is not possible to carry information about record types or
+ similar to &zebra; when using extended services, due to
+ limitations of the <ulink url="&url.z39.50;">&acro.z3950;</ulink>
+ protocol. Therefore, indexing filters can not be chosen on a
+ per-record basis. One and only one general &acro.xml; indexing filter
+ must be defined.
+ <!-- but because it is represented as an OID, we would need some
+ form of proprietary mapping scheme between record type strings and
+ OIDs. -->
+ <!--
+ However, as a minimum, it would be extremely useful to enable
+ people to use &acro.marc21;, assuming grs.marcxml.marc21 as a record
+ type.
+ -->
+ </para>
+ </note>
+
+
+ <sect2 id="administration-extended-services-z3950">
+ <title>Extended services in the &acro.z3950; protocol</title>
+
+ <para>
+ The <ulink url="&url.z39.50;">&acro.z3950;</ulink> standard allows
+ servers to accept special binary <emphasis>extended services</emphasis>
+ protocol packages, which may be used to insert, update and delete
+ records into servers. These carry control and update
+ information to the servers, which are encoded in seven package fields:
+ </para>
+
+ <table id="administration-extended-services-z3950-table" frame="top">
+ <title>Extended services &acro.z3950; Package Fields</title>
+ <tgroup cols="3">
+ <thead>
+ <row>
+ <entry>Parameter</entry>
+ <entry>Value</entry>
+ <entry>Notes</entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry><literal>type</literal></entry>
+ <entry><literal>'update'</literal></entry>
+ <entry>Must be set to trigger extended services</entry>
+ </row>
+ <row>
+ <entry><literal>action</literal></entry>
+ <entry><literal>string</literal></entry>
+ <entry>
+ Extended service action type with
+ one of four possible values: <literal>recordInsert</literal>,
+ <literal>recordReplace</literal>,
+ <literal>recordDelete</literal>,
+ and <literal>specialUpdate</literal>
+ </entry>
+ </row>
+ <row>
+ <entry><literal>record</literal></entry>
+ <entry><literal>&acro.xml; string</literal></entry>
+ <entry>An &acro.xml; formatted string containing the record</entry>
+ </row>
+ <row>
+ <entry><literal>syntax</literal></entry>
+ <entry><literal>'xml'</literal></entry>
+ <entry>XML/SUTRS/MARC. GRS-1 not supported.
+ The default filter (record type) as given by recordType in
+ zebra.cfg is used to parse the record.</entry>
+ </row>
+ <row>
+ <entry><literal>recordIdOpaque</literal></entry>
+ <entry><literal>string</literal></entry>
+ <entry>
+ Optional client-supplied, opaque record
+ identifier used under insert operations.
+ </entry>
+ </row>
+ <row>
+ <entry><literal>recordIdNumber </literal></entry>
+ <entry><literal>positive number</literal></entry>
+ <entry>&zebra;'s internal system number,
+ not allowed for <literal>recordInsert</literal> or
+ <literal>specialUpdate</literal> actions which result in fresh
+ record inserts.
+ </entry>
+ </row>
+ <row>
+ <entry><literal>databaseName</literal></entry>
+ <entry><literal>database identifier</literal></entry>
+ <entry>
+ The name of the database to which the extended services should be
+ applied.
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
+
+ <para>
+ The <literal>action</literal> parameter can be any of
+ <literal>recordInsert</literal> (will fail if the record already exists),
+ <literal>recordReplace</literal> (will fail if the record does not exist),
+ <literal>recordDelete</literal> (will fail if the record does not
+ exist), and
+ <literal>specialUpdate</literal> (will insert or update the record
+ as needed, record deletion is not possible).
+ </para>
+
+ <para>
+ During all actions, the
+ usual rules for internal record ID generation apply, unless an
+ optional <literal>recordIdNumber</literal> &zebra; internal ID or a
+ <literal>recordIdOpaque</literal> string identifier is assigned.
+ The default ID generation is
+ configured using the <literal>recordId:</literal> from
+ <filename>zebra.cfg</filename>.
+ See <xref linkend="zebra-cfg"/>.
+ </para>
+
+ <para>
+ Setting of the <literal>recordIdNumber</literal> parameter,
+ which must be an existing &zebra; internal system ID number, is not
+ allowed during any <literal>recordInsert</literal> or
+ <literal>specialUpdate</literal> action resulting in fresh record
+ inserts.
+ </para>
+
+ <para>
+ When retrieving existing
+ records indexed with &acro.grs1; indexing filters, the &zebra; internal
+ ID number is returned in the field
+ <literal>/*/id:idzebra/localnumber</literal> in the namespace
+ <literal>xmlns:id="http://www.indexdata.dk/zebra/"</literal>,
+ where it can be picked up for later record updates or deletes.
+ </para>
+
+ <para>
+ A new element set for retrieval of internal record
+ data has been added, which can be used to access minimal records
+ containing only the <literal>recordIdNumber</literal> &zebra;
+ internal ID, or the <literal>recordIdOpaque</literal> string
+ identifier. This works for any indexing filter used.
+ See <xref linkend="special-retrieval"/>.
+ </para>
+
+ <para>
+ The <literal>recordIdOpaque</literal> string parameter
+ is an client-supplied, opaque record
+ identifier, which may be used under
+ insert, update and delete operations. The
+ client software is responsible for assigning these to
+ records. This identifier will
+ replace zebra's own automagic identifier generation with a unique
+ mapping from <literal>recordIdOpaque</literal> to the
+ &zebra; internal <literal>recordIdNumber</literal>.
+ <emphasis>The opaque <literal>recordIdOpaque</literal> string
+ identifiers
+ are not visible in retrieval records, nor are
+ searchable, so the value of this parameter is
+ questionable. It serves mostly as a convenient mapping from
+ application domain string identifiers to &zebra; internal ID's.
+ </emphasis>
+ </para>
+ </sect2>
+
+
+ <sect2 id="administration-extended-services-yaz-client">
+ <title>Extended services from yaz-client</title>
+
+ <para>
+ We can now start a yaz-client admin session and create a database:
+ <screen>
+ <![CDATA[
+ $ yaz-client localhost:9999 -u admin/secret
+ Z> adm-create
+ ]]>
+ </screen>
+ Now the <literal>Default</literal> database was created,
+ we can insert an &acro.xml; file (esdd0006.grs
+ from example/gils/records) and index it:
+ <screen>
+ <![CDATA[
+ Z> update insert id1234 esdd0006.grs
+ ]]>
+ </screen>
+ The 3rd parameter - <literal>id1234</literal> here -
+ is the <literal>recordIdOpaque</literal> package field.
+ </para>
+ <para>
+ Actually, we should have a way to specify "no opaque record id" for
+ yaz-client's update command.. We'll fix that.
+ </para>
+ <para>
+ The newly inserted record can be searched as usual:
+ <screen>
+ <![CDATA[
+ Z> f utah
+ Sent searchRequest.
+ Received SearchResponse.
+ Search was a success.
+ Number of hits: 1, setno 1
+ SearchResult-1: term=utah cnt=1
+ records returned: 0
+ Elapsed: 0.014179
+ ]]>
+ </screen>
+ </para>
+ <para>
+ Let's delete the beast, using the same
+ <literal>recordIdOpaque</literal> string parameter:
+ <screen>
+ <![CDATA[
+ Z> update delete id1234
+ No last record (update ignored)
+ Z> update delete 1 esdd0006.grs
+ Got extended services response
+ Status: done
+ Elapsed: 0.072441
+ Z> f utah
+ Sent searchRequest.
+ Received SearchResponse.
+ Search was a success.
+ Number of hits: 0, setno 2
+ SearchResult-1: term=utah cnt=0
+ records returned: 0
+ Elapsed: 0.013610
+ ]]>
+ </screen>
+ </para>
+ <para>
+ If shadow register is enabled in your
+ <filename>zebra.cfg</filename>,
+ you must run the adm-commit command
+ <screen>
+ <![CDATA[
+ Z> adm-commit
+ ]]>
+ </screen>
+ after each update session in order write your changes from the
+ shadow to the life register space.
+ </para>
+ </sect2>
+
+
+ <sect2 id="administration-extended-services-yaz-php">
+ <title>Extended services from yaz-php</title>
+
+ <para>
+ Extended services are also available from the &yaz; &acro.php; client layer. An
+ example of an &yaz;-&acro.php; extended service transaction is given here:
+ <screen>
+ <![CDATA[
+ $record = '<record><title>A fine specimen of a record</title></record>';
+
+ $options = array('action' => 'recordInsert',
+ 'syntax' => 'xml',
+ 'record' => $record,
+ 'databaseName' => 'mydatabase'
+ );
+
+ yaz_es($yaz, 'update', $options);
+ yaz_es($yaz, 'commit', array());
+ yaz_wait();
+
+ if ($error = yaz_error($yaz))
+ echo "$error";
+ ]]>
+ </screen>
+ </para>
+ </sect2>
+
+ <sect2 id="administration-extended-services-debugging">
+ <title>Extended services debugging guide</title>
+ <para>
+ When debugging ES over PHP we recomment the following order of tests:
+ </para>
+
+ <itemizedlist>
+ <listitem>
+ <para>
+ Make sure you have a nice record on your filesystem, which you can
+ index from the filesystem by use of the zebraidx command.
+ Do it exactly as you planned, using one of the GRS-1 filters,
+ or the DOMXML filter.
+ When this works, proceed.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Check that your server setup is OK before you even coded one single
+ line PHP using ES.
+ Take the same record form the file system, and send as ES via
+ <literal>yaz-client</literal> like described in
+ <xref linkend="administration-extended-services-yaz-client"/>,
+ and
+ remeber the <literal>-a</literal> option which tells you what
+ goes over the wire! Notice also the section on permissions:
+ try
+ <screen>
+ perm.anonymous: rw
+ </screen>
+ in <literal>zebra.cfg</literal> to make sure you do not run into
+ permission problems (but never expose such an unsecure setup on the
+ internet!!!). Then, make sure to set the general
+ <literal>recordType</literal> instruction, pointing correctly
+ to the GRS-1 filters,
+ or the DOMXML filters.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ If you insist on using the <literal>sysno</literal> in the
+ <literal>recordIdNumber</literal> setting,
+ please make sure you do only updates and deletes. Zebra's internal
+ system number is not allowed for
+ <literal>recordInsert</literal> or
+ <literal>specialUpdate</literal> actions
+ which result in fresh record inserts.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ If <literal>shadow register</literal> is enabled in your
+ <literal>zebra.cfg</literal>, you must remember running the
+ <screen>
+ Z> adm-commit
+ </screen>
+ command as well.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ If this works, then proceed to do the same thing in your PHP script.
+ </para>
+ </listitem>
+ </itemizedlist>
+
+
+ </sect2>
+
+ </sect1>
+