1 <chapter id="record-model-alvisxslt">
2 <!-- $Id: recordmodel-alvisxslt.xml,v 1.17 2007-02-22 15:44:19 marc Exp $ -->
3 <title>ALVIS &xml; Record Model and Filter Module</title>
7 The functionality of this record model has been improved and
8 replaced by the DOM &xml; record model, see
9 <xref linkend="record-model-domxml"/>. The Alvis &xml; record
10 model is considered obsolete, and will eventually be removed
11 from future releases of the &zebra; software.
16 The record model described in this chapter applies to the fundamental,
18 record type <literal>alvis</literal>, introduced in
19 <xref linkend="componentmodulesalvis"/>.
22 <para> This filter has been developed under the
23 <ulink url="http://www.alvis.info/">ALVIS</ulink> project funded by
24 the European Community under the "Information Society Technologies"
29 <section id="record-model-alvisxslt-filter">
30 <title>ALVIS Record Filter</title>
32 The experimental, loadable Alvis &xml;/&xslt; filter module
33 <literal>mod-alvis.so</literal> is packaged in the GNU/Debian package
34 <literal>libidzebra1.4-mod-alvis</literal>.
35 It is invoked by the <filename>zebra.cfg</filename> configuration statement
37 recordtype.xml: alvis.db/filter_alvis_conf.xml
39 In this example on all data files with suffix
40 <filename>*.xml</filename>, where the
41 Alvis &xslt; filter configuration file is found in the
42 path <filename>db/filter_alvis_conf.xml</filename>.
44 <para>The Alvis &xslt; filter configuration file must be
45 valid &xml;. It might look like this (This example is
46 used for indexing and display of &oai; harvested records):
48 <?xml version="1.0" encoding="UTF-8"?>
50 <schema name="identity" stylesheet="xsl/identity.xsl" />
51 <schema name="index" identifier="http://indexdata.dk/zebra/xslt/1"
52 stylesheet="xsl/oai2index.xsl" />
53 <schema name="dc" stylesheet="xsl/oai2dc.xsl" />
54 <!-- use split level 2 when indexing whole &oai; Record lists -->
55 <split level="2"/>
60 All named stylesheets defined inside
61 <literal>schema</literal> element tags
62 are for presentation after search, including
63 the indexing stylesheet (which is a great debugging help). The
64 names defined in the <literal>name</literal> attributes must be
65 unique, these are the literal <literal>schema</literal> or
66 <literal>element set</literal> names used in
67 <ulink url="http://www.loc.gov/standards/sru/srw/">&srw;</ulink>,
68 <ulink url="&url.sru;">&sru;</ulink> and
69 &z3950; protocol queries.
70 The paths in the <literal>stylesheet</literal> attributes
71 are relative to zebras working directory, or absolute to file
75 The <literal><split level="2"/></literal> decides where the
76 &xml; Reader shall split the
77 collections of records into individual records, which then are
78 loaded into &dom;, and have the indexing &xslt; stylesheet applied.
81 There must be exactly one indexing &xslt; stylesheet, which is
82 defined by the magic attribute
83 <literal>identifier="http://indexdata.dk/zebra/xslt/1"</literal>.
86 <section id="record-model-alvisxslt-internal">
87 <title>ALVIS Internal Record Representation</title>
88 <para>When indexing, an &xml; Reader is invoked to split the input
89 files into suitable record &xml; pieces. Each record piece is then
90 transformed to an &xml; &dom; structure, which is essentially the
91 record model. Only &xslt; transformations can be applied during
92 index, search and retrieval. Consequently, output formats are
93 restricted to whatever &xslt; can deliver from the record &xml;
94 structure, be it other &xml; formats, HTML, or plain text. In case
95 you have <literal>libxslt1</literal> running with E&xslt; support,
96 you can use this functionality inside the Alvis
97 filter configuration &xslt; stylesheets.
101 <section id="record-model-alvisxslt-canonical">
102 <title>ALVIS Canonical Indexing Format</title>
103 <para>The output of the indexing &xslt; stylesheets must contain
104 certain elements in the magic
105 <literal>xmlns:z="http://indexdata.dk/zebra/xslt/1"</literal>
106 namespace. The output of the &xslt; indexing transformation is then
107 parsed using &dom; methods, and the contained instructions are
108 performed on the <emphasis>magic elements and their
112 For example, the output of the command
114 xsltproc xsl/oai2index.xsl one-record.xml
116 might look like this:
118 <?xml version="1.0" encoding="UTF-8"?>
119 <z:record xmlns:z="http://indexdata.dk/zebra/xslt/1"
120 z:id="oai:JTRS:CP-3290---Volume-I"
123 <z:index name="oai_identifier" type="0">
124 oai:JTRS:CP-3290---Volume-I</z:index>
125 <z:index name="oai_datestamp" type="0">2004-07-09</z:index>
126 <z:index name="oai_setspec" type="0">jtrs</z:index>
127 <z:index name="dc_all" type="w">
128 <z:index name="dc_title" type="w">Proceedings of the 4th
129 International Conference and Exhibition:
130 World Congress on Superconductivity - Volume I</z:index>
131 <z:index name="dc_creator" type="w">Kumar Krishen and *Calvin
132 Burnham, Editors</z:index>
137 <para>This means the following: From the original &xml; file
138 <literal>one-record.xml</literal> (or from the &xml; record &dom; of the
139 same form coming from a split input file), the indexing
140 stylesheet produces an indexing &xml; record, which is defined by
141 the <literal>record</literal> element in the magic namespace
142 <literal>xmlns:z="http://indexdata.dk/zebra/xslt/1"</literal>.
143 &zebra; uses the content of
144 <literal>z:id="oai:JTRS:CP-3290---Volume-I"</literal> as internal
145 record ID, and - in case static ranking is set - the content of
146 <literal>z:rank="47896"</literal> as static rank. Following the
147 discussion in <xref linkend="administration-ranking"/>
148 we see that this records is internally ordered
149 lexicographically according to the value of the string
150 <literal>oai:JTRS:CP-3290---Volume-I47896</literal>.
151 The type of action performed during indexing is defined by
152 <literal>z:type="update"></literal>, with recognized values
153 <literal>insert</literal>, <literal>update</literal>, and
154 <literal>delete</literal>.
156 <para>In this example, the following literal indexes are constructed:
165 where the indexing type is defined in the
166 <literal>type</literal> attribute
167 (any value from the standard configuration
168 file <filename>default.idx</filename> will do). Finally, any
169 <literal>text()</literal> node content recursively contained
170 inside the <literal>index</literal> will be filtered through the
171 appropriate char map for character normalization, and will be
172 inserted in the index.
175 Specific to this example, we see that the single word
176 <literal>oai:JTRS:CP-3290---Volume-I</literal> will be literal,
177 byte for byte without any form of character normalization,
178 inserted into the index named <literal>oai:identifier</literal>,
180 <literal>Kumar Krishen and *Calvin Burnham, Editors</literal>
181 will be inserted using the <literal>w</literal> character
182 normalization defined in <filename>default.idx</filename> into
183 the index <literal>dc:creator</literal> (that is, after character
184 normalization the index will keep the individual words
185 <literal>kumar</literal>, <literal>krishen</literal>,
186 <literal>and</literal>, <literal>calvin</literal>,
187 <literal>burnham</literal>, and <literal>editors</literal>), and
188 finally both the texts
189 <literal>Proceedings of the 4th International Conference and Exhibition:
190 World Congress on Superconductivity - Volume I</literal>
192 <literal>Kumar Krishen and *Calvin Burnham, Editors</literal>
193 will be inserted into the index <literal>dc:all</literal> using
194 the same character normalization map <literal>w</literal>.
197 Finally, this example configuration can be queried using &pqf;
198 queries, either transported by &z3950;, (here using a yaz-client)
201 Z> open localhost:9999
205 Z> f @attr 1=dc_creator Kumar
206 Z> scan @attr 1=dc_creator adam
208 Z> f @attr 1=dc_title @attr 4=2 "proceeding congress superconductivity"
209 Z> scan @attr 1=dc_title abc
213 extensions <literal>x-pquery</literal> and
214 <literal>x-pScanClause</literal> to
218 http://localhost:9999/?version=1.1&operation=searchRetrieve&x-pquery=%40attr+1%3Ddc_creator+%40attr+4%3D6+%22the
219 http://localhost:9999/?version=1.1&operation=scan&x-pScanClause=@attr+1=dc_date+@attr+4=2+a
222 See <xref linkend="zebrasrv-sru"/> for more information on &sru;/&srw;
223 configuration, and <xref linkend="gfs-config"/> or the &yaz;
224 <ulink url="&url.yaz.cql;">&cql; section</ulink>
225 for the details or the &yaz; frontend server.
228 Notice that there are no <filename>*.abs</filename>,
229 <filename>*.est</filename>, <filename>*.map</filename>, or other &grs1;
230 filter configuration files involves in this process, and that the
231 literal index names are used during search and retrieval.
237 <section id="record-model-alvisxslt-conf">
238 <title>ALVIS Record Model Configuration</title>
241 <section id="record-model-alvisxslt-index">
242 <title>ALVIS Indexing Configuration</title>
244 As mentioned above, there can be only one indexing
245 stylesheet, and configuration of the indexing process is a synonym
246 of writing an &xslt; stylesheet which produces &xml; output containing the
247 magic elements discussed in
248 <xref linkend="record-model-alvisxslt-internal"/>.
249 Obviously, there are million of different ways to accomplish this
250 task, and some comments and code snippets are in order to lead
251 our Padawan's on the right track to the good side of the force.
254 Stylesheets can be written in the <emphasis>pull</emphasis> or
255 the <emphasis>push</emphasis> style: <emphasis>pull</emphasis>
256 means that the output &xml; structure is taken as starting point of
257 the internal structure of the &xslt; stylesheet, and portions of
258 the input &xml; are <emphasis>pulled</emphasis> out and inserted
259 into the right spots of the output &xml; structure. On the other
260 side, <emphasis>push</emphasis> &xslt; stylesheets are recursively
261 calling their template definitions, a process which is commanded
262 by the input &xml; structure, and are triggered to produce some output &xml;
263 whenever some special conditions in the input stylesheets are
264 met. The <emphasis>pull</emphasis> type is well-suited for input
265 &xml; with strong and well-defined structure and semantics, like the
266 following &oai; indexing example, whereas the
267 <emphasis>push</emphasis> type might be the only possible way to
268 sort out deeply recursive input &xml; formats.
271 A <emphasis>pull</emphasis> stylesheet example used to index
272 &oai; harvested records could use some of the following template
276 <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
277 xmlns:z="http://indexdata.dk/zebra/xslt/1"
278 xmlns:oai="http://www.openarchives.org/&oai;/2.0/"
279 xmlns:oai_dc="http://www.openarchives.org/&oai;/2.0/oai_dc/"
280 xmlns:dc="http://purl.org/dc/elements/1.1/"
283 <xsl:output indent="yes" method="xml" version="1.0" encoding="UTF-8"/>
285 <!-- disable all default text node output -->
286 <xsl:template match="text()"/>
288 <!-- match on oai xml record root -->
289 <xsl:template match="/">
290 <z:record z:id="{normalize-space(oai:record/oai:header/oai:identifier)}"
292 <!-- you might want to use z:rank="{some &xslt; function here}" -->
293 <xsl:apply-templates/>
297 <!-- &oai; indexing templates -->
298 <xsl:template match="oai:record/oai:header/oai:identifier">
299 <z:index name="oai_identifier" type="0">
300 <xsl:value-of select="."/>
306 <!-- DC specific indexing templates -->
307 <xsl:template match="oai:record/oai:metadata/oai_dc:dc/dc:title">
308 <z:index name="dc_title" type="w">
309 <xsl:value-of select="."/>
321 that the names and types of the indexes can be defined in the
322 indexing &xslt; stylesheet <emphasis>dynamically according to
323 content in the original &xml; records</emphasis>, which has
324 opportunities for great power and wizardry as well as grande
328 The following excerpt of a <emphasis>push</emphasis> stylesheet
329 <emphasis>might</emphasis>
330 be a good idea according to your strict control of the &xml;
331 input format (due to rigorous checking against well-defined and
332 tight RelaxNG or &xml; Schema's, for example):
335 <xsl:template name="element-name-indexes">
336 <z:index name="{name()}" type="w">
337 <xsl:value-of select="'1'"/>
342 This template creates indexes which have the name of the working
343 node of any input &xml; file, and assigns a '1' to the index.
345 <literal>find @attr 1=xyz 1</literal>
346 finds all files which contain at least one
347 <literal>xyz</literal> &xml; element. In case you can not control
348 which element names the input files contain, you might ask for
349 disaster and bad karma using this technique.
352 One variation over the theme <emphasis>dynamically created
353 indexes</emphasis> will definitely be unwise:
356 <!-- match on oai xml record root -->
357 <xsl:template match="/">
358 <z:record z:type="update">
360 <!-- create dynamic index name from input content -->
361 <xsl:variable name="dynamic_content">
362 <xsl:value-of select="oai:record/oai:header/oai:identifier"/>
365 <!-- create zillions of indexes with unknown names -->
366 <z:index name="{$dynamic_content}" type="w">
367 <xsl:value-of select="oai:record/oai:metadata/oai_dc:dc"/>
374 Don't be tempted to cross
375 the line to the dark side of the force, Padawan; this leads
376 to suffering and pain, and universal
377 disintegration of your project schedule.
381 <section id="record-model-alvisxslt-elementset">
382 <title>ALVIS Exchange Formats</title>
384 An exchange format can be anything which can be the outcome of an
385 &xslt; transformation, as far as the stylesheet is registered in
386 the main Alvis &xslt; filter configuration file, see
387 <xref linkend="record-model-alvisxslt-filter"/>.
388 In principle anything that can be expressed in &xml;, HTML, and
389 TEXT can be the output of a <literal>schema</literal> or
390 <literal>element set</literal> directive during search, as long as
391 the information comes from the
392 <emphasis>original input record &xml; &dom; tree</emphasis>
393 (and not the transformed and <emphasis>indexed</emphasis> &xml;!!).
396 In addition, internal administrative information from the &zebra;
397 indexer can be accessed during record retrieval. The following
398 example is a summary of the possibilities:
401 <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
402 xmlns:z="http://indexdata.dk/zebra/xslt/1"
405 <!-- register internal zebra parameters -->
406 <xsl:param name="id" select="''"/>
407 <xsl:param name="filename" select="''"/>
408 <xsl:param name="score" select="''"/>
409 <xsl:param name="schema" select="''"/>
411 <xsl:output indent="yes" method="xml" version="1.0" encoding="UTF-8"/>
413 <!-- use then for display of internal information -->
414 <xsl:template match="/">
416 <id><xsl:value-of select="$id"/></id>
417 <filename><xsl:value-of select="$filename"/></filename>
418 <score><xsl:value-of select="$score"/></score>
419 <schema><xsl:value-of select="$schema"/></schema>
430 <section id="record-model-alvisxslt-example">
431 <title>ALVIS Filter &oai; Indexing Example</title>
433 The source code tarball contains a working Alvis filter example in
434 the directory <filename>examples/alvis-oai/</filename>, which
435 should get you started.
438 More example data can be harvested from any &oai; compliant server,
439 see details at the &oai;
440 <ulink url="http://www.openarchives.org/">
441 http://www.openarchives.org/</ulink> web site, and the community
443 <ulink url="http://www.openarchives.org/community/index.html">
444 http://www.openarchives.org/community/index.html</ulink>.
447 <ulink url="http://www.oaforum.org/tutorial/">
448 http://www.oaforum.org/tutorial/</ulink>.
459 <!-- Keep this comment at the end of the file
464 sgml-minimize-attributes:nil
465 sgml-always-quote-attributes:t
468 sgml-parent-document: "zebra.xml"
469 sgml-local-catalogs: nil
470 sgml-namecase-general:t