1 <chapter id="introduction">
2 <!-- $Id: introduction.xml,v 1.52 2008-01-30 08:02:10 marc Exp $ -->
3 <title>Introduction</title>
5 <section id="overview">
6 <title>Overview</title>
9 &zebra; is a free, fast, friendly information management system. It can
10 index records in &acro.xml;/&acro.sgml;, &acro.marc;, e-mail archives and many other
11 formats, and quickly find them using a combination of boolean
12 searching and relevance ranking. Search-and-retrieve applications can
13 be written using &acro.api;s in a wide variety of languages, communicating
14 with the &zebra; server using industry-standard information-retrieval
15 protocols or web services.
18 &zebra; is licensed Open Source, and can be
19 deployed by anyone for any purpose without license fees. The C source
20 code is open to anybody to read and change under the GPL license.
23 &zebra; is a networked component which acts as a
24 reliable &acro.z3950; server
25 for both record/document search, presentation, insert, update and
26 delete operations. In addition, it understands the &acro.sru; family of
27 webservices, which exist in &acro.rest; &acro.get;/&acro.post; and truly
31 &zebra; is available as MS Windows 2003 Server (32 bit) self-extracting
32 package as well as GNU/Debian Linux (32 bit and 64 bit) precompiled
33 packages. It has been deployed successfully on other Unix systems,
34 including Sun Sparc, HP Unix, and many variants of Linux and BSD
38 <ulink url="http://www.indexdata.com/zebra/">http://www.indexdata.com/zebra/</ulink>
39 <ulink url="http://ftp.indexdata.dk/pub/zebra/win32/">http://ftp.indexdata.dk/pub/zebra/win32/</ulink>
40 <ulink url="http://ftp.indexdata.dk/pub/zebra/debian/">http://ftp.indexdata.dk/pub/zebra/debian/</ulink>
44 <ulink url="http://indexdata.dk/zebra/">&zebra;</ulink>
45 is a high-performance, general-purpose structured text
46 indexing and retrieval engine. It reads records in a
47 variety of input formats (eg. email, &acro.xml;, &acro.marc;) and provides access
48 to them through a powerful combination of boolean search
49 expressions and relevance-ranked free-text queries.
53 &zebra; supports large databases (tens of millions of records,
54 tens of gigabytes of data). It allows safe, incremental
55 database updates on live systems. Because &zebra; supports
56 the industry-standard information retrieval protocol, &acro.z3950;,
57 you can search &zebra; databases using an enormous variety of
58 programs and toolkits, both commercial and free, which understand
59 this protocol. Application libraries are available to allow
60 bespoke clients to be written in Perl, C, C++, Java, Tcl, Visual
61 Basic, Python, &acro.php; and more - see the
62 <ulink url="&url.zoom;">&acro.zoom; web site</ulink>
63 for more information on some of these client toolkits.
67 This document is an introduction to the &zebra; system. It explains
68 how to compile the software, how to prepare your first database,
69 and how to configure the server to give you the
70 functionality that you need.
74 <section id="features">
75 <title>&zebra; Features Overview</title>
83 <entry><xref linkend=""/></entry>
89 <entry><xref linkend=""/></entry>
95 <entry><xref linkend=""/></entry>
100 <section id="features-document">
101 <title>&zebra; Document Model</title>
103 <table id="table-features-document" frame="top">
104 <title>&zebra; document model</title>
106 <colspec colwidth="1*" colname="feature"/>
107 <colspec colwidth="1*" colname="availability"/>
108 <colspec colwidth="3*" colname="notes"/>
109 <colspec colwidth="2*" colname="references"/>
112 <entry>Feature</entry>
113 <entry>Availability</entry>
115 <entry>Reference</entry>
120 <entry>Complex semi-structured Documents</entry>
121 <entry>&acro.xml; and &acro.grs1; Documents</entry>
122 <entry>Both &acro.xml; and &acro.grs1; documents exhibit a &acro.dom; like internal
123 representation allowing for complex indexing and display rules</entry>
124 <entry><xref linkend="record-model-alvisxslt"/> and
125 <xref linkend="grs"/></entry>
128 <entry>Input document formats</entry>
129 <entry>&acro.xml;, &acro.sgml;, Text, ISO2709 (&acro.marc;)</entry>
131 A system of input filters driven by
132 regular expressions allows most ASCII-based
133 data formats to be easily processed.
134 &acro.sgml;, &acro.xml;, ISO2709 (&acro.marc;), and raw text are also
136 <entry><xref linkend="componentmodules"/></entry>
139 <entry>Document storage</entry>
140 <entry>Index-only, Key storage, Document storage</entry>
141 <entry>Data can be, and usually is, imported
142 into &zebra;'s own storage, but &zebra; can also refer to
143 external files, building and maintaining indexes of "live"
153 <section id="features-search">
154 <title>&zebra; Search Features</title>
156 <table id="table-features-search" frame="top">
157 <title>&zebra; search functionality</title>
159 <colspec colwidth="1*" colname="feature"/>
160 <colspec colwidth="1*" colname="availability"/>
161 <colspec colwidth="3*" colname="notes"/>
162 <colspec colwidth="2*" colname="references"/>
165 <entry>Feature</entry>
166 <entry>Availability</entry>
168 <entry>Reference</entry>
173 <entry>Query languages</entry>
174 <entry>&acro.cql; and &acro.rpn;/&acro.pqf;</entry>
175 <entry>The type-1 Reverse Polish Notation (&acro.rpn;)
176 and its textual representation Prefix Query Format (&acro.pqf;) are
177 supported. The Common Query Language (&acro.cql;) can be configured as
178 a mapping from &acro.cql; to &acro.rpn;/&acro.pqf;</entry>
179 <entry><xref linkend="querymodel-query-languages-pqf"/> and
180 <xref linkend="querymodel-cql-to-pqf"/></entry>
183 <entry>Complex boolean query tree</entry>
184 <entry>&acro.cql; and &acro.rpn;/&acro.pqf;</entry>
185 <entry>Both &acro.cql; and &acro.rpn;/&acro.pqf; allow atomic query parts (&acro.apt;) to
186 be combined into complex boolean query trees</entry>
187 <entry><xref linkend="querymodel-rpn-tree"/></entry>
190 <entry>Field search</entry>
191 <entry>user defined</entry>
192 <entry>Atomic query parts (&acro.apt;) are either general, or
193 directed at user-specified document fields
195 <entry><xref linkend="querymodel-atomic-queries"/>,
196 <xref linkend="querymodel-use-string"/>,
197 <xref linkend="querymodel-bib1-use"/>, and
198 <xref linkend="querymodel-idxpath-use"/></entry>
201 <entry>Data normalization</entry>
202 <entry>user defined</entry>
203 <entry>Data normalization, text tokenization and character
204 mappings can be applied during indexing and searching</entry>
205 <entry><xref linkend="fields-and-charsets"/></entry>
208 <entry>Predefined field types</entry>
209 <entry>user defined</entry>
210 <entry>Data fields can be indexed as phrase, as into word
211 tokenized text, as numeric values, url's, dates, and raw binary
213 <entry><xref linkend="character-map-files"/> and
214 <xref linkend="querymodel-pqf-apt-mapping-structuretype"/>
218 <entry>Regular expression matching</entry>
219 <entry>available</entry>
220 <entry>Full regular expression matching and "approximate
221 matching" (eg. spelling mistake corrections) are handled.</entry>
222 <entry><xref linkend="querymodel-regular"/></entry>
225 <entry>Term truncation</entry>
226 <entry>left, right, left-and-right</entry>
227 <entry>The truncation attribute specifies whether variations of
228 one or more characters are allowed between search term and hit
229 terms, or not. Using non-default truncation attributes will
230 broaden the document hit set of a search query.</entry>
231 <entry><xref linkend="querymodel-bib1-truncation"/></entry>
234 <entry>Fuzzy searches</entry>
235 <entry>Spelling correction</entry>
236 <entry>In addition, fuzzy searches are implemented, where one
237 spelling mistake in search terms is matched</entry>
238 <entry><xref linkend="querymodel-bib1-truncation"/></entry>
245 <section id="features-scan">
246 <title>&zebra; Index Scanning</title>
248 <table id="table-features-scan" frame="top">
249 <title>&zebra; index scanning</title>
251 <colspec colwidth="1*" colname="feature"/>
252 <colspec colwidth="1*" colname="availability"/>
253 <colspec colwidth="3*" colname="notes"/>
254 <colspec colwidth="2*" colname="references"/>
257 <entry>Feature</entry>
258 <entry>Availability</entry>
260 <entry>Reference</entry>
266 <entry>term suggestions</entry>
267 <entry><literal>Scan</literal> on a given named index returns all the
268 indexed terms in lexicographical order near the given start
269 term. This can be used to create drop-down menus and search
271 <entry><xref linkend="querymodel-operation-type-scan"/> and
272 <xref linkend="querymodel-atomic-queries"/>
276 <entry>Facetted browsing</entry>
277 <entry>available</entry>
278 <entry>Zebra 2.1 and allows retrieval of facets for
281 <entry><xref linkend="querymodel-zebra-attr-scan"/></entry>
284 <entry>Drill-down or refine-search</entry>
285 <entry>partially</entry>
286 <entry>scanning in result sets can be used to implement
287 drill-down in search clients</entry>
288 <entry><xref linkend="querymodel-zebra-attr-scan"/></entry>
295 <section id="features-presentation">
296 <title>&zebra; Document Presentation</title>
298 <table id="table-features-presentation" frame="top">
299 <title>&zebra; document presentation</title>
301 <colspec colwidth="1*" colname="feature"/>
302 <colspec colwidth="1*" colname="availability"/>
303 <colspec colwidth="3*" colname="notes"/>
304 <colspec colwidth="2*" colname="references"/>
307 <entry>Feature</entry>
308 <entry>Availability</entry>
310 <entry>Reference</entry>
315 <entry>Hit count</entry>
317 <entry>Search results include at any time the total hit count of a given
318 query, either exact computed, or approximative, in case that the
319 hit count exceeds a possible pre-defined hit set truncation
322 <xref linkend="querymodel-zebra-local-attr-limit"/> and
323 <xref linkend="zebra-cfg"/>
327 <entry>Paged result sets</entry>
329 <entry>Paging of search requests and present/display request
330 can return any successive number of records from any start
331 position in the hit set, i.e. it is trivial to provide search
332 results in successive pages of any size.</entry>
336 <entry>&acro.xml; document transformations</entry>
337 <entry>&acro.xslt; based</entry>
338 <entry> Record presentation can be performed in many
339 pre-defined &acro.xml; data
340 formats, where the original &acro.xml; records are on-the-fly transformed
341 through any preconfigured &acro.xslt; transformation. It is therefore
342 trivial to present records in short/full &acro.xml; views, transforming to
343 RSS, Dublin Core, or other &acro.xml; based data formats, or transform
344 records to XHTML snippets ready for inserting in XHTML pages.</entry>
346 <xref linkend="record-model-alvisxslt-elementset"/></entry>
349 <entry>Binary record transformations</entry>
350 <entry>&acro.marc;, &acro.usmarc;, &acro.marc21; and &acro.marcxml;</entry>
351 <entry>post-filter record transformations</entry>
355 <entry>Record Syntaxes</entry>
357 <entry> Multiple record syntaxes
358 for data retrieval: &acro.grs1;, &acro.sutrs;,
359 &acro.xml;, ISO2709 (&acro.marc;), etc. Records can be mapped between
360 record syntaxes and schemas on the fly.</entry>
364 <entry>&zebra; internal metadata</entry>
366 <entry> &zebra; internal document metadata can be fetched in
367 &acro.sutrs; and &acro.xml; record syntaxes. Those are useful in client
368 applications.</entry>
369 <entry><xref linkend="special-retrieval"/></entry>
372 <entry>&zebra; internal raw record data</entry>
374 <entry> &zebra; internal raw, binary record data can be fetched in
375 &acro.sutrs; and &acro.xml; record syntaxes, leveraging %zebra; to a
376 binary storage system</entry>
377 <entry><xref linkend="special-retrieval"/></entry>
380 <entry>&zebra; internal record field data</entry>
382 <entry> &zebra; internal record field data can be fetched in
383 &acro.sutrs; and &acro.xml; record syntaxes. This makes very fast minimal
384 record data displays possible.</entry>
385 <entry><xref linkend="special-retrieval"/></entry>
392 <section id="features-sort-rank">
393 <title>&zebra; Sorting and Ranking</title>
395 <table id="table-features-sort-rank" frame="top">
396 <title>&zebra; sorting and ranking</title>
398 <colspec colwidth="1*" colname="feature"/>
399 <colspec colwidth="1*" colname="availability"/>
400 <colspec colwidth="3*" colname="notes"/>
401 <colspec colwidth="2*" colname="references"/>
404 <entry>Feature</entry>
405 <entry>Availability</entry>
407 <entry>Reference</entry>
413 <entry>numeric, lexicographic</entry>
414 <entry>Sorting on the basis of alpha-numeric and numeric data
415 is supported. Alphanumeric sorts can be configured for
416 different data encodings and locales for European languages.</entry>
417 <entry><xref linkend="administration-ranking-sorting"/> and
418 <xref linkend="querymodel-zebra-attr-sorting"/></entry>
421 <entry>Combined sorting</entry>
423 <entry>Sorting on the basis of combined sorts  e.g. combinations of
424 ascending/descending sorts of lexicographical/numeric/date field data
426 <entry><xref linkend="administration-ranking-sorting"/></entry>
429 <entry>Relevance ranking</entry>
430 <entry>TF-IDF like</entry>
431 <entry>Relevance-ranking of free-text queries is supported
432 using a TF-IDF like algorithm.</entry>
433 <entry><xref linkend="administration-ranking-dynamic"/></entry>
436 <entry>Static pre-ranking</entry>
438 <entry>Enables pre-index time ranking of documents where hit
439 lists are ordered first by ascending static rank, then by
440 ascending document ID.</entry>
441 <entry><xref linkend="administration-ranking-static"/></entry>
449 <section id="features-updates">
450 <title>&zebra; Live Updates</title>
453 <table id="table-features-updates" frame="top">
454 <title>&zebra; live updates</title>
456 <colspec colwidth="1*" colname="feature"/>
457 <colspec colwidth="1*" colname="availability"/>
458 <colspec colwidth="3*" colname="notes"/>
459 <colspec colwidth="2*" colname="references"/>
462 <entry>Feature</entry>
463 <entry>Availability</entry>
465 <entry>Reference</entry>
470 <entry>Incremental and batch updates</entry>
472 <entry>It is possible to schedule record inserts/updates/deletes in any
473 quantity, from single individual handled records to batch updates
474 in strikes of any size, as well as total re-indexing of all records
475 from file system. </entry>
476 <entry><xref linkend="zebraidx"/></entry>
479 <entry>Remote updates</entry>
480 <entry>&acro.z3950; extended services</entry>
481 <entry>Updates can be performed from remote locations using the
482 &acro.z3950; extended services. Access to extended services can be
483 login-password protected.</entry>
484 <entry><xref linkend="administration-extended-services"/> and
485 <xref linkend="zebra-cfg"/></entry>
488 <entry>Live updates</entry>
489 <entry>transaction based</entry>
490 <entry> Data updates are transaction based and can be performed
491 on running &zebra; systems. Full searchability is preserved
492 during life data update due to use of shadow disk areas for
493 update operations. Multiple update transactions at the same
494 time are lined up, to be performed one after each other. Data
495 integrity is preserved.</entry>
496 <entry><xref linkend="shadow-registers"/></entry>
503 <section id="features-protocol">
504 <title>&zebra; Networked Protocols</title>
506 <table id="table-features-protocol" frame="top">
507 <title>&zebra; networked protocols</title>
509 <colspec colwidth="1*" colname="feature"/>
510 <colspec colwidth="1*" colname="availability"/>
511 <colspec colwidth="3*" colname="notes"/>
512 <colspec colwidth="2*" colname="references"/>
515 <entry>Feature</entry>
516 <entry>Availability</entry>
518 <entry>Reference</entry>
523 <entry>Fundamental operations</entry>
524 <entry>&acro.z3950;/&acro.sru; <literal>explain</literal>,
525 <literal>search</literal>, <literal>scan</literal>, and
526 <literal>update</literal></entry>
528 <entry><xref linkend="querymodel-operation-types"/></entry>
531 <entry>&acro.z3950; protocol support</entry>
533 <entry> Protocol facilities supported are:
534 <literal>init</literal>, <literal>search</literal>,
535 <literal>present</literal> (retrieval),
536 Segmentation (support for very large records),
537 <literal>delete</literal>, <literal>scan</literal>
538 (index browsing), <literal>sort</literal>,
539 <literal>close</literal> and support for the <literal>update</literal>
540 Extended Service to add or replace an existing &acro.xml;
541 record. Piggy-backed presents are honored in the search
542 request. Named result sets are supported.</entry>
543 <entry><xref linkend="protocol-support"/></entry>
546 <entry>Web Service support</entry>
547 <entry>&acro.sru;</entry>
548 <entry> The protocol operations <literal>explain</literal>,
549 <literal>searchRetrieve</literal> and <literal>scan</literal>
550 are supported. <ulink url="&url.cql;">&acro.cql;</ulink> to internal
551 query model &acro.rpn;
552 conversion is supported. Extended RPN queries
553 for search/retrieve and scan are supported.</entry>
554 <entry><xref linkend="zebrasrv-sru-support"/></entry>
561 <section id="features-scalability">
562 <title>&zebra; Data Size and Scalability</title>
564 <table id="table-features-scalability" frame="top">
565 <title>&zebra; data size and scalability</title>
567 <colspec colwidth="1*" colname="feature"/>
568 <colspec colwidth="1*" colname="availability"/>
569 <colspec colwidth="3*" colname="notes"/>
570 <colspec colwidth="2*" colname="references"/>
573 <entry>Feature</entry>
574 <entry>Availability</entry>
576 <entry>Reference</entry>
581 <entry>No of records</entry>
582 <entry>40-60 million</entry>
587 <entry>Data size</entry>
588 <entry>100 GB of record data</entry>
589 <entry>&zebra; based applications have successfully indexed up
590 to 100 GB of record data</entry>
594 <entry>Scale out</entry>
595 <entry>multiple discs</entry>
600 <entry>Performance</entry>
601 <entry><literal>O(n * log N)</literal></entry>
602 <entry> &zebra; query speed and performance is affected roughly by
603 <literal>O(log N)</literal>,
604 where <literal>N</literal> is the total database size, and by
605 <literal>O(n)</literal>, where <literal>n</literal> is the
606 specific query hit set size.</entry>
610 <entry>Average search times</entry>
612 <entry> Even on very large size databases hit rates of 20 queries per
613 seconds with average query answering time of 1 second are possible,
614 provided that the boolean queries are constructed sufficiently
615 precise to result in hit sets of the order of 1000 to 5.000
620 <entry>Large databases</entry>
621 <entry>64 bit file pointers</entry>
622 <entry>64 file pointers assure that register files can extend
623 the 2 GB limit. Logical files can be
624 automatically partitioned over multiple disks, thus allowing for
625 large databases.</entry>
633 <section id="features-platforms">
634 <title>&zebra; Supported Platforms</title>
636 <table id="table-features-platforms" frame="top">
637 <title>&zebra; supported platforms</title>
639 <colspec colwidth="1*" colname="feature"/>
640 <colspec colwidth="1*" colname="availability"/>
641 <colspec colwidth="3*" colname="notes"/>
642 <colspec colwidth="2*" colname="references"/>
645 <entry>Feature</entry>
646 <entry>Availability</entry>
648 <entry>Reference</entry>
655 <entry>GNU Linux (32 and 64bit), journaling Reiser or (better)
657 on disks. NFS file systems are not supported.
658 GNU/Debian Linux packages are available</entry>
659 <entry><xref linkend="installation-debian"/></entry>
663 <entry>tar-ball</entry>
664 <entry>&zebra; is written in portable C, so it runs on most
666 Usual tar-ball install possible on many major Unix systems</entry>
667 <entry><xref linkend="installation-unix"/></entry>
670 <entry>Windows</entry>
671 <entry>NT/2000/2003/XP</entry>
672 <entry>&zebra; runs as well on Windows (NT/2000/2003/XP).
673 Windows installer packages available</entry>
674 <entry><xref linkend="installation-win32"/></entry>
684 <section id="introduction-apps">
685 <title>References and &zebra; based Applications</title>
687 &zebra; has been deployed in numerous applications, in both the
688 academic and commercial worlds, in application domains as diverse
689 as bibliographic catalogues, Geo-spatial information, structured
690 vocabulary browsing, government information locators, civic
691 information systems, environmental observations, museum information
695 Notable applications include the following:
699 <section id="koha-ils">
700 <title>Koha free open-source ILS</title>
702 <ulink url="http://www.koha.org/">Koha</ulink> is a full-featured
703 open-source ILS, initially developed in
704 New Zealand by Katipo Communications Ltd, and first deployed in
705 January of 2000 for Horowhenua Library Trust. It is currently
706 maintained by a team of software providers and library technology
707 staff from around the globe.
710 <ulink url="http://liblime.com/">LibLime</ulink>,
711 a company that is marketing and supporting Koha, adds in
712 the new release of Koha 3.0 the &zebra;
713 database server to drive its bibliographic database.
716 In early 2005, the Koha project development team began looking at
717 ways to improve &acro.marc; support and overcome scalability limitations
718 in the Koha 2.x series. After extensive evaluations of the best
719 of the Open Source textual database engines - including MySQL
720 full-text searching, PostgreSQL, Lucene and Plucene - the team
724 "&zebra; completely eliminates scalability limitations, because it
725 can support tens of millions of records." explained Joshua
726 Ferraro, LibLime's Technology President and Koha's Project
727 Release Manager. "Our performance tests showed search results in
728 under a second for databases with over 5 million records on a
729 modest i386 900Mhz test server."
732 "&zebra; also includes support for true boolean search expressions
733 and relevance-ranked free-text queries, both of which the Koha
734 2.x series lack. &zebra; also supports incremental and safe
735 database updates, which allow on-the-fly record
736 management. Finally, since &zebra; has at its heart the &acro.z3950;
737 protocol, it greatly improves Koha's support for that critical
741 Although the bibliographic database will be moved to &zebra;, Koha
742 3.0 will continue to use a relational SQL-based database design
743 for the 'factual' database. "Relational database managers have
744 their strengths, in spite of their inability to handle large
745 numbers of bibliographic records efficiently," summed up Ferraro,
746 "We're taking the best from both worlds in our redesigned Koha
750 See also LibLime's newsletter article
751 <ulink url="http://www.liblime.com/newsletter/2006/01/features/koha-earns-its-stripes/">
752 Koha Earns its Stripes</ulink>.
757 <section id="kete-dom">
758 <title>Kete Open Source Digital Library and Archiving software</title>
760 <ulink url="http://kete.net.nz/">Kete</ulink> is a digital object
761 management repository, initially developed in
762 New Zealand. Initial development has
763 been a partnership between the Horowhenua Library Trust and
764 Katipo Communications Ltd. funded as part of the Community
765 Partnership Fund in 2006.
766 Kete is purpose built
767 software to enable communities to build their own digital
768 libraries, archives and repositories.
771 It is based on Ruby-on-Rails and MySQL, and integrates the &zebra; server
772 and the &yaz; toolkit for indexing and retrieval of it's content.
773 Zebra is run as separate computer process from the Kete
777 url="http://kete.net.nz/documentation/topics/show/139-managing-zebra">manages
781 Why does Kete wants to use Zebra?? Speed, Scalability and easy
782 integration with Koha. Read their
784 url="http://kete.net.nz/blog/topics/show/44-who-what-why-when-answering-some-of-the-niggly-development-questions">detailled
785 reasoning here.</ulink>
789 <section id="emilda-ils">
790 <title>Emilda open source ILS</title>
792 <ulink url="http://www.emilda.org/">Emilda</ulink>
793 is a complete Integrated Library System, released under the
794 GNU General Public License. It has a
795 full featured Web-OPAC, allowing comprehensive system management
796 from virtually any computer with an Internet connection, has
797 template based layout allowing anyone to alter the visual
798 appearance of Emilda, and is
799 &acro.xml; based language for fast and easy portability to virtually any
801 Currently, Emilda is used at three schools in Espoo, Finland.
804 As a surplus, 100% &acro.marc; compatibility has been achieved using the
805 &zebra; Server from Index Data as backend server.
809 <section id="reindex-ils">
810 <title>ReIndex.Net web based ILS</title>
812 <ulink url="http://www.reindex.net/index.php?lang=en">Reindex.net</ulink>
813 is a netbased library service offering all
814 traditional functions on a very high level plus many new
815 services. Reindex.net is a comprehensive and powerful WEB system
816 based on standards such as &acro.xml; and &acro.z3950;.
817 updates. Reindex supports &acro.marc21;, dan&acro.marc; eller Dublin Core with
821 Reindex.net runs on GNU/Debian Linux with &zebra; and Simpleserver
823 Data for bibliographic data. The relational database system
824 Sybase 9 &acro.xml; is used for
826 Internally &acro.marcxml; is used for bibliographical records. Update
827 utilizes &acro.z3950; extended services.
831 <section id="dads-article-database">
832 <title>DADS - the DTV Article Database
835 DADS is a huge database of more than ten million records, totalling
836 over ten gigabytes of data. The records are metadata about academic
837 journal articles, primarily scientific; about 10% of these
838 metadata records link to the full text of the articles they
839 describe, a body of about a terabyte of information (although the
840 full text is not indexed.)
843 It allows students and researchers at DTU (Danmarks Tekniske
844 Universitet, the Technical College of Denmark) to find and order
845 articles from multiple databases in a single query. The database
846 contains literature on all engineering subjects. It's available
847 on-line through a web gateway, though currently only to registered
851 More information can be found at
852 <ulink url="http://www.dtv.dk/"/> and
853 <ulink url="http://dads.dtv.dk"/>
857 <section id="infonet-eprints">
858 <title>Infonet Eprints</title>
860 The InfoNet Eprints service from the
861 <ulink url="http://www.dtv.dk/">
862 Technical Knowledge Center of Denmark</ulink>
863 provides access to documents stored in
864 eprint/preprint servers and institutional research archives around
865 the world. The service is based on Open Archives Initiative metadata
866 harvesting of selected scientific archives around the world. These
867 open archives offer free and unrestricted access to their contents.
870 Infonet Eprints currently holds 1.4 million records from 16 archives.
871 The online search facility is found at
872 <ulink url="http://preprints.cvt.dk"/>.
876 <section id="alvis-project">
879 The <ulink url="http://www.alvis.info/alvis/">Alvis</ulink> EU
880 project run under the 6th Framework (IST-1-002068-STP)
881 is building a semantic-based peer-to-peer search engine. A
882 consortium of eleven partners from six different European
883 Community countries plus Switzerland and China contribute
884 with expertise in a broad range of specialties including network
885 topologies, routing algorithms, linguistic analysis and
889 The &zebra; information retrieval indexing machine is used inside
890 the Alvis framework to
891 manage huge collections of natural language processed and
892 enhanced &acro.xml; data, coming from a topic relevant web crawl.
893 In this application, &zebra; swallows and manages 37GB of &acro.xml; data
894 in about 4 hours, resulting in search times of fractions of
901 <title>ULS (Union List of Serials)</title>
904 has created a union catalogue for the periodicals of the
905 twenty-one constituent libraries of the University of London and
906 the University of Westminster
907 (<ulink url="http://www.m25lib.ac.uk/ULS/"/>).
908 They have achieved this using an
909 unusual architecture, which they describe as a
910 ``non-distributed virtual union catalogue''.
913 The member libraries send in data files representing their
914 periodicals, including both brief bibliographic data and summary
915 holdings. Then 21 individual &acro.z3950; targets are created, each
916 using &zebra;, and all mounted on the single hardware server.
917 The live service provides a web gateway allowing &acro.z3950; searching
918 of all of the targets or a selection of them. &zebra;'s small
919 footprint allows a relatively modest system to comfortably host
923 More information can be found at
924 <ulink url="http://www.m25lib.ac.uk/ULS/"/>
929 <title>NLI-&acro.z3950; - a Natural Language Interface for Libraries</title>
931 Fernuniversität Hagen in Germany have developed a natural
932 language interface for access to library databases.
934 url="http://ki212.fernuni-hagen.de/nli/NLIintro.html"/> -->
935 In order to evaluate this interface for recall and precision, they
936 chose &zebra; as the basis for retrieval effectiveness. The &zebra;
937 server contains a copy of the GIRT database, consisting of more
938 than 76000 records in &acro.sgml; format (bibliographic records from
939 social science), which are mapped to &acro.marc; for presentation.
942 (GIRT is the German Indexing and Retrieval Testdatabase. It is a
943 standard German-language test database for intelligent indexing
944 and retrieval systems. See
945 <ulink url="http://www.gesis.org/forschung/informationstechnologie/clef-delos.htm"/>)
948 Evaluation will take place as part of the TREC/CLEF campaign 2003
949 <ulink url="http://clef.iei.pi.cnr.it"/>.
950 <!-- or <ulink url="http://www4.eurospider.ch/CLEF/"/> -->
953 For more information, contact Johannes Leveling
954 <email>Johannes.Leveling@FernUni-Hagen.De</email>
958 <section id="various-web-indexes">
959 <title>Various web indexes</title>
961 &zebra; has been used by a variety of institutions to construct
962 indexes of large web sites, typically in the region of tens of
963 millions of pages. In this role, it functions somewhat similarly
964 to the engine of google or altavista, but for a selected intranet
965 or a subset of the whole Web.
968 For example, Liverpool University's web-search facility (see on
970 <ulink url="http://www.liv.ac.uk/"/>
971 and many sub-pages) works by relevance-searching a &zebra; database
972 which is populated by the Harvest-NG web-crawling software.
975 For more information on Liverpool university's intranet search
976 architecture, contact John Gilbertson
977 <email>jgilbert@liverpool.ac.uk</email>
981 has recently modified the Harvest web indexer to use &zebra; as
982 its native repository engine. His comments on the switch over
983 from the old engine are revealing:
986 The first results after some testing with &zebra; are very
987 promising. The tests were done with around 220,000 SOIF files,
988 which occupies 1.6GB of disk space.
991 Building the index from scratch takes around one hour with &zebra;
992 where [old-engine] needs around five hours. While [old-engine]
993 blocks search requests when updating its index, &zebra; can still
994 answer search requests.
996 &zebra; supports incremental indexing which will speed up indexing
1000 While the search time of [old-engine] varies from some seconds
1001 to some minutes depending how expensive the query is, &zebra;
1002 usually takes around one to three seconds, even for expensive
1005 &zebra; can search more than 100 times faster than [old-engine]
1006 and can process multiple search requests simultaneously
1009 I am very happy to see such nice software available under GPL.
1017 <section id="introduction-support">
1018 <title>Support</title>
1020 You can get support for &zebra; from at least three sources.
1023 First, there's the &zebra; web site at
1024 <ulink url="&url.idzebra;"/>,
1025 which always has the most recent version available for download.
1026 If you have a problem with &zebra;, the first thing to do is see
1027 whether it's fixed in the current release.
1030 Second, there's the &zebra; mailing list. Its home page at
1031 <ulink url="&url.idzebra.mailinglist;"/>
1032 includes a complete archive of all messages that have ever been
1033 posted on the list. The &zebra; mailing list is used both for
1034 announcements from the authors (new
1035 releases, bug fixes, etc.) and general discussion. You are welcome
1036 to seek support there. Join by filling the form on the list home page.
1039 Third, it's possible to buy a commercial support contract, with
1040 well defined service levels and response times, from Index Data.
1042 <ulink url="&url.indexdata.support;"/>
1047 <!-- Keep this comment at the end of the file
1052 sgml-minimize-attributes:nil
1053 sgml-always-quote-attributes:t
1056 sgml-parent-document: "zebra.xml"
1057 sgml-local-catalogs: nil
1058 sgml-namecase-general:t