1 <chapter id="introduction">
2 <!-- $Id: introduction.xml,v 1.46 2007-02-05 13:35:12 marc Exp $ -->
3 <title>Introduction</title>
5 <section id="overview">
6 <title>Overview</title>
9 &zebra; is a free, fast, friendly information management system. It can
10 index records in &xml;/&sgml;, &marc;, e-mail archives and many other
11 formats, and quickly find them using a combination of boolean
12 searching and relevance ranking. Search-and-retrieve applications can
13 be written using &api;s in a wide variety of languages, communicating
14 with the &zebra; server using industry-standard information-retrieval
15 protocols or web services.
18 &zebra; is licensed Open Source, and can be
19 deployed by anyone for any purpose without license fees. The C source
20 code is open to anybody to read and change under the GPL license.
23 &zebra; is a networked component which acts as a reliable &z3950; server
24 for both record/document search, presentation, insert, update and
25 delete operations. In addition, it understands the &sru; family of
26 webservices, which exist in &rest; &get;/&post; and truly &soap; flavors.
29 &zebra; is available as MS Windows 2003 Server (32 bit) self-extracting
30 package as well as GNU/Debian Linux (32 bit and 64 bit) precompiled
31 packages. It has been deployed successfully on other Unix systems,
32 including Sun Sparc, HP Unix, and many variants of Linux and BSD
36 <ulink url="http://www.indexdata.com/zebra/">http://www.indexdata.com/zebra/</ulink>
37 <ulink url="http://ftp.indexdata.dk/pub/zebra/win32/">http://ftp.indexdata.dk/pub/zebra/win32/</ulink>
38 <ulink url="http://ftp.indexdata.dk/pub/zebra/debian/">http://ftp.indexdata.dk/pub/zebra/debian/</ulink>
42 <ulink url="http://indexdata.dk/zebra/">&zebra;</ulink>
43 is a high-performance, general-purpose structured text
44 indexing and retrieval engine. It reads records in a
45 variety of input formats (eg. email, &xml;, &marc;) and provides access
46 to them through a powerful combination of boolean search
47 expressions and relevance-ranked free-text queries.
51 &zebra; supports large databases (tens of millions of records,
52 tens of gigabytes of data). It allows safe, incremental
53 database updates on live systems. Because &zebra; supports
54 the industry-standard information retrieval protocol, &z3950;,
55 you can search &zebra; databases using an enormous variety of
56 programs and toolkits, both commercial and free, which understand
57 this protocol. Application libraries are available to allow
58 bespoke clients to be written in Perl, C, C++, Java, Tcl, Visual
59 Basic, Python, &php; and more - see the
60 <ulink url="&url.zoom;">&zoom; web site</ulink>
61 for more information on some of these client toolkits.
65 This document is an introduction to the &zebra; system. It explains
66 how to compile the software, how to prepare your first database,
67 and how to configure the server to give you the
68 functionality that you need.
72 <section id="features">
73 <title>&zebra; Features Overview</title>
81 <entry><xref linkend=""/></entry>
87 <entry><xref linkend=""/></entry>
93 <entry><xref linkend=""/></entry>
98 <section id="features-document">
99 <title>&zebra; Document Model</title>
101 <table id="table-features-document" frame="top">
102 <title>&zebra; document model</title>
106 <entry>Feature</entry>
107 <entry>Availability</entry>
109 <entry>Reference</entry>
114 <entry>Complex semi-structured Documents</entry>
115 <entry>&xml; and &grs1; Documents</entry>
116 <entry>Both &xml; and &grs1; documents exhibit a &dom; like internal
117 representation allowing for complex indexing and display rules</entry>
118 <entry><xref linkend="record-model-alvisxslt"/> and
119 <xref linkend="grs"/></entry>
122 <entry>Input document formats</entry>
123 <entry>&xml;, &sgml;, Text, ISO2709 (&marc;)</entry>
125 A system of input filters driven by
126 regular expressions allows most ASCII-based
127 data formats to be easily processed.
128 &sgml;, &xml;, ISO2709 (&marc;), and raw text are also
130 <entry><xref linkend="componentmodules"/></entry>
133 <entry>Document storage</entry>
134 <entry>Index-only, Key storage, Document storage</entry>
135 <entry>Data can be, and usually is, imported
136 into &zebra;'s own storage, but &zebra; can also refer to
137 external files, building and maintaining indexes of "live"
147 <section id="features-search">
148 <title>&zebra; Search Features</title>
150 <table id="table-features-search" frame="top">
151 <title>&zebra; search functionality</title>
155 <entry>Feature</entry>
156 <entry>Availability</entry>
158 <entry>Reference</entry>
163 <entry>Query languages</entry>
164 <entry>&cql; and &rpn;/&pqf;</entry>
165 <entry>The type-1 Reverse Polish Notation (&rpn;)
166 and it's textual representation Prefix Query Format (&pqf;) are
167 supported. The Common Query Language (&cql;) can be configured as
168 a mapping from &cql; to &rpn;/&pqf;</entry>
169 <entry><xref linkend="querymodel-query-languages-pqf"/> and
170 <xref linkend="querymodel-cql-to-pqf"/></entry>
173 <entry>Complex boolean query tree</entry>
174 <entry>&cql; and &rpn;/&pqf;</entry>
175 <entry>Both &cql; and &rpn;/&pqf; allow atomic query parts (&apt;) to
176 be combined into complex boolean query trees</entry>
177 <entry><xref linkend="querymodel-rpn-tree"/></entry>
180 <entry>Field search</entry>
181 <entry>user defined</entry>
182 <entry>Atomic query parts (&apt;) are either general, or
183 directed at user-specified document fields
185 <entry><xref linkend="querymodel-atomic-queries"/>,
186 <xref linkend="querymodel-use-string"/>,
187 <xref linkend="querymodel-bib1-use"/>, and
188 <xref linkend="querymodel-idxpath-use"/></entry>
191 <entry>Data normalization</entry>
192 <entry>user defined</entry>
193 <entry>Data normalization, text tokenization and character
194 mappings can be applied during indexing and searching</entry>
195 <entry><xref linkend="fields-and-charsets"/></entry>
198 <entry>Predefined field types</entry>
199 <entry>user defined</entry>
200 <entry>Data fields can be indexed as phrase, as into word
201 tokenized text, as numeric values, url's, dates, and raw binary
203 <entry><xref linkend="character-map-files"/> and
204 <xref linkend="querymodel-pqf-apt-mapping-structuretype"/>
208 <entry>Regular expression matching</entry>
209 <entry>available</entry>
210 <entry>Full regular expression matching and "approximate
211 matching" (eg. spelling mistake corrections) are handled.</entry>
212 <entry><xref linkend="querymodel-regular"/></entry>
215 <entry>Term truncation</entry>
216 <entry>left, right, left-and-right</entry>
217 <entry>The truncation attribute specifies whether variations of
218 one or more characters are allowed between search term and hit
219 terms, or not. Using non-default truncation attributes will
220 broaden the document hit set of a search query.</entry>
221 <entry><xref linkend="querymodel-bib1-truncation"/></entry>
224 <entry>Fuzzy searches</entry>
225 <entry>Spelling correction</entry>
226 <entry>In addition, fuzzy searches are implemented, where one
227 spelling mistake in search terms is matched</entry>
228 <entry><xref linkend="querymodel-bib1-truncation"/></entry>
235 <section id="features-scan">
236 <title>&zebra; Index Scanning</title>
238 <table id="table-features-scan" frame="top">
239 <title>&zebra; index scanning</title>
243 <entry>Feature</entry>
244 <entry>Availability</entry>
246 <entry>Reference</entry>
252 <entry>term suggestions</entry>
253 <entry><literal>Scan</literal> on a given named index returns all the
254 indexed terms in lexicographical order near the given start
255 term. This can be used to create drop-down menues and search
257 <entry><xref linkend="querymodel-operation-type-scan"/> and
258 <xref linkend="querymodel-atomic-queries"/>
262 <entry>Facetted browsing</entry>
263 <entry>partial</entry>
264 <entry>&zebra; supports <literal>scan inside a hit
265 set</literal> from a previous search, thus reducing the listed
267 subset of terms found in the documents/records of the hit
269 <entry><xref linkend="querymodel-zebra-attr-scan"/></entry>
272 <entry>Drill-down or refine-search</entry>
273 <entry>partially</entry>
274 <entry>scanning in result sets can be used to implement
275 drill-down in search clients</entry>
276 <entry><xref linkend="querymodel-zebra-attr-scan"/></entry>
283 <section id="features-presentation">
284 <title>&zebra; Document Presentation</title>
286 <table id="table-features-presentation" frame="top">
287 <title>&zebra; document presentation</title>
291 <entry>Feature</entry>
292 <entry>Availability</entry>
294 <entry>Reference</entry>
299 <entry>Hit count</entry>
301 <entry>Search results include at any time the total hit count of a given
302 query, either exact computed, or approximative, in case that the
303 hit count exceeds a possible pre-defined hit set truncation
306 <xref linkend="querymodel-zebra-local-attr-limit"/> and
307 <xref linkend="zebra-cfg"/>
311 <entry>Paged result sets</entry>
313 <entry>Paging of search requests and present/display request
314 can return any successive number of records from any start
315 position in the hit set, i.e. it is trivial to provide search
316 results in successive pages of any size.</entry>
320 <entry>&xml; document transformations</entry>
321 <entry>&xslt; based</entry>
322 <entry> Record presentation can be performed in many
323 pre-defined &xml; data
324 formats, where the original &xml; records are on-the-fly transformed
325 through any preconfigured &xslt; transformation. It is therefore
326 trivial to present records in short/full &xml; views, transforming to
327 RSS, Dublin Core, or other &xml; based data formats, or transform
328 records to XHTML snippets ready for inserting in XHTML pages.</entry>
330 <xref linkend="record-model-alvisxslt-elementset"/></entry>
333 <entry>Binary record transformations</entry>
334 <entry>&marc;, &usmarc;, &marc21; and &marcxml;</entry>
335 <entry>post-filter record transformations</entry>
339 <entry>Record Syntaxes</entry>
341 <entry> Multiple record syntaxes
342 for data retrieval: &grs1;, &sutrs;,
343 &xml;, ISO2709 (&marc;), etc. Records can be mapped between
344 record syntaxes and schemas on the fly.</entry>
348 <entry>&zebra; internal metadata</entry>
350 <entry> &zebra; internal document metadata can be fetched in
351 &sutrs; and &xml; record syntaxes. Those are useful in client
352 applications.</entry>
353 <entry><xref linkend="special-retrieval"/></entry>
356 <entry>&zebra; internal raw record data</entry>
358 <entry> &zebra; internal raw, binary record data can be fetched in
359 &sutrs; and &xml; record syntaxes, leveraging %zebra; to a
360 binary storage system</entry>
361 <entry><xref linkend="special-retrieval"/></entry>
364 <entry>&zebra; internal record field data</entry>
366 <entry> &zebra; internal record field data can be fetched in
367 &sutrs; and &xml; record syntaxes. This makes very fast minimal
368 record data displays possible.</entry>
369 <entry><xref linkend="special-retrieval"/></entry>
376 <section id="features-sort-rank">
377 <title>&zebra; Sorting and Ranking</title>
379 <table id="table-features-sort-rank" frame="top">
380 <title>&zebra; sorting and ranking</title>
384 <entry>Feature</entry>
385 <entry>Availability</entry>
387 <entry>Reference</entry>
393 <entry>numeric, lexicographic</entry>
394 <entry>Sorting on the basis of alpha-numeric and numeric data
395 is supported. Alphanumeric sorts can be configured for
396 different data encodings and locales for European languages.</entry>
397 <entry><xref linkend="administration-ranking-sorting"/> and
398 <xref linkend="querymodel-zebra-attr-sorting"/></entry>
401 <entry>Combined sorting</entry>
403 <entry>Sorting on the basis of combined sorts  e.g. combinations of
404 ascending/descending sorts of lexicographical/numeric/date field data
406 <entry><xref linkend="administration-ranking-sorting"/></entry>
409 <entry>Relevance ranking</entry>
410 <entry>TF-IDF like</entry>
411 <entry>Relevance-ranking of free-text queries is supported
412 using a TF-IDF like algorithm.</entry>
413 <entry><xref linkend="administration-ranking-dynamic"/></entry>
416 <entry>Static pre-ranking</entry>
418 <entry>Enables pre-index time ranking of documents where hit
419 lists are ordered first by ascending static rank, then by
420 ascending document ID.</entry>
421 <entry><xref linkend="administration-ranking-static"/></entry>
429 <section id="features-updates">
430 <title>&zebra; Live Updates</title>
433 <table id="table-features-updates" frame="top">
434 <title>&zebra; live updates</title>
438 <entry>Feature</entry>
439 <entry>Availability</entry>
441 <entry>Reference</entry>
446 <entry>Incremental and batch updates</entry>
448 <entry>It is possible to schedule record inserts/updates/deletes in any
449 quantity, from single individual handled records to batch updates
450 in strikes of any size, as well as total re-indexing of all records
451 from file system. </entry>
452 <entry><xref linkend="zebraidx"/></entry>
455 <entry>Remote updates</entry>
456 <entry>&z3950; extended services</entry>
457 <entry>Updates can be performed from remote locations using the
458 &z3950; extended services. Access to extended services can be
459 login-password protected.</entry>
460 <entry><xref linkend="administration-extended-services"/> and
461 <xref linkend="zebra-cfg"/></entry>
464 <entry>Live updates</entry>
465 <entry>transaction based</entry>
466 <entry> Data updates are transaction based and can be performed
467 on running &zebra; systems. Full searchability is preserved
468 during life data update due to use of shadow disk areas for
469 update operations. Multiple update transactions at the same
470 time are lined up, to be performed one after each other. Data
471 integrity is preserved.</entry>
472 <entry><xref linkend="shadow-registers"/></entry>
479 <section id="features-protocol">
480 <title>&zebra; Networked Protocols</title>
482 <table id="table-features-protocol" frame="top">
483 <title>&zebra; networked protocols</title>
487 <entry>Feature</entry>
488 <entry>Availability</entry>
490 <entry>Reference</entry>
495 <entry>Fundamental operations</entry>
496 <entry>&z3950;/&sru; <literal>explain</literal>,
497 <literal>search</literal>, <literal>scan</literal>, and
498 <literal>update</literal></entry>
500 <entry><xref linkend="querymodel-operation-types"/></entry>
503 <entry>&z3950; protocol support</entry>
505 <entry> Protocol facilities supported are:
506 <literal>init</literal>, <literal>search</literal>,
507 <literal>present</literal> (retrieval),
508 Segmentation (support for very large records),
509 <literal>delete</literal>, <literal>scan</literal>
510 (index browsing), <literal>sort</literal>,
511 <literal>close</literal> and support for the <literal>update</literal>
512 Extended Service to add or replace an existing &xml;
513 record. Piggy-backed presents are honored in the search
514 request. Named result sets are supported.</entry>
515 <entry><xref linkend="protocol-support"/></entry>
518 <entry>Web Service support</entry>
519 <entry>&sru_gps;</entry>
520 <entry> The protocol operations <literal>explain</literal>,
521 <literal>searchRetrieve</literal> and <literal>scan</literal>
522 are supported. <ulink url="&url.cql;">&cql;</ulink> to internal
524 conversion is supported. Extended RPN queries
525 for search/retrieve and scan are supported.</entry>
526 <entry><xref linkend="zebrasrv-sru-support"/></entry>
533 <section id="features-scalability">
534 <title>&zebra; Data Size and Scalability</title>
536 <table id="table-features-scalability" frame="top">
537 <title>&zebra; data size and scalability</title>
541 <entry>Feature</entry>
542 <entry>Availability</entry>
544 <entry>Reference</entry>
549 <entry>No of records</entry>
550 <entry>40-60 million</entry>
555 <entry>Data size</entry>
556 <entry>100 GB of record data</entry>
557 <entry>&zebra; based applications have sucessfully indexed up
558 to 100 GB of record data</entry>
562 <entry>Scale out</entry>
563 <entry>multiple discs</entry>
568 <entry>Performance</entry>
569 <entry><literal>O(n * log N)</literal></entry>
570 <entry> &zebra; query speed and performance is affected roughly by
571 <literal>O(log N)</literal>,
572 where <literal>N</literal> is the total database size, and by
573 <literal>O(n)</literal>, where <literal>n</literal> is the
574 specific query hit set size.</entry>
578 <entry>Average search times</entry>
580 <entry> Even on very large size databases hit rates of 20 queries per
581 seconds with average query answering time of 1 second are possible,
582 provided that the boolean queries are constructed sufficiently
583 precise to result in hit sets of the order of 1000 to 5.000
588 <entry>Large databases</entry>
589 <entry>64 bit file pointers</entry>
590 <entry>64 file pointers assure that register files can extend
591 the 2 GB limit. Logical files can be
592 automatically partitioned over multiple disks, thus allowing for
593 large databases.</entry>
601 <section id="features-platforms">
602 <title>&zebra; Supported Platforms</title>
604 <table id="table-features-platforms" frame="top">
605 <title>&zebra; supported platforms</title>
609 <entry>Feature</entry>
610 <entry>Availability</entry>
612 <entry>Reference</entry>
619 <entry>GNU Linux (32 and 64bit), journaling Reiser or (better)
621 on disks. NFS filesystems are not supported.
622 GNU/Debian Linux packages are available</entry>
623 <entry><xref linkend="installation-debian"/></entry>
627 <entry>tarball</entry>
628 <entry>&zebra; is written in portable C, so it runs on most
630 Usual tarball install possible on many major Unix systems</entry>
631 <entry><xref linkend="installation-unix"/></entry>
634 <entry>Windows</entry>
635 <entry>NT/2000/2003/XP</entry>
636 <entry>&zebra; runs as well on Windows (NT/2000/2003/XP).
637 Windows installer packages available</entry>
638 <entry><xref linkend="installation-win32"/></entry>
648 <section id="introduction-apps">
649 <title>References and &zebra; based Applications</title>
651 &zebra; has been deployed in numerous applications, in both the
652 academic and commercial worlds, in application domains as diverse
653 as bibliographic catalogues, geospatial information, structured
654 vocabulary browsing, government information locators, civic
655 information systems, environmental observations, museum information
659 Notable applications include the following:
663 <section id="koha-ils">
664 <title>Koha free open-source ILS</title>
666 <ulink url="http://www.koha.org/">Koha</ulink> is a full-featured
667 open-source ILS, initially developed in
668 New Zealand by Katipo Communications Ltd, and first deployed in
669 January of 2000 for Horowhenua Library Trust. It is currently
670 maintained by a team of software providers and library technology
671 staff from around the globe.
674 <ulink url="http://liblime.com/">LibLime</ulink>,
675 a company that is marketing and supporting Koha, adds in
676 the new release of Koha 3.0 the &zebra;
677 database server to drive its bibliographic database.
680 In early 2005, the Koha project development team began looking at
681 ways to improve &marc; support and overcome scalability limitations
682 in the Koha 2.x series. After extensive evaluations of the best
683 of the Open Source textual database engines - including MySQL
684 full-text searching, PostgreSQL, Lucene and Plucene - the team
688 "&zebra; completely eliminates scalability limitations, because it
689 can support tens of millions of records." explained Joshua
690 Ferraro, LibLime's Technology President and Koha's Project
691 Release Manager. "Our performance tests showed search results in
692 under a second for databases with over 5 million records on a
693 modest i386 900Mhz test server."
696 "&zebra; also includes support for true boolean search expressions
697 and relevance-ranked free-text queries, both of which the Koha
698 2.x series lack. &zebra; also supports incremental and safe
699 database updates, which allow on-the-fly record
700 management. Finally, since &zebra; has at its heart the &z3950;
701 protocol, it greatly improves Koha's support for that critical
705 Although the bibliographic database will be moved to &zebra;, Koha
706 3.0 will continue to use a relational SQL-based database design
707 for the 'factual' database. "Relational database managers have
708 their strengths, in spite of their inability to handle large
709 numbers of bibliographic records efficiently," summed up Ferraro,
710 "We're taking the best from both worlds in our redesigned Koha
714 See also LibLime's newsletter article
715 <ulink url="http://www.liblime.com/newsletter/2006/01/features/koha-earns-its-stripes/">
716 Koha Earns its Stripes</ulink>.
720 <section id="emilda-ils">
721 <title>Emilda open source ILS</title>
723 <ulink url="http://www.emilda.org/">Emilda</ulink>
724 is a complete Integrated Library System, released under the
725 GNU General Public License. It has a
726 full featured Web-OPAC, allowing comprehensive system management
727 from virtually any computer with an Internet connection, has
728 template based layout allowing anyone to alter the visual
729 appearance of Emilda, and is
730 &xml; based language for fast and easy portability to virtually any
732 Currently, Emilda is used at three schools in Espoo, Finland.
735 As a surplus, 100% &marc; compatibility has been achieved using the
736 &zebra; Server from Index Data as backend server.
740 <section id="reindex-ils">
741 <title>ReIndex.Net web based ILS</title>
743 <ulink url="http://www.reindex.net/index.php?lang=en">Reindex.net</ulink>
744 is a netbased library service offering all
745 traditional functions on a very high level plus many new
746 services. Reindex.net is a comprehensive and powerful WEB system
747 based on standards such as &xml; and &z3950;.
748 updates. Reindex supports &marc21;, dan&marc; eller Dublin Core with
752 Reindex.net runs on GNU/Debian Linux with &zebra; and Simpleserver
754 Data for bibliographic data. The relational database system
755 Sybase 9 &xml; is used for
757 Internally &marcxml; is used for bibliographical records. Update
758 utilizes &z3950; extended services.
762 <section id="dads-article-database">
763 <title>DADS - the DTV Article Database
766 DADS is a huge database of more than ten million records, totalling
767 over ten gigabytes of data. The records are metadata about academic
768 journal articles, primarily scientific; about 10% of these
769 metadata records link to the full text of the articles they
770 describe, a body of about a terabyte of information (although the
771 full text is not indexed.)
774 It allows students and researchers at DTU (Danmarks Tekniske
775 Universitet, the Technical College of Denmark) to find and order
776 articles from multiple databases in a single query. The database
777 contains literature on all engineering subjects. It's available
778 on-line through a web gateway, though currently only to registered
782 More information can be found at
783 <ulink url="http://www.dtv.dk/"/> and
784 <ulink url="http://dads.dtv.dk"/>
788 <section id="infonet-eprints">
789 <title>Infonet Eprints</title>
791 The InfoNet Eprints service from the
792 <ulink url="http://www.dtv.dk/">
793 Technical Knowledge Center of Denmark</ulink>
794 provides access to documents stored in
795 eprint/preprint servers and institutional research archives around
796 the world. The service is based on Open Archives Initiative metadata
797 harvesting of selected scientific archives around the world. These
798 open archives offer free and unrestricted access to their contents.
801 Infonet Eprints currently holds 1.4 million records from 16 archives.
802 The online search facility is found at
803 <ulink url="http://preprints.cvt.dk"/>.
807 <section id="alvis-project">
810 The <ulink url="http://www.alvis.info/alvis/">Alvis</ulink> EU
811 project run under the 6th Framework (IST-1-002068-STP)
812 is building a semantic-based peer-to-peer search engine. A
813 consortium of eleven partners from six different European
814 Community countries plus Switzerland and China contribute
815 with expertise in a broad range of specialties including network
816 topologies, routing algorithms, linguistic analysis and
820 The &zebra; information retrieval indexing machine is used inside
821 the Alvis framework to
822 manage huge collections of natural language processed and
823 enhanced &xml; data, coming from a topic relevant web crawl.
824 In this application, &zebra; swallows and manages 37GB of &xml; data
825 in about 4 hours, resulting in search times of fractions of
832 <title>ULS (Union List of Serials)</title>
835 has created a union catalogue for the periodicals of the
836 twenty-one constituent libraries of the University of London and
837 the University of Westminster
838 (<ulink url="http://www.m25lib.ac.uk/ULS/"/>).
839 They have achieved this using an
840 unusual architecture, which they describe as a
841 ``non-distributed virtual union catalogue''.
844 The member libraries send in data files representing their
845 periodicals, including both brief bibliographic data and summary
846 holdings. Then 21 individual &z3950; targets are created, each
847 using &zebra;, and all mounted on the single hardware server.
848 The live service provides a web gateway allowing &z3950; searching
849 of all of the targets or a selection of them. &zebra;'s small
850 footprint allows a relatively modest system to comfortably host
854 More information can be found at
855 <ulink url="http://www.m25lib.ac.uk/ULS/"/>
860 <title>NLI-&z3950; - a Natural Language Interface for Libraries</title>
862 Fernuniversität Hagen in Germany have developed a natural
863 language interface for access to library databases.
865 url="http://ki212.fernuni-hagen.de/nli/NLIintro.html"/> -->
866 In order to evaluate this interface for recall and precision, they
867 chose &zebra; as the basis for retrieval effectiveness. The &zebra;
868 server contains a copy of the GIRT database, consisting of more
869 than 76000 records in &sgml; format (bibliographic records from
870 social science), which are mapped to &marc; for presentation.
873 (GIRT is the German Indexing and Retrieval Testdatabase. It is a
874 standard German-language test database for intelligent indexing
875 and retrieval systems. See
876 <ulink url="http://www.gesis.org/forschung/informationstechnologie/clef-delos.htm"/>)
879 Evaluation will take place as part of the TREC/CLEF campaign 2003
880 <ulink url="http://clef.iei.pi.cnr.it"/>.
881 <!-- or <ulink url="http://www4.eurospider.ch/CLEF/"/> -->
884 For more information, contact Johannes Leveling
885 <email>Johannes.Leveling@FernUni-Hagen.De</email>
889 <section id="various-web-indexes">
890 <title>Various web indexes</title>
892 &zebra; has been used by a variety of institutions to construct
893 indexes of large web sites, typically in the region of tens of
894 millions of pages. In this role, it functions somewhat similarly
895 to the engine of google or altavista, but for a selected intranet
896 or a subset of the whole Web.
899 For example, Liverpool University's web-search facility (see on
901 <ulink url="http://www.liv.ac.uk/"/>
902 and many sub-pages) works by relevance-searching a &zebra; database
903 which is populated by the Harvest-NG web-crawling software.
906 For more information on Liverpool university's intranet search
907 architecture, contact John Gilbertson
908 <email>jgilbert@liverpool.ac.uk</email>
912 has recently modified the Harvest web indexer to use &zebra; as
913 its native repository engine. His comments on the switch over
914 from the old engine are revealing:
917 The first results after some testing with &zebra; are very
918 promising. The tests were done with around 220,000 SOIF files,
919 which occupies 1.6GB of disk space.
922 Building the index from scratch takes around one hour with &zebra;
923 where [old-engine] needs around five hours. While [old-engine]
924 blocks search requests when updating its index, &zebra; can still
925 answer search requests.
927 &zebra; supports incremental indexing which will speed up indexing
931 While the search time of [old-engine] varies from some seconds
932 to some minutes depending how expensive the query is, &zebra;
933 usually takes around one to three seconds, even for expensive
936 &zebra; can search more than 100 times faster than [old-engine]
937 and can process multiple search requests simultaneously
940 I am very happy to see such nice software available under GPL.
948 <section id="introduction-support">
949 <title>Support</title>
951 You can get support for &zebra; from at least three sources.
954 First, there's the &zebra; web site at
955 <ulink url="&url.idzebra;"/>,
956 which always has the most recent version available for download.
957 If you have a problem with &zebra;, the first thing to do is see
958 whether it's fixed in the current release.
961 Second, there's the &zebra; mailing list. Its home page at
962 <ulink url="&url.idzebra.mailinglist;"/>
963 includes a complete archive of all messages that have ever been
964 posted on the list. The &zebra; mailing list is used both for
965 announcements from the authors (new
966 releases, bug fixes, etc.) and general discussion. You are welcome
967 to seek support there. Join by filling the form on the list home page.
970 Third, it's possible to buy a commercial support contract, with
971 well defined service levels and response times, from Index Data.
973 <ulink url="&url.indexdata.support;"/>
979 <section id="future">
980 <title>Future Directions</title>
983 These are some of the plans that we have for the software in the near
984 and far future, ordered approximately as we expect to work on them.
992 Improved support for &xml; in search and retrieval. Eventually,
993 the goal is for &zebra; to pull double duty as a flexible
994 information retrieval engine and high-performance &xml;
995 repository. The recent addition of XPath searching is one
996 example of the kind of enhancement we're working on.
999 There is also the experimental <literal>ALVIS &xslt;</literal>
1000 &xml; input filter, which unleashes the full power of &dom; based
1001 &xslt; transformations during indexing and record retrieval. Work
1002 on this filter has been sponsored by the ALVIS EU project
1003 <ulink url="http://www.alvis.info/alvis/"/>. We expect this filter to
1004 mature soon, as it is planned to be included in the version 2.0
1011 Finalisation and documentation of &zebra;'s C programming
1012 &api;, allowing updates, database management and other functions
1013 not readily expressed in &z3950;. We will also consider
1014 exposing the &api; through &soap;.
1020 Improved free-text searching. We're first and foremost octet jockeys and
1021 we're actively looking for organisations or people who'd like
1022 to contribute experience in relevance ranking and text
1031 Programmers thrive on user feedback. If you are interested in a
1032 facility that you don't see mentioned here, or if there's something
1033 you think we could do better, please drop us a mail. Better still,
1034 implement it and send us the patches.
1037 If you think it's all really neat, you're welcome to drop us a line
1038 saying that, too. You can email us on
1039 <email>info@indexdata.dk</email>
1040 or check the contact info at the end of this manual.
1045 <!-- Keep this comment at the end of the file
1050 sgml-minimize-attributes:nil
1051 sgml-always-quote-attributes:t
1054 sgml-parent-document: "zebra.xml"
1055 sgml-local-catalogs: nil
1056 sgml-namecase-general:t