1 <?xml version="1.0" standalone="no"?>
2 <!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.1//EN"
3 "http://www.oasis-open.org/docbook/xml/4.1/docbookx.dtd"
5 <!ENTITY local SYSTEM "local.ent">
6 <!ENTITY manref SYSTEM "manref.xml">
7 <!ENTITY progref SYSTEM "progref.xml">
8 <!ENTITY % common SYSTEM "common/common.ent">
10 <!-- Next line allows imagedata/@format="PDF" and is taken from
11 http://lists.oasis-open.org/archives/docbook/200303/msg00163.html
13 <!ENTITY % local.notation.class "| PDF">
14 <!-- Next line is necessary for some XML parsers, for reasons I
15 don't understand. I got this from
16 http://lists.oasis-open.org/archives/docbook/200303/msg00180.html
18 <!NOTATION PDF SYSTEM "PDF">
20 <!-- $Id: book.xml,v 1.36 2006-09-07 09:42:53 mike Exp $ -->
23 <title>Metaproxy - User's Guide and Reference</title>
25 <firstname>Mike</firstname><surname>Taylor</surname>
28 <firstname>Adam</firstname><surname>Dickmeiss</surname>
32 <holder>Index Data ApS</holder>
36 Metaproxy is a universal router, proxy and encapsulated
37 metasearcher for information retrieval protocols. It accepts,
38 processes, interprets and redirects requests from IR clients using
39 standard protocols such as
40 <ulink url="&url.z39.50;">ANSI/NISO Z39.50</ulink>
41 (and in the future <ulink url="&url.sru;">SRU</ulink>
42 and <ulink url="&url.srw;">SRW</ulink>), as
43 well as functioning as a limited
44 <ulink url="&url.http;">HTTP</ulink> server.
45 Metaproxy is configured by an XML file which
46 specifies how the software should function in terms of routes that
47 the request packets can take through the proxy, each step on a
48 route being an instantiation of a filter. Filters come in many
49 types, one for each operation: accepting Z39.50 packets, logging,
50 query transformation, multiplexing, etc. Further filter-types can
51 be added as loadable modules to extend Metaproxy functionality,
55 Metaproxy is <emphasis>not</emphasis> open-source software, but
56 may be freely downloaded, unpacked, inspected, built and run for
57 evaluation purposes. Deployment requires a separate, commercial,
63 <imagedata fileref="common/id.png" format="PNG"/>
66 <imagedata fileref="common/id.eps" format="EPS"/>
73 <chapter id="introduction">
74 <title>Introduction</title>
78 <ulink url="&url.metaproxy;">Metaproxy</ulink>
79 is a standalone program that acts as a universal router, proxy and
80 encapsulated metasearcher for information retrieval protocols such
81 as <ulink url="&url.z39.50;">Z39.50</ulink>, and in the future
82 <ulink url="&url.sru;">SRU</ulink> and <ulink url="&url.srw;">SRW</ulink>.
83 To clients, it acts as a server of these protocols: it can be searched,
84 records can be retrieved from it, etc.
85 To servers, it acts as a client: it searches in them,
86 retrieves records from them, etc. it satisfies its clients'
87 requests by transforming them, multiplexing them, forwarding them
88 on to zero or more servers, merging the results, transforming
89 them, and delivering them back to the client. In addition, it
90 acts as a simple <ulink url="&url.http;">HTTP</ulink> server; support
91 for further protocols can be added in a modular fashion, through the
92 creation of new filters.
97 Fish, bananas, cold pyjamas,
98 Mutton, beef and trout!
99 - attributed to Cole Porter.
102 Metaproxy is a more capable alternative to
103 <ulink url="&url.yazproxy;">YAZ Proxy</ulink>,
104 being more powerful, flexible, configurable and extensible. Among
105 its many advantages over the older, more pedestrian work are
106 support for multiplexing (encapsulated metasearching), routing by
107 database name, authentication and authorisation and serving local
108 files via HTTP. Equally significant, its modular architecture
109 facilitites the creation of pluggable modules implementing further
113 This manual will describe how to install Metaproxy
114 before giving an overview of its architecture, then discussing the
115 key concept of a filter in some depth and giving an overview of
116 the various filter types, then discussing the configuration file
117 format. After this come several optional chapters which may be
118 freely skipped: a detailed discussion of virtual databases and
119 multi-database searching, some notes on writing extensions
120 (additional filter types) and a high-level description of the
121 source code. Finally comes the reference guide, which contains
122 instructions for invoking the <command>metaproxy</command>
123 program, and detailed information on each type of filter,
128 <chapter id="license">
129 <title>The Metaproxy License</title>
130 <orderedlist numeration="arabic">
133 You are allowed to download this software for evaluation purposes.
134 You can unpack it, build it, run it, see how it works and how it fits
135 your needs, all at zero cost.
140 You may NOT deploy the software. For the purposes of this license,
141 deployment means running it for any purpose other than evaluation,
142 whether or not you or anyone else makes a profit from doing so. If
143 you wish to deploy the software, you must first contact Index Data and
144 arrange to purchase a DEPLOYMENT LICENCE. If you are unsure
145 whether or not your proposed use of the software constitutes
146 deployment, email us at <literal>info@indexdata.com</literal>
152 You may modify your copy of the software (fix bugs, add features)
153 if you need to. We encourage you to send your changes back to us for
154 integration into the master copy, but you are not obliged to do so. You
155 may NOT pass your changes on to any other party.
160 There is NO WARRANTY for this software, to the extent permitted by
161 applicable law. We provide the software ``as is'' without warranty of
162 any kind, either expressed or implied, including, but not limited to, the
163 implied warranties of MERCHANTABILITY and FITNESS FOR A
164 PARTICULAR PURPOSE. The entire risk as to the quality and
165 performance of the software is with you. Should the software prove
166 defective, you assume the cost of all necessary servicing, repair or
167 correction. In no event unless required by applicable law will we be
168 liable to you for damages, arising out of the use of the software,
169 including but not limited to loss of data or data being rendered
175 All rights to the software are reserved by Index Data except where
176 this license explicitly says otherwise.
182 <chapter id="installation">
183 <title>Installation</title>
185 Metaproxy depends on the following tools/libraries:
187 <varlistentry><term><ulink url="&url.yazplusplus;">YAZ++</ulink></term>
190 This is a C++ library based on <ulink url="&url.yaz;">YAZ</ulink>.
194 <varlistentry><term><ulink url="&url.libxslt;">Libxslt</ulink></term>
196 <para>This is an XSLT processor - based on
197 <ulink url="&url.libxml2;">Libxml2</ulink>. Both Libxml2 and
198 Libxslt must be installed with the development components
199 (header files, etc.) as well as the run-time libraries.
203 <varlistentry><term><ulink url="&url.boost;">Boost</ulink></term>
206 The popular C++ library. Initial versions of Metaproxy
207 was built with 1.33.0. Version 1.33.1 works too.
214 In order to compile Metaproxy a modern C++ compiler is
215 required. Boost, in particular, requires the C++ compiler
216 to facilitate the newest features. Refer to Boost
217 <ulink url="&url.boost.compilers.status;">Compiler Status</ulink>
218 for more information.
221 We have succesfully built Metaproxy using the compilers
222 <ulink url="&url.gcc;">GCC</ulink> version 4.0 and
223 <ulink url="&url.vstudio;">Microsoft Visual Studio</ulink> 2003/2005.
226 <section id="installation.unix">
227 <title>Installation on Unix (from Source)</title>
229 Here is a quick step-by-step guide on how to compile all the
230 tools that Metaproxy uses. Only few systems have none of the required
231 tools binary packages. If, for example, Libxml2/libxslt are already
232 installed as development packages use those (and omit compilation).
239 gunzip -c libxml2-version.tar.gz|tar xf -
247 gunzip -c libxslt-version.tar.gz|tar xf -
258 gunzip -c yaz-version.tar.gz|tar xf -
266 gunzip -c yazpp-version.tar.gz|tar xf -
277 gunzip -c boost-version.tar.gz|tar xf -
288 gunzip -c metaproxy-version.tar.gz|tar xf -
297 <section id="installation.debian">
298 <title>Installation on Debian GNU/Linux</title>
300 All dependencies for Metaproxy are available as
301 <ulink url="&url.debian;">Debian</ulink>
302 packages for the sarge (stable in 2005) and etch (testing in 2005)
306 The procedures for Debian based systems, such as
307 <ulink url="&url.ubuntu;">Ubuntu</ulink> is probably similar
310 There is currently no official Debian package for YAZ++.
311 And the Debian package for YAZ is probably too old.
312 Update the <filename>/etc/apt/sources.list</filename>
313 to include the Index Data repository.
314 See YAZ' <ulink url="&url.yaz.download.debian;">Download Debian</ulink>
315 for more information.
318 apt-get install libxslt1-dev
319 apt-get install libyazpp-dev
320 apt-get install libboost-dev
321 apt-get install libboost-thread-dev
322 apt-get install libboost-date-time-dev
323 apt-get install libboost-program-options-dev
324 apt-get install libboost-test-dev
327 With these packages installed, the usual configure + make
328 procedure can be used for Metaproxy as outlined in
329 <xref linkend="installation.unix"/>.
333 <section id="installation.windows">
334 <title>Installation on Windows</title>
336 Metaproxy can be compiled with Microsoft
337 <ulink url="&url.vstudio;">Visual Studio</ulink>.
338 Version 2003 (C 7.1) and 2005 (C 8.0) is known to work.
340 <section id="installation.windows.boost">
343 Get Boost from its <ulink url="&url.boost;">home page</ulink>.
344 You also need Boost Jam (an alternative to make).
345 That's also available from the Boost home page.
346 The files to be downloaded are called something like:
347 <filename>boost_1_33-1.exe</filename>
349 <filename>boost-jam-3.1.12-1-ntx86.zip</filename>.
350 Unpack Boost Jam first. Put <filename>bjam.exe</filename>
351 in your system path. Make a command prompt and ensure
352 it can be found automatically. If not check the PATH.
353 The Boost .exe is a self-extracting exe with
354 complete source for Boost. Compile that source with
355 Boost Jam (An alternative to Make).
356 The compilation takes a while.
357 For Visual Studio 2003, use
359 bjam "-sTOOLS=vc-7_1"
361 Here <literal>vc-7_1</literal> refers to a "Toolset" (compiler system).
362 For Visual Studio 2005, use
364 bjam "-sTOOLS=vc-8_0"
366 To install the libraries in a common place, use
368 bjam "-sTOOLS=vc-7_1" install
370 (or vc-8_0 for VS 2005).
373 By default, the Boost build process installs the resulting
374 libraries + header files in
375 <literal>\boost\lib</literal>, <literal>\boost\include</literal>.
378 For more informatation about installing Boost refer to the
379 <ulink url="&url.boost.getting.started;">getting started</ulink>
384 <section id="installation.windows.libxslt">
385 <title>Libxslt</title>
387 <ulink url="&url.libxslt;">Libxslt</ulink> can be downloaded
389 <ulink url="&url.libxml2.download.win32;">here</ulink>.
392 Libxslt has other dependencies, but thes can all be downloaded
393 from the same site. Get the following:
394 iconv, zlib, libxml2, libxslt.
398 <section id="installation.windows.yaz">
401 <ulink url="&url.yaz;">YAZ</ulink> can be downloaded
403 <ulink url="&url.yaz.download.win32;">here</ulink>.
407 <section id="installation.windows.yazplusplus">
410 Get <ulink url="&url.yazplusplus;">YAZ++</ulink> as well.
411 Version 1.0 or later is required. For now get it from
413 <ulink url="&url.snapshot.download;">Snapshot area</ulink>.
416 YAZ++ includes NMAKE makefiles, similar to those found in the
421 <section id="installation.windows.metaproxy">
422 <title>Metaproxy</title>
424 Metaproxy is shipped with NMAKE makfiles as well - similar
425 to those found in the YAZ++/YAZ packages. Adjust this Makefile
426 to point to the proper locations of Boost, Libxslt, Libxml2,
427 zlib, iconv, yaz and yazpp.
431 <varlistentry><term><literal>DEBUG</literal></term>
433 If set to 1, the software is
434 compiled with debugging libraries (code generation is
435 multi-threaded debug DLL).
436 If set to 0, the software is compiled with release libraries
437 (code generation is multi-threaded DLL).
442 <term><literal>BOOST</literal></term>
445 Boost install location
451 <term><literal>BOOST_VERSION</literal></term>
454 Boost version (replace . with _).
460 <term><literal>BOOST_TOOLSET</literal></term>
469 <term><literal>LIBXSLT_DIR</literal>,
470 <literal>LIBXML2_DIR</literal> ..</term>
473 Specify the locations of Libxslt, libiconv, libxml2 and
482 After succesful compilation you'll find
483 <literal>metaproxy.exe</literal> in the
484 <literal>bin</literal> directory.
492 <chapter id="architecture">
493 <title>The Metaproxy Architecture</title>
495 The Metaproxy architecture is based on three concepts:
496 the <emphasis>package</emphasis>,
497 the <emphasis>route</emphasis>
498 and the <emphasis>filter</emphasis>.
502 <term>Packages</term>
505 A package is request or response, encoded in some protocol,
506 issued by a client, making its way through Metaproxy, send to or
507 received from a server, or sent back to the client.
510 The core of a package is the protocol unit - for example, a
511 Z39.50 Init Request or Search Response, or an SRU searchRetrieve
512 URL or Explain Response. In addition to this core, a package
513 also carries some extra information added and used by Metaproxy
517 In general, packages are doctored as they pass through
518 Metaproxy. For example, when the proxy performs authentication
519 and authorisation on a Z39.50 Init request, it removes the
520 authentication credentials from the package so that they are not
521 passed onto the back-end server; and when search-response
522 packages are obtained from multiple servers, they are merged
523 into a single unified package that makes its way back to the
532 Packages make their way through routes, which can be thought of
533 as programs that operate on the package data-type. Each
534 incoming package initially makes its way through a default
535 route, but may be switched to a different route based on various
536 considerations. Routes are made up of sequences of filters (see
545 Filters provide the individual instructions within a route, and
546 effect the necessary transformations on packages. A particular
547 configuration of Metaproxy is essentially a set of filters,
548 described by configuration details and arranged in order in one
549 or more routes. There are many kinds of filter - about a dozen
550 at the time of writing with more appearing all the time - each
551 performing a specific function and configured by different
555 The word ``filter'' is sometimes used rather loosely, in two
556 different ways: it may be used to mean a particular
557 <emphasis>type</emphasis> of filter, as when we speak of ``the
558 auth_simple filter'' or ``the multi filter''; or it may be used
559 to be a specific <emphasis>instance</emphasis> of a filter
560 within a Metaproxy configuration. For example, a single
561 configuration will often contain multiple instances of the
562 <literal>z3950_client</literal> filter. In
563 operational terms, of these is a separate filter. In practice,
564 context always make it clear which sense of the word ``filter''
568 Extensibility of Metaproxy is primarily through the creation of
569 plugins that provide new filters. The filter API is small and
570 conceptually simple, but there are many details to master. See
572 <link linkend="extensions">extensions</link>.
578 Since packages are created and handled by the system itself, and
579 routes are conceptually simple, most of the remainder of this
580 document concentrates on filters. After a brief overview of the
581 filter types follows, along with some thoughts on possible future
588 <chapter id="filters">
589 <title>Filters</title>
593 <title>Introductory notes</title>
595 It's useful to think of Metaproxy as an interpreter providing a small
596 number of primitives and operations, but operating on a very
597 complex data type, namely the ``package''.
600 A package represents a Z39.50 or SRU/W request (whether for Init,
601 Search, Scan, etc.) together with information about where it came
602 from. Packages are created by front-end filters such as
603 <literal>frontend_net</literal> (see below), which reads them from
604 the network; other front-end filters are possible. They then pass
605 along a route consisting of a sequence of filters, each of which
606 transforms the package and may also have side-effects such as
607 generating logging. Eventually, the route will yield a response,
608 which is sent back to the origin.
611 There are many kinds of filter: some that are defined statically
612 as part of Metaproxy, and others may be provided by third parties
613 and dynamically loaded. They all conform to the same simple API
614 of essentially two methods: <function>configure()</function> is
615 called at startup time, and is passed a DOM tree representing that
616 part of the configuration file that pertains to this filter
617 instance: it is expected to walk that tree extracting relevant
618 information; and <function>process()</function> is called every
619 time the filter has to processes a package.
622 While all filters provide the same API, there are different modes
623 of functionality. Some filters are sources: they create
625 (<literal>frontend_net</literal>);
626 others are sinks: they consume packages and return a result
627 (<literal>z3950_client</literal>,
628 <literal>backend_test</literal>,
629 <literal>http_file</literal>);
630 the others are true filters, that read, process and pass on the
631 packages they are fed
632 (<literal>auth_simple</literal>,
633 <literal>log</literal>,
634 <literal>multi</literal>,
635 <literal>query_rewrite</literal>,
636 <literal>session_shared</literal>,
637 <literal>template</literal>,
638 <literal>virt_db</literal>).
643 <section id="overview.filter.types">
644 <title>Overview of filter types</title>
646 We now briefly consider each of the types of filter supported by
647 the core Metaproxy binary. This overview is intended to give a
648 flavour of the available functionality; more detailed information
649 about each type of filter is included below in
650 <link linkend="filterref"
651 >the reference guide to Metaproxy filters</link>.
654 The filters are here named by the string that is used as the
655 <literal>type</literal> attribute of a
656 <literal><filter></literal> element in the configuration
657 file to request them, with the name of the class that implements
658 them in parentheses. (The classname is not needed for normal
659 configuration and use of Metaproxy; it is useful only to
663 The filters are here listed in alphabetical order:
670 New virt_db-alike that does inteligent peer choice, explain merging,
671 adds FD&N to explain. Keeps init responses (like "virt_db Classic"),
672 makes routing choices based on local explain knowledge. Ref IDDI
675 Filter to convert Explain Classic to ZeeRex.
677 CQL2PQF (which needs augmented ZeeRex) - MARC for Talis.
681 Figure out what additional information we need in:
682 ZeeRex (check against D3.1)
683 Init request (e.g. loop detection)
684 Query package (e.g. number of hops)
685 Query response (e.g. record source)
690 <title><literal>auth_simple</literal>
691 (mp::filter::AuthSimple)</title>
693 Simple authentication and authorisation. The configuration
694 specifies the name of a file that is the user register, which
695 lists <varname>username</varname>:<varname>password</varname>
696 pairs, one per line, colon separated. When a session begins, it
697 is rejected unless username and passsword are supplied, and match
698 a pair in the register. The configuration file may also specific
699 the name of another file that is the target register: this lists
700 lists <varname>username</varname>:<varname>dbname</varname>,<varname>dbname</varname>...
701 sets, one per line, with multiple database names separated by
702 commas. When a search is processed, it is rejected unless the
703 database to be searched is one of those listed as available to
709 <title><literal>backend_test</literal>
710 (mp::filter::Backend_test)</title>
712 A sink that provides dummy responses in the manner of the
713 <literal>yaz-ztest</literal> Z39.50 server. This is useful only
714 for testing. Seriously, you don't need this. Pretend you didn't
715 even read this section.
720 <title><literal>frontend_net</literal>
721 (mp::filter::FrontendNet)</title>
723 A source that accepts Z39.50 connections from a port
724 specified in the configuration, reads protocol units, and
725 feeds them into the next filter in the route. When the result is
726 revceived, it is returned to the original origin.
731 <title><literal>http_file</literal>
732 (mp::filter::HttpFile)</title>
734 A sink that returns the contents of files from the local
735 filesystem in response to HTTP requests. (Yes, Virginia, this
736 does mean that Metaproxy is also a Web-server in its spare time. So
737 far it does not contain either an email-reader or a Lisp
738 interpreter, but that day is surely coming.)
743 <title><literal>log</literal>
744 (mp::filter::Log)</title>
746 Writes logging information to standard output, and passes on
747 the package unchanged.
752 <title><literal>multi</literal>
753 (mp::filter::Multi)</title>
755 Performs multi-database searching.
757 <link linkend="multidb">the extended discussion</link>
758 of virtual databases and multi-database searching below.
763 <title><literal>query_rewrite</literal>
764 (mp::filter::QueryRewrite)</title>
766 Rewrites Z39.50 Type-1 and Type-101 (``RPN'') queries by a
767 three-step process: the query is transliterated from Z39.50
768 packet structures into an XML representation; that XML
769 representation is transformed by an XSLT stylesheet; and the
770 resulting XML is transliterated back into the Z39.50 packet
776 <title><literal>session_shared</literal>
777 (mp::filter::SessionShared)</title>
779 When this is finished, it will implement global sharing of
780 result sets (i.e. between threads and therefore between
781 clients), yielding performance improvements especially when
782 incoming requests are from a stateless environment such as a
783 web-server, in which the client process representing a session
784 might be any one of many. However:
788 This filter is not yet completed.
794 <title><literal>template</literal>
795 (mp::filter::Template)</title>
797 Does nothing at all, merely passing the packet on. (Maybe it
798 should be called <literal>nop</literal> or
799 <literal>passthrough</literal>?) This exists not to be used, but
800 to be copied - to become the skeleton of new filters as they are
801 written. As with <literal>backend_test</literal>, this is not
802 intended for civilians.
807 <title><literal>virt_db</literal>
808 (mp::filter::Virt_db)</title>
810 Performs virtual database selection: based on the name of the
811 database in the search request, a server is selected, and its
812 address added to the request in a <literal>VAL_PROXY</literal>
813 otherInfo packet. It will subsequently be used by a
814 <literal>z3950_client</literal> filter.
816 <link linkend="multidb">the extended discussion</link>
817 of virtual databases and multi-database searching below.
822 <title><literal>z3950_client</literal>
823 (mp::filter::Z3950Client)</title>
825 Performs Z39.50 searching and retrieval by proxying the
826 packages that are passed to it. Init requests are sent to the
827 address specified in the <literal>VAL_PROXY</literal> otherInfo
828 attached to the request: this may have been specified by client,
829 or generated by a <literal>virt_db</literal> filter earlier in
830 the route. Subsequent requests are sent to the same address,
831 which is remembered at Init time in a Session object.
837 <section id="future.directions">
838 <title>Future directions</title>
840 Some other filters that do not yet exist, but which would be
841 useful, are briefly described. These may be added in future
842 releases (or may be created by third parties, as loadable
848 <term><literal>frontend_cli</literal> (source)</term>
851 Command-line interface for generating requests.
856 <term><literal>frontend_sru</literal> (source)</term>
859 Receive SRU (and perhaps SRW) requests.
864 <term><literal>sru2z3950</literal> (filter)</term>
867 Translate SRU requests into Z39.50 requests.
872 <term><literal>sru_client</literal> (sink)</term>
875 SRU searching and retrieval.
880 <term><literal>srw_client</literal> (sink)</term>
883 SRW searching and retrieval.
888 <term><literal>opensearch_client</literal> (sink)</term>
891 A9 OpenSearch searching and retrieval.
901 <chapter id="configuration">
902 <title>Configuration: the Metaproxy configuration file format</title>
906 <title>Introductory notes</title>
908 If Metaproxy is an interpreter providing operations on packages, then
909 its configuration file can be thought of as a program for that
910 interpreter. Configuration is by means of a single file, the name
911 of which is supplied as the sole command-line argument to the
912 <command>metaproxy</command> program. (See
913 <link linkend="progref">the reference guide</link>
914 below for more information on invoking Metaproxy.)
917 The configuration files are written in XML. (But that's just an
918 implementation detail - they could just as well have been written
919 in YAML or Lisp-like S-expressions, or in a custom syntax.)
922 Since XML has been chosen, an XML schema,
923 <filename>config.xsd</filename>, is provided for validating
924 configuration files. This file is supplied in the
925 <filename>etc</filename> directory of the Metaproxy distribution. It
926 can be used by (among other tools) the <command>xmllint</command>
927 program supplied as part of the <literal>libxml2</literal>
931 xmllint --noout --schema etc/config.xsd my-config-file.xml
934 (A recent version of <literal>libxml2</literal> is required, as
935 support for XML Schemas is a relatively recent addition.)
939 <section id="overview.xml.structure">
940 <title>Overview of XML structure</title>
942 All elements and attributes are in the namespace
943 <ulink url="http://indexdata.dk/yp2/config/1"/>.
944 This is most easily achieved by setting the default namespace on
945 the top-level element, as here:
948 <yp2 xmlns="http://indexdata.dk/yp2/config/1">
951 The top-level element is <yp2>. This contains a
952 <start> element, a <filters> element and a
953 <routes> element, in that order. <filters> is
954 optional; the other two are mandatory. All three are
958 The <start> element is empty, but carries a
959 <literal>route</literal> attribute, whose value is the name of
960 route at which to start running - analogous to the name of the
961 start production in a formal grammar.
964 If present, <filters> contains zero or more <filter>
965 elements. Each filter carries a <literal>type</literal> attribute
966 which specifies what kind of filter is being defined
967 (<literal>frontend_net</literal>, <literal>log</literal>, etc.)
968 and contain various elements that provide suitable configuration
969 for a filter of its type. The filter-specific elements are
971 <link linkend="filterref">the reference guide below</link>.
972 Filters defined in this part of the file must carry an
973 <literal>id</literal> attribute so that they can be referenced
977 <routes> contains one or more <route> elements, each
978 of which must carry an <literal>id</literal> element. One of the
979 routes must have the ID value that was specified as the start
980 route in the <start> element's <literal>route</literal>
981 attribute. Each route contains zero or more <filter>
982 elements. These are of two types. They may be empty, but carry a
983 <literal>refid</literal> attribute whose value is the same as the
984 <literal>id</literal> of a filter previously defined in the
985 <filters> section. Alternatively, a route within a filter
986 may omit the <literal>refid</literal> attribute, but contain
987 configuration elements similar to those used for filters defined
988 in the <filters> section. (In other words, each filter in a
989 route may be included either by reference or by physical
995 <section id="example.configuration">
996 <title>An example configuration</title>
998 The following is a small, but complete, Metaproxy configuration
999 file (included in the distribution as
1000 <literal>metaproxy/etc/config0.xml</literal>).
1001 This file defines a very simple configuration that simply proxies
1002 to whatever back-end server the client requests, but logs each
1003 request and response. This can be useful for debugging complex
1004 client-server dialogues.
1006 <screen><![CDATA[<?xml version="1.0"?>
1007 <yp2 xmlns="http://indexdata.dk/yp2/config/1">
1008 <start route="start"/>
1010 <filter id="frontend" type="frontend_net">
1013 <filter id="backend" type="z3950_client">
1018 <filter refid="frontend"/>
1019 <filter type="log"/>
1020 <filter refid="backend"/>
1026 It works by defining a single route, called
1027 <literal>start</literal>, which consists of a sequence of three
1028 filters. The first and last of these are included by reference:
1029 their <literal><filter></literal> elements have
1030 <literal>refid</literal> attributes that refer to filters defined
1031 within the prior <literal><filters></literal> section. The
1032 middle filter is included inline in the route.
1035 The three filters in the route are as follows: first, a
1036 <literal>frontend_net</literal> filter accepts Z39.50 requests
1037 from any host on port 9000; then these requests are passed through
1038 a <literal>log</literal> filter that emits a message for each
1039 request; they are then fed into a <literal>z3950_client</literal>
1040 filter, which forwards the requests to the client-specified
1041 back-end Z39.509 server. When the response arrives, it is handed
1042 back to the <literal>log</literal> filter, which emits another
1043 message; and then to the front-end filter, which returns the
1044 response to the client.
1051 <chapter id="multidb">
1052 <title>Virtual databases and multi-database searching</title>
1056 <title>Introductory notes</title>
1058 Two of Metaproxy's filters are concerned with multiple-database
1059 operations. Of these, <literal>virt_db</literal> can work alone
1060 to control the routing of searches to one of a number of servers,
1061 while <literal>multi</literal> can work together with
1062 <literal>virt_db</literal> to perform multi-database searching, merging
1063 the results into a unified result-set - ``metasearch in a box''.
1066 The interaction between
1067 these two filters is necessarily complex: it reflects the real,
1068 irreducible complexity of multi-database searching in a protocol such
1069 as Z39.50 that separates initialisation from searching, and in
1070 which the database to be searched is not known at initialisation
1074 It's possible to use these filters without understanding the
1075 details of their functioning and the interaction between them; the
1076 next two sections of this chapter are ``HOWTO'' guides for doing
1077 just that. However, debugging complex configurations will require
1078 a deeper understanding, which the last two sections of this
1079 chapters attempt to provide.
1084 <section id="multidb.virt_db">
1085 <title>Virtual databases with the <literal>virt_db</literal> filter</title>
1087 Working alone, the purpose of the
1088 <literal>virt_db</literal>
1089 filter is to route search requests to one of a selection of
1090 back-end databases. In this way, a single Z39.50 endpoint
1091 (running Metaproxy) can provide access to several different
1092 underlying services, including those that would otherwise be
1093 inaccessible due to firewalls. In many useful configurations, the
1094 back-end databases are local to the Metaproxy installation, but
1095 the software does not enforce this, and any valid Z39.50 servers
1096 may be used as back-ends.
1099 For example, a <literal>virt_db</literal>
1100 filter could be set up so that searches in the virtual database
1101 ``lc'' are forwarded to the Library of Congress bibliographic
1102 catalogue server, and searches in the virtual database ``marc''
1103 are forwarded to the toy database of MARC records that Index Data
1104 hosts for testing purposes. A <literal>virt_db</literal>
1105 configuration to make this switch would look like this:
1107 <screen><![CDATA[<filter type="virt_db">
1109 <database>lc</database>
1110 <target>z3950.loc.gov:7090/voyager</target>
1113 <database>marc</database>
1114 <target>indexdata.dk/marc</target>
1116 </filter>]]></screen>
1118 As well as being useful in it own right, this filter also provides
1119 the foundation for multi-database searching.
1124 <section id="multidb.multi">
1125 <title>Multi-database search with the <literal>multi</literal> filter</title>
1127 To arrange for Metaproxy to broadcast searches to multiple back-end
1128 servers, the configuration needs to include two components: a
1129 <literal>virt_db</literal>
1130 filter that specifies multiple
1131 <literal><target></literal>
1132 elements, and a subsequent
1133 <literal>multi</literal>
1134 filter. Here, for example, is a complete configuration that
1135 broadcasts searches to both the Library of Congress catalogue and
1136 Index Data's tiny testing database of MARC records:
1138 <screen><![CDATA[<?xml version="1.0"?>
1139 <yp2 xmlns="http://indexdata.dk/yp2/config/1">
1140 <start route="start"/>
1143 <filter type="frontend_net">
1144 <threads>10</threads>
1147 <filter type="virt_db">
1149 <database>lc</database>
1150 <target>z3950.loc.gov:7090/voyager</target>
1153 <database>marc</database>
1154 <target>indexdata.dk/marc</target>
1157 <database>all</database>
1158 <target>z3950.loc.gov:7090/voyager</target>
1159 <target>indexdata.dk/marc</target>
1162 <filter type="multi"/>
1163 <filter type="z3950_client">
1164 <timeout>30</timeout>
1171 <literal>virt_db</literal>
1172 filter that specifies multiple
1173 <literal><target></literal>
1174 elements but without a subsequent
1175 <literal>multi</literal>
1176 filter yields surprising and undesirable results, as will be
1177 described below. Don't do that.)
1180 Metaproxy can be invoked with this configuration as follows:
1182 <screen>../src/metaproxy --config config-simple-multi.xml</screen>
1184 And thereafter, Z39.50 clients can connect to the running server
1185 (on port 9000, as specified in the configuration) and search in
1186 any of the databases
1187 <literal>lc</literal> (the Library of Congress catalogue),
1188 <literal>marc</literal> (Index Data's test database of MARC records)
1190 <literal>all</literal> (both of these). As an example, a session
1191 using the YAZ command-line client <literal>yaz-client</literal> is
1192 here included (edited for brevity and clarity):
1194 <screen><![CDATA[$ yaz-client @:9000
1198 Search was a success.
1199 Number of hits: 10000, setno 1
1203 Search was a success.
1204 Number of hits: 10, setno 3
1208 Search was a success.
1209 Number of hits: 10010, setno 4
1212 [marc]Record type: USmarc
1215 005 00000000000000.0
1216 008 910710c19910701nju 00010 eng
1220 100 10 $a Jack Collins
1221 245 10 $a How to program a computer
1227 [VOYAGER]Record type: USmarc
1229 005 20041229102447.0
1230 008 030910s2004 caua 000 0 eng
1231 035 $a (DLC) 2003112666
1232 906 $a 7 $b cbc $c orignew $d 4 $e epcn $f 20 $g y-gencatlg
1233 925 0 $a acquire $b 1 shelf copy $x policy default
1234 955 $a pc10 2003-09-10 $a pv12 2004-06-23 to SSCD; $h sj05 2004-11-30 $e sj05 2004-11-30 to Shelf.
1237 040 $a DLC $c DLC $d DLC
1238 050 00 $a MLCM 2004/03312 (G)
1239 245 10 $a 007, everything or nothing : $b Prima's official strategy guide / $c created by Kaizen Media Group.
1240 246 3 $a Double-O-seven, everything or nothing
1241 246 30 $a Prima's official strategy guide
1242 260 $a Roseville, CA : $b Prima Games, $c c2004.
1243 300 $a 161 p. : $b col. ill. ; $c 28 cm.
1244 500 $a "Platforms: Nintendo GameCube, Macintosh, PC, PlayStation 2 computer entertainment system, Xbox"--P. [4] of cover.
1245 650 0 $a Video games.
1246 710 2 $a Kaizen Media Group.
1247 856 42 $3 Publisher description $u http://www.loc.gov/catdir/description/random052/2003112666.html
1252 As can be seen, the first record in the result set is from the
1253 Index Data test database, and the second from the Library of
1254 Congress database. The result-set continues alternating records
1255 round-robin style until the point where one of the databases'
1256 records are exhausted.
1259 This example uses only two back-end databases; more may be used.
1260 There is no limitation imposed on the number of databases that may
1261 be metasearched in this way: issues of resource usage and
1262 administrative complexity dictate the practical limits.
1265 What happens when one of the databases doesn't respond? By default,
1266 the entire multi-database search fails, and the appropriate
1267 diagnostic is returned to the client. This is usually appropriate
1268 during development, when technicians need maximum information, but
1269 can be inconvenient in deployment, when users typically don't want
1270 to be bothered with problems of this kind and prefer just to get
1271 the records from the databases that are available. To obtain this
1272 latter behaviour add an empty
1273 <literal><hideunavailable></literal>
1275 <literal>multi</literal> filter:
1277 <screen><![CDATA[ <filter type="multi">
1279 </filter>]]></screen>
1281 Under this regime, an error is reported to the client only if
1282 <emphasis>all</emphasis> the databases in a multi-database search
1288 <section id="multidb.what">
1289 <title>What's going on?</title>
1291 <title>Lark's vomit</title>
1293 This section goes into a level of technical detail that is
1294 probably not necessary in order to configure and use Metaproxy.
1295 It is provided only for those who like to know how things work.
1296 You should feel free to skip on to the next section if this one
1297 doesn't seem like fun.
1301 Hold on tight - this may get a little hairy.
1304 In the general course of things, a Z39.50 Init request may carry
1305 with it an otherInfo packet of type <literal>VAL_PROXY</literal>,
1306 whose value indicates the address of a Z39.50 server to which the
1307 ultimate connection is to be made. (This otherInfo packet is
1308 supported by YAZ-based Z39.50 clients and servers, but has not yet
1309 been ratified by the Maintenance Agency and so is not widely used
1310 in non-Index Data software. We're working on it.)
1311 The <literal>VAL_PROXY</literal> packet functions
1312 analogously to the absoluteURI-style Request-URI used with the GET
1313 method when a web browser asks a proxy to forward its request: see
1315 <ulink url="http://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html#sec5.1.2"
1316 >Request-URI</ulink>
1318 <ulink url="http://www.w3.org/Protocols/rfc2616/rfc2616.html"
1319 >the HTTP 1.1 specification</ulink>.
1322 Within Metaproxy, Search requests that are part of the same
1323 session as an Init request that carries a
1324 <literal>VAL_PROXY</literal> otherInfo are also annotated with the
1325 same information. The role of the <literal>virt_db</literal>
1326 filter is to rewrite this otherInfo packet dependent on the
1327 virtual database that the client wants to search.
1330 When Metaproxy receives a Z39.50 Init request from a client, it
1331 doesn't immediately forward that request to the back-end server.
1332 Why not? Because it doesn't know <emphasis>which</emphasis>
1333 back-end server to forward it to until the client sends a Search
1334 request that specifies the database that it wants to search in.
1335 Instead, it just treasures the Init request up in its heart; and,
1336 later, the first time the client does a search on one of the
1337 specified virtual databases, a connection is forged to the
1338 appropriate server and the Init request is forwarded to it. If,
1339 later in the session, the same client searches in a different
1340 virtual database, then a connection is forged to the server that
1341 hosts it, and the same cached Init request is forwarded there,
1345 All of this clever Init-delaying is done by the
1346 <literal>frontend_net</literal> filter. The
1347 <literal>virt_db</literal> filter knows nothing about it; in
1348 fact, because the Init request that is received from the client
1349 doesn't get forwarded until a Search request is received, the
1350 <literal>virt_db</literal> filter (and the
1351 <literal>z3950_client</literal> filter behind it) doesn't even get
1352 invoked at Init time. The <emphasis>only</emphasis> thing that a
1353 <literal>virt_db</literal> filter ever does is rewrite the
1354 <literal>VAL_PROXY</literal> otherInfo in the requests that pass
1358 It is possible for a <literal>virt_db</literal> filter to contain
1360 <literal><target></literal>
1361 elements. What does this mean? Only that the filter will add
1362 multiple <literal>VAL_PROXY</literal> otherInfo packets to the
1363 Search requests that pass through it. That's because the virtual
1364 DB filter is dumb, and does exactly what it's told - no more, no
1366 If a Search request with multiple <literal>VAL_PROXY</literal>
1367 otherInfo packets reaches a <literal>z3950_client</literal>
1368 filter, this is an error. That filter doesn't know how to deal
1369 with multiple targets, so it will either just pick one and search
1370 in it, or (better) fail with an error message.
1373 The <literal>multi</literal> filter comes to the rescue! This is
1374 the only filter that knows how to deal with multiple
1375 <literal>VAL_PROXY</literal> otherInfo packets, and it does so by
1376 making multiple copies of the entire Search request: one for each
1377 <literal>VAL_PROXY</literal>. Each of these new copies is then
1378 passed down through the remaining filters in the route. (The
1379 copies are handled in parallel though the
1380 spawning of new threads.) Since the copies each have only one
1381 <literal>VAL_PROXY</literal> otherInfo, they can be handled by the
1382 <literal>z3950_client</literal> filter, which happily deals with
1383 each one individually. When the results of the individual
1384 searches come back up to the <literal>multi</literal> filter, it
1385 merges them into a single Search response, which is what
1386 eventually makes it back to the client.
1391 <section id="multidb.picture">
1392 <title>A picture is worth a thousand words (but only five hundred on 64-bit architectures)</title>
1396 <imagedata fileref="multi.pdf" format="PDF" scale="50"/>
1399 <imagedata fileref="multi.png" format="PNG"/>
1402 <!-- Fall back if none of the images can be used -->
1404 [Here there should be a diagram showing the progress of
1405 packages through the filters during a simple virtual-database
1406 search and a multi-database search, but is seems that your
1407 toolchain has not been able to include the diagram in this
1408 document. This is because of LaTeX suckage. Time to move to
1409 OpenOffice. Yes, really.]
1412 <!-- ### This used to work with an older version of DocBook
1414 <para>Caption: progress of packages through filters.</para>
1417 </inlinemediaobject>
1424 <chapter id="extensions">
1425 <title>Writing extensions for Metaproxy</title>
1426 <para>### To be written</para>
1432 <chapter id="classes">
1433 <title>Classes in the Metaproxy source code</title>
1437 <title>Introductory notes</title>
1439 <emphasis>Stop! Do not read this!</emphasis>
1440 You won't enjoy it at all. You should just skip ahead to
1441 <link linkend="refguide">the reference guide</link>,
1443 <!-- The remainder of this paragraph is lifted verbatim from
1444 Douglas Adams' _Hitch Hiker's Guide to the Galaxy_, chapter 8 -->
1445 you things you really need to know, like the fact that the
1446 fabulously beautiful planet Bethselamin is now so worried about
1447 the cumulative erosion by ten billion visiting tourists a year
1448 that any net imbalance between the amount you eat and the amount
1449 you excrete whilst on the planet is surgically removed from your
1450 bodyweight when you leave: so every time you go to the lavatory it
1451 is vitally important to get a receipt.
1454 This chapter contains documentation of the Metaproxy source code, and is
1455 of interest only to maintainers and developers. If you need to
1456 change Metaproxy's behaviour or write a new filter, then you will most
1457 likely find this chapter helpful. Otherwise it's a waste of your
1458 good time. Seriously: go and watch a film or something.
1459 <citetitle>This is Spinal Tap</citetitle> is particularly good.
1462 Still here? OK, let's continue.
1465 In general, classes seem to be named big-endianly, so that
1466 <literal>FactoryFilter</literal> is not a filter that filters
1467 factories, but a factory that produces filters; and
1468 <literal>FactoryStatic</literal> is a factory for the statically
1469 registered filters (as opposed to those that are dynamically
1474 <section id="individual.classes">
1475 <title>Individual classes</title>
1477 The classes making up the Metaproxy application are here listed by
1478 class-name, with the names of the source files that define them in
1483 <title><literal>mp::FactoryFilter</literal>
1484 (<filename>factory_filter.cpp</filename>)</title>
1486 A factory class that exists primarily to provide the
1487 <literal>create()</literal> method, which takes the name of a
1488 filter class as its argument and returns a new filter of that
1489 type. To enable this, the factory must first be populated by
1490 calling <literal>add_creator()</literal> for static filters (this
1491 is done by the <literal>FactoryStatic</literal> class, see below)
1492 and <literal>add_creator_dyn()</literal> for filters loaded
1498 <title><literal>mp::FactoryStatic</literal>
1499 (<filename>factory_static.cpp</filename>)</title>
1501 A subclass of <literal>FactoryFilter</literal> which is
1502 responsible for registering all the statically defined filter
1503 types. It does this by knowing about all those filters'
1504 structures, which are listed in its constructor. Merely
1505 instantiating this class registers all the static classes. It is
1506 for the benefit of this class that <literal>struct
1507 metaproxy_1_filter_struct</literal> exists, and that all the filter
1508 classes provide a static object of that type.
1513 <title><literal>mp::filter::Base</literal>
1514 (<filename>filter.cpp</filename>)</title>
1516 The virtual base class of all filters. The filter API is, on the
1517 surface at least, extremely simple: two methods.
1518 <literal>configure()</literal> is passed a DOM tree representing
1519 that part of the configuration file that pertains to this filter
1520 instance, and is expected to walk that tree extracting relevant
1521 information. And <literal>process()</literal> processes a
1522 package (see below). That surface simplicitly is a bit
1523 misleading, as <literal>process()</literal> needs to know a lot
1524 about the <literal>Package</literal> class in order to do
1530 <title><literal>mp::filter::AuthSimple</literal>,
1531 <literal>Backend_test</literal>, etc.
1532 (<filename>filter_auth_simple.cpp</filename>,
1533 <filename>filter_backend_test.cpp</filename>, etc.)</title>
1535 Individual filters. Each of these is implemented by a header and
1536 a source file, named <filename>filter_*.hpp</filename> and
1537 <filename>filter_*.cpp</filename> respectively. All the header
1538 files should be pretty much identical, in that they declare the
1539 class, including a private <literal>Rep</literal> class and a
1540 member pointer to it, and the two public methods. The only extra
1541 information in any filter header is additional private types and
1542 members (which should really all be in the <literal>Rep</literal>
1543 anyway) and private methods (which should also remain known only
1544 to the source file, but C++'s brain-damaged design requires this
1545 dirty laundry to be exhibited in public. Thanks, Bjarne!)
1548 The source file for each filter needs to supply:
1553 A definition of the private <literal>Rep</literal> class.
1558 Some boilerplate constructors and destructors.
1563 A <literal>configure()</literal> method that uses the
1564 appropriate XML fragment.
1569 Most important, the <literal>process()</literal> method that
1570 does all the actual work.
1577 <title><literal>mp::Package</literal>
1578 (<filename>package.cpp</filename>)</title>
1580 Represents a package on its way through the series of filters
1581 that make up a route. This is essentially a Z39.50 or SRU APDU
1582 together with information about where it came from, which is
1583 modified as it passes through the various filters.
1588 <title><literal>mp::Pipe</literal>
1589 (<filename>pipe.cpp</filename>)</title>
1591 This class provides a compatibility layer so that we have an IPC
1592 mechanism that works the same under Unix and Windows. It's not
1593 particularly exciting.
1598 <title><literal>mp::RouterChain</literal>
1599 (<filename>router_chain.cpp</filename>)</title>
1606 <title><literal>mp::RouterFleXML</literal>
1607 (<filename>router_flexml.cpp</filename>)</title>
1614 <title><literal>mp::Session</literal>
1615 (<filename>session.cpp</filename>)</title>
1622 <title><literal>mp::ThreadPoolSocketObserver</literal>
1623 (<filename>thread_pool_observer.cpp</filename>)</title>
1630 <title><literal>mp::util</literal>
1631 (<filename>util.cpp</filename>)</title>
1633 A namespace of various small utility functions and classes,
1634 collected together for convenience. Most importantly, includes
1635 the <literal>mp::util::odr</literal> class, a wrapper for YAZ's
1641 <title><literal>mp::xml</literal>
1642 (<filename>xmlutil.cpp</filename>)</title>
1644 A namespace of various XML utility functions and classes,
1645 collected together for convenience.
1651 <section id="other.source.files">
1652 <title>Other Source Files</title>
1654 In addition to the Metaproxy source files that define the classes
1655 described above, there are a few additional files which are
1656 briefly described here:
1660 <term><literal>metaproxy_prog.cpp</literal></term>
1663 The main function of the <command>metaproxy</command> program.
1668 <term><literal>ex_router_flexml.cpp</literal></term>
1671 Identical to <literal>metaproxy_prog.cpp</literal>: it's not clear why.
1676 <term><literal>test_*.cpp</literal></term>
1679 Unit-tests for various modules.
1685 ### Still to be described:
1686 <literal>ex_filter_frontend_net.cpp</literal>,
1687 <literal>filter_dl.cpp</literal>,
1688 <literal>plainfile.cpp</literal>,
1689 <literal>tstdl.cpp</literal>.
1696 <chapter id="refguide">
1697 <title>Reference guide</title>
1699 The material in this chapter is drawn directly from the individual
1700 manual entries. In particular, the Metaproxy invocation section is
1701 available using <command>man metaproxy</command>, and the section
1702 on each individual filter is available using the name of the filter
1703 as the argument to the <command>man</command> command.
1707 <section id="progref">
1708 <title>Metaproxy invocation</title>
1713 <section id="filterref">
1714 <title>Reference guide to Metaproxy filters</title>
1720 <!-- Keep this comment at the end of the file
1725 sgml-minimize-attributes:nil
1726 sgml-always-quote-attributes:t
1729 sgml-parent-document: nil
1730 sgml-local-catalogs: nil
1731 sgml-namecase-general:t