1 <?xml version="1.0" standalone="no"?>
2 <!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.1//EN"
3 "http://www.oasis-open.org/docbook/xml/4.1/docbookx.dtd"
5 <!ENTITY % local SYSTEM "local.ent">
7 <!ENTITY manref SYSTEM "manref.xml">
8 <!ENTITY progref SYSTEM "progref.xml">
9 <!ENTITY % common SYSTEM "common/common.ent">
11 <!-- Next line allows imagedata/@format="PDF" and is taken from
12 http://lists.oasis-open.org/archives/docbook/200303/msg00163.html
14 <!ENTITY % local.notation.class "| PDF">
15 <!-- Next line is necessary for some XML parsers, for reasons I
16 don't understand. I got this from
17 http://lists.oasis-open.org/archives/docbook/200303/msg00180.html
19 <!NOTATION PDF SYSTEM "PDF">
21 <!-- $Id: book.xml,v 1.57 2007-03-30 11:35:04 marc Exp $ -->
24 <title>Metaproxy - User's Guide and Reference</title>
27 <firstname>Adam</firstname><surname>Dickmeiss</surname>
30 <firstname>Marc</firstname><surname>Cromme</surname>
33 <firstname>Mike</firstname><surname>Taylor</surname>
36 <releaseinfo>&version;</releaseinfo>
38 <year>2005-2007</year>
39 <holder>Index Data ApS</holder>
43 This manual is part of Metaproxy version &version;.
46 Metaproxy is a universal router, proxy and encapsulated
47 metasearcher for information retrieval protocols. It accepts,
48 processes, interprets and redirects requests from IR clients using
49 standard protocols such as the binary
50 <ulink url="&url.z39.50;">ANSI/NISO Z39.50</ulink>
51 and the information search and retireval
52 web services <ulink url="&url.sru;">SRU</ulink>
53 and <ulink url="&url.srw;">SRW</ulink>, as
54 well as functioning as a limited
55 <ulink url="&url.http;">HTTP</ulink> server.
58 Metaproxy is configured by an XML file which
59 specifies how the software should function in terms of routes that
60 the request packets can take through the proxy, each step on a
61 route being an instantiation of a filter. Filters come in many
62 types, one for each operation: accepting Z39.50 packets, logging,
63 query transformation, multiplexing, etc. Further filter-types can
64 be added as loadable modules to extend Metaproxy functionality,
68 Metaproxy is <emphasis>not</emphasis> open-source software, but
69 may be freely downloaded, unpacked, inspected, built and run for
70 evaluation purposes. Deployment requires a separate, commercial,
76 <imagedata fileref="common/id.png" format="PNG"/>
79 <imagedata fileref="common/id.eps" format="EPS"/>
86 <chapter id="introduction">
87 <title>Introduction</title>
91 <ulink url="&url.metaproxy;">Metaproxy</ulink>
92 is a stand alone program that acts as a universal router, proxy and
93 encapsulated metasearcher for information retrieval protocols such
94 as <ulink url="&url.z39.50;">Z39.50</ulink>, and in the future
95 <ulink url="&url.sru;">SRU</ulink> and <ulink url="&url.srw;">SRW</ulink>.
96 To clients, it acts as a server of these protocols: it can be searched,
97 records can be retrieved from it, etc.
98 To servers, it acts as a client: it searches in them,
99 retrieves records from them, etc. it satisfies its clients'
100 requests by transforming them, multiplexing them, forwarding them
101 on to zero or more servers, merging the results, transforming
102 them, and delivering them back to the client. In addition, it
103 acts as a simple <ulink url="&url.http;">HTTP</ulink> server; support
104 for further protocols can be added in a modular fashion, through the
105 creation of new filters.
110 Fish, bananas, cold pyjamas,
111 Mutton, beef and trout!
112 - attributed to Cole Porter.
115 Metaproxy is a more capable alternative to
116 <ulink url="&url.yazproxy;">YAZ Proxy</ulink>,
117 being more powerful, flexible, configurable and extensible. Among
118 its many advantages over the older, more pedestrian work are
119 support for multiplexing (encapsulated metasearching), routing by
120 database name, authentication and authorization and serving local
121 files via HTTP. Equally significant, its modular architecture
122 facilitites the creation of pluggable modules implementing further
126 This manual will describe how to install Metaproxy
127 before giving an overview of its architecture, then discussing the
128 key concept of a filter in some depth and giving an overview of
129 the various filter types, then discussing the configuration file
130 format. After this come several optional chapters which may be
131 freely skipped: a detailed discussion of virtual databases and
132 multi-database searching, some notes on writing extensions
133 (additional filter types) and a high-level description of the
134 source code. Finally comes the reference guide, which contains
135 instructions for invoking the <command>metaproxy</command>
136 program, and detailed information on each type of filter,
141 <chapter id="license">
142 <title>The Metaproxy License</title>
143 <orderedlist numeration="arabic">
146 You are allowed to download this software for evaluation purposes.
147 You can unpack it, build it, run it, see how it works and how it fits
148 your needs, all at zero cost.
153 You may NOT deploy the software. For the purposes of this license,
154 deployment means running it for any purpose other than evaluation,
155 whether or not you or anyone else makes a profit from doing so. If
156 you wish to deploy the software, you must first contact Index Data and
157 arrange to purchase a DEPLOYMENT LICENCE. If you are unsure
158 whether or not your proposed use of the software constitutes
159 deployment, email us at <literal>info@indexdata.com</literal>
165 You may modify your copy of the software (fix bugs, add features)
166 if you need to. We encourage you to send your changes back to us for
167 integration into the master copy, but you are not obliged to do so. You
168 may NOT pass your changes on to any other party.
173 There is NO WARRANTY for this software, to the extent permitted by
174 applicable law. We provide the software ``as is'' without warranty of
175 any kind, either expressed or implied, including, but not limited to, the
176 implied warranties of MERCHANTABILITY and FITNESS FOR A
177 PARTICULAR PURPOSE. The entire risk as to the quality and
178 performance of the software is with you. Should the software prove
179 defective, you assume the cost of all necessary servicing, repair or
180 correction. In no event unless required by applicable law will we be
181 liable to you for damages, arising out of the use of the software,
182 including but not limited to loss of data or data being rendered
188 All rights to the software are reserved by Index Data except where
189 this license explicitly says otherwise.
195 <chapter id="installation">
196 <title>Installation</title>
198 Metaproxy depends on the following tools/libraries:
200 <varlistentry><term><ulink url="&url.yazplusplus;">YAZ++</ulink></term>
203 This is a C++ library based on <ulink url="&url.yaz;">YAZ</ulink>.
207 <varlistentry><term><ulink url="&url.libxslt;">Libxslt</ulink></term>
209 <para>This is an XSLT processor - based on
210 <ulink url="&url.libxml2;">Libxml2</ulink>. Both Libxml2 and
211 Libxslt must be installed with the development components
212 (header files, etc.) as well as the run-time libraries.
216 <varlistentry><term><ulink url="&url.boost;">Boost</ulink></term>
219 The popular C++ library. Initial versions of Metaproxy
220 was built with 1.33.0. Version 1.33.1 works too.
227 In order to compile Metaproxy a modern C++ compiler is
228 required. Boost, in particular, requires the C++ compiler
229 to facilitate the newest features. Refer to Boost
230 <ulink url="&url.boost.compilers.status;">Compiler Status</ulink>
231 for more information.
234 We have successfully built Metaproxy using the compilers
235 <ulink url="&url.gcc;">GCC</ulink> version 4.0 and
236 <ulink url="&url.vstudio;">Microsoft Visual Studio</ulink> 2003/2005.
239 <section id="installation.unix">
240 <title>Installation on Unix (from Source)</title>
242 Here is a quick step-by-step guide on how to compile all the
243 tools that Metaproxy uses. Only few systems have none of the required
244 tools binary packages. If, for example, Libxml2/libxslt are already
245 installed as development packages use those (and omit compilation).
252 gunzip -c libxml2-version.tar.gz|tar xf -
260 gunzip -c libxslt-version.tar.gz|tar xf -
271 gunzip -c yaz-version.tar.gz|tar xf -
279 gunzip -c yazpp-version.tar.gz|tar xf -
290 gunzip -c boost-version.tar.gz|tar xf -
301 gunzip -c metaproxy-version.tar.gz|tar xf -
310 <section id="installation.debian">
311 <title>Installation on Debian GNU/Linux</title>
313 All dependencies for Metaproxy are available as
314 <ulink url="&url.debian;">Debian</ulink>
315 packages for the sarge (stable in 2005) and etch (testing in 2005)
319 The procedures for Debian based systems, such as
320 <ulink url="&url.ubuntu;">Ubuntu</ulink> is probably similar
323 There is currently no official Debian package for YAZ++.
324 And the Debian package for YAZ is probably too old.
325 Update the <filename>/etc/apt/sources.list</filename>
326 to include the Index Data repository.
327 See YAZ' <ulink url="&url.yaz.download.debian;">Download Debian</ulink>
328 for more information.
331 apt-get install libxslt1-dev
332 apt-get install libyazpp-dev
333 apt-get install libboost-dev
334 apt-get install libboost-thread-dev
335 apt-get install libboost-date-time-dev
336 apt-get install libboost-program-options-dev
337 apt-get install libboost-test-dev
340 With these packages installed, the usual configure + make
341 procedure can be used for Metaproxy as outlined in
342 <xref linkend="installation.unix"/>.
346 <section id="installation.rpm">
347 <title>Installation on RPM based Linux Systems</title>
349 All external dependencies for Metaproxy are available as
350 RPM packages, either from your distribution site, or from the
351 <ulink url="http://fr.rpmfind.net/">RPMfind</ulink> site.
354 For example, an installation of the requires Boost C++ development
355 libraries on RedHat Fedora C4 and C5 can be done like this:
357 wget ftp://fr.rpmfind.net/wlinux/fedora/core/updates/testing/4/SRPMS/boost-1.33.0-3.fc4.src.rpm
358 sudo rpmbuild --buildroot src/ --rebuild -p fc4/boost-1.33.0-3.fc4.src.rpm
359 sudo rpm -U /usr/src/redhat/RPMS/i386/boost-*rpm
363 The <ulink url="&url.yaz;">YAZ</ulink> library is needed to
364 compile &metaproxy;, see there
365 for more information on available RPM packages.
368 There is currently no official RPM package for YAZ++.
369 See the <ulink url="&url.yaz.pp;">YAZ++</ulink> pages
370 for more information on a Unix tarball install.
373 With these packages installed, the usual configure + make
374 procedure can be used for Metaproxy as outlined in
375 <xref linkend="installation.unix"/>.
379 <section id="installation.windows">
380 <title>Installation on Windows</title>
382 Metaproxy can be compiled with Microsoft
383 <ulink url="&url.vstudio;">Visual Studio</ulink>.
384 Version 2003 (C 7.1) and 2005 (C 8.0) is known to work.
386 <section id="installation.windows.boost">
389 Get Boost from its <ulink url="&url.boost;">home page</ulink>.
390 You also need Boost Jam (an alternative to make).
391 That's also available from the Boost home page.
392 The files to be downloaded are called something like:
393 <filename>boost_1_33-1.exe</filename>
395 <filename>boost-jam-3.1.12-1-ntx86.zip</filename>.
396 Unpack Boost Jam first. Put <filename>bjam.exe</filename>
397 in your system path. Make a command prompt and ensure
398 it can be found automatically. If not check the PATH.
399 The Boost .exe is a self-extracting exe with
400 complete source for Boost. Compile that source with
401 Boost Jam (An alternative to Make).
402 The compilation takes a while.
403 For Visual Studio 2003, use
405 bjam "-sTOOLS=vc-7_1"
407 Here <literal>vc-7_1</literal> refers to a "Toolset" (compiler system).
408 For Visual Studio 2005, use
410 bjam "-sTOOLS=vc-8_0"
412 To install the libraries in a common place, use
414 bjam "-sTOOLS=vc-7_1" install
416 (or vc-8_0 for VS 2005).
419 By default, the Boost build process installs the resulting
420 libraries + header files in
421 <literal>\boost\lib</literal>, <literal>\boost\include</literal>.
424 For more information about installing Boost refer to the
425 <ulink url="&url.boost.getting.started;">getting started</ulink>
430 <section id="installation.windows.libxslt">
431 <title>Libxslt</title>
433 <ulink url="&url.libxslt;">Libxslt</ulink> can be downloaded
435 <ulink url="&url.libxml2.download.win32;">here</ulink>.
438 Libxslt has other dependencies, but these can all be downloaded
439 from the same site. Get the following:
440 iconv, zlib, libxml2, libxslt.
444 <section id="installation.windows.yaz">
447 <ulink url="&url.yaz;">YAZ</ulink> can be downloaded
449 <ulink url="&url.yaz.download.win32;">here</ulink>.
453 <section id="installation.windows.yazplusplus">
456 Get <ulink url="&url.yazplusplus;">YAZ++</ulink> as well.
457 Version 1.0 or later is required. For now get it from
459 <ulink url="&url.snapshot.download;">Snapshot area</ulink>.
462 YAZ++ includes NMAKE makefiles, similar to those found in the
467 <section id="installation.windows.metaproxy">
468 <title>Metaproxy</title>
470 Metaproxy is shipped with NMAKE makefiles as well - similar
471 to those found in the YAZ++/YAZ packages. Adjust this Makefile
472 to point to the proper locations of Boost, Libxslt, Libxml2,
473 zlib, iconv, yaz and yazpp.
477 <varlistentry><term><literal>DEBUG</literal></term>
479 If set to 1, the software is
480 compiled with debugging libraries (code generation is
481 multi-threaded debug DLL).
482 If set to 0, the software is compiled with release libraries
483 (code generation is multi-threaded DLL).
488 <term><literal>BOOST</literal></term>
491 Boost install location
497 <term><literal>BOOST_VERSION</literal></term>
500 Boost version (replace . with _).
506 <term><literal>BOOST_TOOLSET</literal></term>
515 <term><literal>LIBXSLT_DIR</literal>,
516 <literal>LIBXML2_DIR</literal> ..</term>
519 Specify the locations of Libxslt, libiconv, libxml2 and
528 After successful compilation you'll find
529 <literal>metaproxy.exe</literal> in the
530 <literal>bin</literal> directory.
538 <chapter id="yazproxy-comparison">
539 <title>YAZ Proxy Comparison</title>
541 The table below lists facilities either supported by either
542 <ulink url="&url.yazproxy;">YAZ Proxy</ulink> or Metaproxy.
544 <table id="yazproxy-comparison-table">
545 <title>Metaproxy / YAZ Proxy comparison</title>
549 <entry>Facility</entry>
550 <entry>Metaproxy</entry>
551 <entry>YAZ Proxy</entry>
556 <entry>Z39.50 server</entry>
557 <entry>Using filter <literal>frontend_net</literal></entry>
558 <entry>Supported</entry>
561 <entry>SRU server</entry>
562 <entry>Supported with filter <literal>sru_z3950</literal></entry>
563 <entry>Supported</entry>
566 <entry>Z39.50 client</entry>
567 <entry>Supported with filter <literal>z3950_client</literal></entry>
568 <entry>Supported</entry>
571 <entry>SRU client</entry>
572 <entry>Unsupported</entry>
573 <entry>Unsupported</entry>
576 <entry>Connection reuse</entry>
577 <entry>Supported with filter <literal>session_shared</literal></entry>
578 <entry>Supported</entry>
581 <entry>Connection share</entry>
582 <entry>Supported with filter <literal>session_shared</literal></entry>
583 <entry>Unsupported</entry>
586 <entry>Result set reuse</entry>
587 <entry>Supported with filter <literal>session_shared</literal></entry>
588 <entry>Within one Z39.50 session / HTTP keep-alive</entry>
591 <entry>Record cache</entry>
592 <entry>Unsupported</entry>
593 <entry>Supported for last result set within one Z39.50/HTTP-keep alive session</entry>
596 <entry>Z39.50 Virtual database, i.e. select any Z39.50 target for database</entry>
597 <entry>Supported with filter <literal>virt_db</literal></entry>
598 <entry>Unsupported</entry>
601 <entry>SRU Virtual database, i.e. select any Z39.50 target for path</entry>
602 <entry>Supported with filter <literal>virt_db</literal>,
603 <literal>sru_z3950</literal></entry>
604 <entry>Supported</entry>
607 <entry>Multi target search</entry>
608 <entry>Supported with filter <literal>multi</literal> (round-robin)</entry>
609 <entry>Unsupported</entry>
612 <entry>Retrieval and search limits</entry>
613 <entry>Unsupported</entry>
614 <entry>Supported</entry>
617 <entry>Bandwidth limits</entry>
618 <entry>Unsupported</entry>
619 <entry>Supported</entry>
622 <entry>Connect limits</entry>
623 <entry>Unsupported</entry>
624 <entry>Supported</entry>
627 <entry>Retrieval sanity check and conversions</entry>
628 <entry>Supported using filter <literal>record_transform</literal></entry>
629 <entry>Supported</entry>
632 <entry>Query check</entry>
634 Supported in a limited way using <literal>query_rewrite</literal>
636 <entry>Supported</entry>
639 <entry>Query rewrite</entry>
640 <entry>Supported with <literal>query_rewrite</literal></entry>
641 <entry>Unsupported</entry>
644 <entry>Session invalidate for -1 hits</entry>
645 <entry>Unsupported</entry>
646 <entry>Supported</entry>
649 <entry>Architecture</entry>
650 <entry>Multi-threaded + select for networked modules such as
651 <literal>frontend_net</literal>)</entry>
652 <entry>Single-threaded using select</entry>
656 <entry>Extensability</entry>
657 <entry>Most functionality implemented as loadable modules</entry>
658 <entry>Unsupported and experimental</entry>
662 <entry><ulink url="&url.usemarcon;">USEMARCON</ulink></entry>
663 <entry>Unsupported</entry>
664 <entry>Supported</entry>
668 <entry>Portability</entry>
670 Requires YAZ, YAZ++ and modern C++ compiler supporting
671 <ulink url="&url.boost;">Boost</ulink>.
674 Requires YAZ and YAZ++.
675 STL is not required so pretty much any C++ compiler out there should work.
684 <chapter id="architecture">
685 <title>The Metaproxy Architecture</title>
687 The Metaproxy architecture is based on three concepts:
688 the <emphasis>package</emphasis>,
689 the <emphasis>route</emphasis>
690 and the <emphasis>filter</emphasis>.
694 <term>Packages</term>
697 A package is request or response, encoded in some protocol,
698 issued by a client, making its way through Metaproxy, send to or
699 received from a server, or sent back to the client.
702 The core of a package is the protocol unit - for example, a
703 Z39.50 Init Request or Search Response, or an SRU searchRetrieve
704 URL or Explain Response. In addition to this core, a package
705 also carries some extra information added and used by Metaproxy
709 In general, packages are doctored as they pass through
710 Metaproxy. For example, when the proxy performs authentication
711 and authorization on a Z39.50 Init request, it removes the
712 authentication credentials from the package so that they are not
713 passed onto the back-end server; and when search-response
714 packages are obtained from multiple servers, they are merged
715 into a single unified package that makes its way back to the
724 Packages make their way through routes, which can be thought of
725 as programs that operate on the package data-type. Each
726 incoming package initially makes its way through a default
727 route, but may be switched to a different route based on various
728 considerations. Routes are made up of sequences of filters (see
737 Filters provide the individual instructions within a route, and
738 effect the necessary transformations on packages. A particular
739 configuration of Metaproxy is essentially a set of filters,
740 described by configuration details and arranged in order in one
741 or more routes. There are many kinds of filter - about a dozen
742 at the time of writing with more appearing all the time - each
743 performing a specific function and configured by different
747 The word ``filter'' is sometimes used rather loosely, in two
748 different ways: it may be used to mean a particular
749 <emphasis>type</emphasis> of filter, as when we speak of ``the
750 auth_simple filter'' or ``the multi filter''; or it may be used
751 to be a specific <emphasis>instance</emphasis> of a filter
752 within a Metaproxy configuration. For example, a single
753 configuration will often contain multiple instances of the
754 <literal>z3950_client</literal> filter. In
755 operational terms, of these is a separate filter. In practice,
756 context always make it clear which sense of the word ``filter''
760 Extensibility of Metaproxy is primarily through the creation of
761 plugins that provide new filters. The filter API is small and
762 conceptually simple, but there are many details to master. See
764 <link linkend="filters">Filters</link>.
770 Since packages are created and handled by the system itself, and
771 routes are conceptually simple, most of the remainder of this
772 document concentrates on filters. After a brief overview of the
773 filter types follows, along with some thoughts on possible future
780 <chapter id="filters">
781 <title>Filters</title>
785 <title>Introductory notes</title>
787 It's useful to think of Metaproxy as an interpreter providing a small
788 number of primitives and operations, but operating on a very
789 complex data type, namely the ``package''.
792 A package represents a Z39.50 or SRU/W request (whether for Init,
793 Search, Scan, etc.) together with information about where it came
794 from. Packages are created by front-end filters such as
795 <literal>frontend_net</literal> (see below), which reads them from
796 the network; other front-end filters are possible. They then pass
797 along a route consisting of a sequence of filters, each of which
798 transforms the package and may also have side-effects such as
799 generating logging. Eventually, the route will yield a response,
800 which is sent back to the origin.
803 There are many kinds of filter: some that are defined statically
804 as part of Metaproxy, and others may be provided by third parties
805 and dynamically loaded. They all conform to the same simple API
806 of essentially two methods: <function>configure()</function> is
807 called at startup time, and is passed an XML DOM tree representing that
808 part of the configuration file that pertains to this filter
809 instance: it is expected to walk that tree extracting relevant
810 information; and <function>process()</function> is called every
811 time the filter has to processes a package.
814 While all filters provide the same API, there are different modes
815 of functionality. Some filters are sources: they create
817 (<literal>frontend_net</literal>);
818 others are sinks: they consume packages and return a result
819 (<literal>backend_test</literal>,
820 <literal>bounce</literal>,
821 <literal>http_file</literal>,
822 <literal>z3950_client</literal>);
823 the others are true filters, that read, process and pass on the
824 packages they are fed
825 (<literal>auth_simple</literal>,
826 <literal>log</literal>,
827 <literal>multi</literal>,
828 <literal>query_rewrite</literal>,
829 <literal>record_transform</literal>,
830 <literal>session_shared</literal>,
831 <literal>sru_z3950</literal>,
832 <literal>template</literal>,
833 <literal>virt_db</literal>).
838 <section id="overview.filter.types">
839 <title>Overview of filter types</title>
841 We now briefly consider each of the types of filter supported by
842 the core Metaproxy binary. This overview is intended to give a
843 flavor of the available functionality; more detailed information
844 about each type of filter is included below in
845 <xref linkend="reference"/>.
848 The filters are here named by the string that is used as the
849 <literal>type</literal> attribute of a
850 <literal><filter></literal> element in the configuration
851 file to request them, with the name of the class that implements
852 them in parentheses. (The classname is not needed for normal
853 configuration and use of Metaproxy; it is useful only to
857 The filters are here listed in alphabetical order:
864 New virt_db-alike that does inteligent peer choice, explain merging,
865 adds FD&N to explain. Keeps init responses (like "virt_db Classic"),
866 makes routing choices based on local explain knowledge. Ref IDDI
869 Filter to convert Explain Classic to ZeeRex.
871 CQL2PQF (which needs augmented ZeeRex) - MARC for Talis.
875 Figure out what additional information we need in:
876 ZeeRex (check against D3.1)
877 Init request (e.g. loop detection)
878 Query package (e.g. number of hops)
879 Query response (e.g. record source)
884 <title><literal>auth_simple</literal>
885 (mp::filter::AuthSimple)</title>
887 Simple authentication and authorization. The configuration
888 specifies the name of a file that is the user register, which
889 lists <varname>username</varname>:<varname>password</varname>
890 pairs, one per line, colon separated. When a session begins, it
891 is rejected unless username and passsword are supplied, and match
892 a pair in the register. The configuration file may also specific
893 the name of another file that is the target register: this lists
894 lists <varname>username</varname>:<varname>dbname</varname>,<varname>dbname</varname>...
895 sets, one per line, with multiple database names separated by
896 commas. When a search is processed, it is rejected unless the
897 database to be searched is one of those listed as available to
903 <title><literal>backend_test</literal>
904 (mp::filter::Backend_test)</title>
906 A partial sink that provides dummy responses in the manner of the
907 <literal>yaz-ztest</literal> Z39.50 server. This is useful only
908 for testing. Seriously, you don't need this. Pretend you didn't
909 even read this section.
914 <title><literal>bounce</literal>
915 (mp::filter::Bounce)</title>
917 A sink that swallows <emphasis>all packages</emphasis>,
918 and returns them almost unprocessed.
919 It never sends any package of any type further down the row, but
920 sets Z39.50 packages to Z_Close, and HTTP_Request packages to
921 HTTP_Response err code 400 packages, and adds a suitable bounce
923 The bounce filter is usually added at end of each filter chain route
924 to prevent infinite hanging of for example HTTP
925 requests packages when only the Z39.50 client partial sink
926 filter is found in the
932 <title><literal>cql_rpn</literal>
933 (mp::filter::CQLtoRPN)</title>
935 A query language transforming filter which catches Z39.50
936 <literal>searchRequest</literal>
937 packages containing <literal>CQL</literal> queries, transforms
938 those to <literal>RPN</literal> queries,
939 and sends the <literal>searchRequests</literal> on to the next
940 filters. It is among other things useful in a SRU context.
945 <title><literal>frontend_net</literal>
946 (mp::filter::FrontendNet)</title>
948 A source that accepts Z39.50 connections from a port
949 specified in the configuration, reads protocol units, and
950 feeds them into the next filter in the route. When the result is
951 received, it is returned to the original origin.
956 <title><literal>http_file</literal>
957 (mp::filter::HttpFile)</title>
959 A partial sink which swallows only
960 <literal>HTTP_Request</literal> packages, and
961 returns the contents of files from the local
962 filesystem in response to HTTP requests.
963 It lets Z39.50 packages and all other forthcoming package types
966 does mean that Metaproxy is also a Web-server in its spare time. So
967 far it does not contain either an email-reader or a Lisp
968 interpreter, but that day is surely coming.)
973 <title><literal>load_balance</literal>
974 (mp::filter::LoadBalance)</title>
976 Performs load balancing for incoming Z39.50 init requests.
977 It is used together with the <literal>virt_db</literal> filter,
978 but unlike the <literal>multi</literal> filter it does send an
979 entire session to only one of the virtual backends. The
980 <literal>load_balance</literal> filter is assuming that
981 all backend targets have equal content, and chooses the backend
982 with least load cost for a new session.
985 This filter is experimental and yet not mature for heavy load
993 <title><literal>log</literal>
994 (mp::filter::Log)</title>
996 Writes logging information to standard output, and passes on
997 the package unchanged. A log file name can be specified, as well
998 as multiple different logging formats.
1003 <title><literal>multi</literal>
1004 (mp::filter::Multi)</title>
1006 Performs multi-database searching.
1008 <link linkend="multidb">the extended discussion</link>
1009 of virtual databases and multi-database searching below.
1014 <title><literal>query_rewrite</literal>
1015 (mp::filter::QueryRewrite)</title>
1017 Rewrites Z39.50 <literal>Type-1</literal>
1018 and <literal>Type-101</literal> (``<literal>RPN</literal>'')
1020 three-step process: the query is transliterated from Z39.50
1021 packet structures into an XML representation; that XML
1022 representation is transformed by an XSLT stylesheet; and the
1023 resulting XML is transliterated back into the Z39.50 packet
1030 <title><literal>record_transform</literal>
1031 (mp::filter::RecordTransform)</title>
1033 This filter acts only on Z3950 present requests, and let all
1034 other types of packages and requests pass untouched. It's use is
1035 twofold: blocking Z3950 present requests, which the backend
1036 server does not understand and can not honor, and transforming
1037 the present syntax and elementset name according to the rules
1038 specified, to fetch only existing record formats, and transform
1039 them on the fly to requested record syntaxes.
1044 <title><literal>session_shared</literal>
1045 (mp::filter::SessionShared)</title>
1047 This filter implements global sharing of
1048 result sets (i.e. between threads and therefore between
1049 clients), yielding performance improvements by clever resource
1055 <title><literal>sru_z3950</literal>
1056 (mp::filter::SRUtoZ3950)</title>
1058 This filter transforms valid
1059 SRU GET/POST/SOAP searchRetrieve requests to Z3950 init, search,
1060 and present requests, and wraps the
1061 received hit counts and XML records into suitable SRU response
1063 The <literal>sru_z3950</literal> filter processes also SRU
1064 GET/POST/SOAP explain requests, returning
1065 either the absolute minimum required by the standard, or a full
1066 pre-defined ZeeReX explain record.
1068 <ulink url="&url.zeerex.explain;">ZeeReX Explain</ulink>
1069 standard pages and the
1070 <ulink url="&url.sru.explain;">SRU Explain</ulink> pages
1071 for more information on the correct explain syntax.
1072 SRU scan requests are not supported yet.
1077 <title><literal>template</literal>
1078 (mp::filter::Template)</title>
1080 Does nothing at all, merely passing the packet on. (Maybe it
1081 should be called <literal>nop</literal> or
1082 <literal>passthrough</literal>?) This exists not to be used, but
1083 to be copied - to become the skeleton of new filters as they are
1084 written. As with <literal>backend_test</literal>, this is not
1085 intended for civilians.
1090 <title><literal>virt_db</literal>
1091 (mp::filter::VirtualDB)</title>
1093 Performs virtual database selection: based on the name of the
1094 database in the search request, a server is selected, and its
1095 address added to the request in a <literal>VAL_PROXY</literal>
1096 otherInfo packet. It will subsequently be used by a
1097 <literal>z3950_client</literal> filter.
1099 <link linkend="multidb">the extended discussion</link>
1100 of virtual databases and multi-database searching below.
1105 <title><literal>z3950_client</literal>
1106 (mp::filter::Z3950Client)</title>
1108 A partial sink which swallows only Z39.50 packages.
1109 It performs Z39.50 searching and retrieval by proxying the
1110 packages that are passed to it. Init requests are sent to the
1111 address specified in the <literal>VAL_PROXY</literal> otherInfo
1112 attached to the request: this may have been specified by client,
1113 or generated by a <literal>virt_db</literal> filter earlier in
1114 the route. Subsequent requests are sent to the same address,
1115 which is remembered at Init time in a Session object.
1116 HTTP_Request packages and all other forthcoming package types
1117 are passed untouched.
1123 <title><literal>zeerex_explain</literal>
1124 (mp::filter::ZeerexExplain)</title>
1126 This filter acts as a sink for
1127 Z39.50 explain requests, returning a static ZeeReX
1128 Explain XML record from the config section. All other packages
1131 <ulink url="&url.zeerex.explain;">ZeeReX Explain</ulink>
1133 for more information on the correct explain syntax.
1137 This filter is not yet completed.
1146 <section id="future.directions">
1147 <title>Future directions</title>
1149 Some other filters that do not yet exist, but which would be
1150 useful, are briefly described. These may be added in future
1151 releases (or may be created by third parties, as loadable
1157 <term><literal>frontend_cli</literal> (source)</term>
1160 Command-line interface for generating requests.
1165 <term><literal>sru_client</literal> (sink)</term>
1168 SRU/GET and SRU/SOAP searching and retrieval.
1173 <term><literal>opensearch_client</literal> (sink)</term>
1176 A9 OpenSearch searching and retrieval.
1186 <chapter id="configuration">
1187 <title>Configuration: the Metaproxy configuration file format</title>
1191 <title>Introductory notes</title>
1193 If Metaproxy is an interpreter providing operations on packages, then
1194 its configuration file can be thought of as a program for that
1195 interpreter. Configuration is by means of a single XML file, the name
1196 of which is supplied as the sole command-line argument to the
1197 <command>metaproxy</command> program. (See
1198 <xref linkend="reference"/> below for more information on invoking
1203 <section id="overview.xml.structure">
1204 <title>Overview of the config file XML structure</title>
1206 All elements and attributes are in the namespace
1207 <ulink url="http://indexdata.com/metaproxy"/>.
1208 This is most easily achieved by setting the default namespace on
1209 the top-level element, as here:
1212 <metaproxy xmlns="http://indexdata.com/metaproxy" version="1.0">
1215 The top-level element is <metaproxy>. This contains a
1216 <start> element, a <filters> element and a
1217 <routes> element, in that order. <filters> is
1218 optional; the other two are mandatory. All three are
1222 The <start> element is empty, but carries a
1223 <literal>route</literal> attribute, whose value is the name of
1224 route at which to start running - analogous to the name of the
1225 start production in a formal grammar.
1228 If present, <filters> contains zero or more <filter>
1229 elements. Each filter carries a <literal>type</literal> attribute
1230 which specifies what kind of filter is being defined
1231 (<literal>frontend_net</literal>, <literal>log</literal>, etc.)
1232 and contain various elements that provide suitable configuration
1233 for a filter of its type. The filter-specific elements are
1235 <xref linkend="reference"/>.
1236 Filters defined in this part of the file must carry an
1237 <literal>id</literal> attribute so that they can be referenced
1241 <routes> contains one or more <route> elements, each
1242 of which must carry an <literal>id</literal> element. One of the
1243 routes must have the ID value that was specified as the start
1244 route in the <start> element's <literal>route</literal>
1245 attribute. Each route contains zero or more <filter>
1246 elements. These are of two types. They may be empty, but carry a
1247 <literal>refid</literal> attribute whose value is the same as the
1248 <literal>id</literal> of a filter previously defined in the
1249 <filters> section. Alternatively, a route within a filter
1250 may omit the <literal>refid</literal> attribute, but contain
1251 configuration elements similar to those used for filters defined
1252 in the <filters> section. (In other words, each filter in a
1253 route may be included either by reference or by physical
1259 <section id="example.configuration">
1260 <title>An example configuration</title>
1262 The following is a small, but complete, Metaproxy configuration
1263 file (included in the distribution as
1264 <literal>metaproxy/etc/config1.xml</literal>).
1265 This file defines a very simple configuration that simply proxies
1266 to whatever back-end server the client requests, but logs each
1267 request and response. This can be useful for debugging complex
1268 client-server dialogues.
1270 <screen><![CDATA[<?xml version="1.0"?>
1271 <metaproxy xmlns="http://indexdata.com/metaproxy" version="1.0">
1272 <start route="start"/>
1274 <filter id="frontend" type="frontend_net">
1277 <filter id="backend" type="z3950_client">
1282 <filter refid="frontend"/>
1283 <filter type="log"/>
1284 <filter refid="backend"/>
1285 <filter type="bounce"/>
1291 It works by defining a single route, called
1292 <literal>start</literal>, which consists of a sequence of four
1293 filters. The first and last of these are included by reference:
1294 their <literal><filter></literal> elements have
1295 <literal>refid</literal> attributes that refer to filters defined
1296 within the prior <literal><filters></literal> section. The
1297 middle filter is included inline in the route.
1300 The four filters in the route are as follows: first, a
1301 <literal>frontend_net</literal> filter accepts Z39.50 requests
1302 from any host on port 9000; then these requests are passed through
1303 a <literal>log</literal> filter that emits a message for each
1304 request; they are then fed into a <literal>z3950_client</literal>
1305 filter, which forwards all Z39.50 requests to the client-specified
1306 back-end Z39.509 server. Those Z39.50 packages are returned by the
1307 <literal>z3950_client</literal> filter, with the response data
1308 filled by the external Z39.50 server targeted.
1309 All non-Z39.50 packages are passed through to the
1310 <literal>bounce</literal> filter, which definitely bounces
1311 everything, including fish, bananas, cold pyjamas,
1312 mutton, beef and trout packages.
1313 When the response arrives, it is handed
1314 back to the <literal>log</literal> filter, which emits another
1315 message; and then to the <literal>frontend_net</literal> filter,
1316 which returns the response to the client.
1320 <section id="config-file-modularity">
1321 <title>Config file modularity</title>
1323 Metaproxy XML configuration snippets can be reused by other
1324 filters using the <literal>XInclude</literal> standard, as seen in
1325 the <literal>/etc/config-sru-to-z3950.xml</literal> example SRU
1328 <filter id="sru" type="sru_z3950">
1329 <database name="Default">
1330 <xi:include xmlns:xi="http://www.w3.org/2001/XInclude"
1331 href="explain.xml"/>
1338 <section id="config-file-syntax-check">
1339 <title>Config file syntax checking</title>
1341 The distribution contains RelaxNG Compact and XML syntax checking
1342 files, as well as XML Schema files. These are found in the
1345 xml/schema/metaproxy.rnc
1346 xml/schema/metaproxy.rng
1347 xml/schema/metaproxy.xsd
1349 and can be used to verify or debug the XML structure of
1350 configuration files. For example, using the utility
1351 <filename>xmllint</filename>, syntax checking is done like this:
1353 xmllint --noout --schema xml/schema/metaproxy.xsd etc/config-local.xml
1354 xmllint --noout --relaxng xml/schema/metaproxy.rng etc/config-local.xml
1356 (A recent version of <literal>libxml2</literal> is required, as
1357 support for XML Schemas is a relatively recent addition.)
1360 You can of course use any other RelaxNG or XML Schema compliant tool
1368 <chapter id="multidb">
1369 <title>Virtual databases and multi-database searching</title>
1373 <title>Introductory notes</title>
1375 Two of Metaproxy's filters are concerned with multiple-database
1376 operations. Of these, <literal>virt_db</literal> can work alone
1377 to control the routing of searches to one of a number of servers,
1378 while <literal>multi</literal> can work together with
1379 <literal>virt_db</literal> to perform multi-database searching, merging
1380 the results into a unified result-set - ``metasearch in a box''.
1383 The interaction between
1384 these two filters is necessarily complex: it reflects the real,
1385 irreducible complexity of multi-database searching in a protocol such
1386 as Z39.50 that separates initialization from searching, and in
1387 which the database to be searched is not known at initialization
1391 It's possible to use these filters without understanding the
1392 details of their functioning and the interaction between them; the
1393 next two sections of this chapter are ``HOW-TO'' guides for doing
1394 just that. However, debugging complex configurations will require
1395 a deeper understanding, which the last two sections of this
1396 chapters attempt to provide.
1401 <section id="multidb.virt_db">
1402 <title>Virtual databases with the <literal>virt_db</literal> filter</title>
1404 Working alone, the purpose of the
1405 <literal>virt_db</literal>
1406 filter is to route search requests to one of a selection of
1407 back-end databases. In this way, a single Z39.50 endpoint
1408 (running Metaproxy) can provide access to several different
1409 underlying services, including those that would otherwise be
1410 inaccessible due to firewalls. In many useful configurations, the
1411 back-end databases are local to the Metaproxy installation, but
1412 the software does not enforce this, and any valid Z39.50 servers
1413 may be used as back-ends.
1416 For example, a <literal>virt_db</literal>
1417 filter could be set up so that searches in the virtual database
1418 ``lc'' are forwarded to the Library of Congress bibliographic
1419 catalogue server, and searches in the virtual database ``marc''
1420 are forwarded to the toy database of MARC records that Index Data
1421 hosts for testing purposes. A <literal>virt_db</literal>
1422 configuration to make this switch would look like this:
1424 <screen><![CDATA[<filter type="virt_db">
1426 <database>lc</database>
1427 <target>z3950.loc.gov:7090/voyager</target>
1430 <database>marc</database>
1431 <target>indexdata.com/marc</target>
1433 </filter>]]></screen>
1435 As well as being useful in it own right, this filter also provides
1436 the foundation for multi-database searching.
1441 <section id="multidb.multi">
1442 <title>Multi-database search with the <literal>multi</literal> filter</title>
1444 To arrange for Metaproxy to broadcast searches to multiple back-end
1445 servers, the configuration needs to include two components: a
1446 <literal>virt_db</literal>
1447 filter that specifies multiple
1448 <literal><target></literal>
1449 elements, and a subsequent
1450 <literal>multi</literal>
1451 filter. Here, for example, is a complete configuration that
1452 broadcasts searches to both the Library of Congress catalogue and
1453 Index Data's tiny testing database of MARC records:
1455 <screen><![CDATA[<?xml version="1.0"?>
1456 <metaproxy xmlns="http://indexdata.com/metaproxy" version="1.0">
1457 <start route="start"/>
1460 <filter type="frontend_net">
1461 <threads>10</threads>
1464 <filter type="virt_db">
1466 <database>lc</database>
1467 <target>z3950.loc.gov:7090/voyager</target>
1470 <database>marc</database>
1471 <target>indexdata.com/marc</target>
1474 <database>all</database>
1475 <target>z3950.loc.gov:7090/voyager</target>
1476 <target>indexdata.com/marc</target>
1479 <filter type="multi"/>
1480 <filter type="z3950_client">
1481 <timeout>30</timeout>
1483 <filter type="bounce"/>
1486 </metaproxy>]]></screen>
1489 <literal>virt_db</literal>
1490 filter that specifies multiple
1491 <literal><target></literal>
1492 elements but without a subsequent
1493 <literal>multi</literal>
1494 filter yields surprising and undesirable results, as will be
1495 described below. Don't do that.)
1498 Metaproxy can be invoked with this configuration as follows:
1500 <screen>../src/metaproxy --config config-simple-multi.xml</screen>
1502 And thereafter, Z39.50 clients can connect to the running server
1503 (on port 9000, as specified in the configuration) and search in
1504 any of the databases
1505 <literal>lc</literal> (the Library of Congress catalogue),
1506 <literal>marc</literal> (Index Data's test database of MARC records)
1508 <literal>all</literal> (both of these). As an example, a session
1509 using the YAZ command-line client <literal>yaz-client</literal> is
1510 here included (edited for brevity and clarity):
1512 <screen><![CDATA[$ yaz-client @:9000
1516 Search was a success.
1517 Number of hits: 10000, setno 1
1521 Search was a success.
1522 Number of hits: 10, setno 3
1526 Search was a success.
1527 Number of hits: 10010, setno 4
1530 [marc]Record type: USmarc
1533 005 00000000000000.0
1534 008 910710c19910701nju 00010 eng
1538 100 10 $a Jack Collins
1539 245 10 $a How to program a computer
1545 [VOYAGER]Record type: USmarc
1547 005 20041229102447.0
1548 008 030910s2004 caua 000 0 eng
1549 035 $a (DLC) 2003112666
1550 906 $a 7 $b cbc $c orignew $d 4 $e epcn $f 20 $g y-gencatlg
1551 925 0 $a acquire $b 1 shelf copy $x policy default
1552 955 $a pc10 2003-09-10 $a pv12 2004-06-23 to SSCD; $h sj05 2004-11-30 $e sj05 2004-11-30 to Shelf.
1555 040 $a DLC $c DLC $d DLC
1556 050 00 $a MLCM 2004/03312 (G)
1557 245 10 $a 007, everything or nothing : $b Prima's official strategy guide / $c created by Kaizen Media Group.
1558 246 3 $a Double-O-seven, everything or nothing
1559 246 30 $a Prima's official strategy guide
1560 260 $a Roseville, CA : $b Prima Games, $c c2004.
1561 300 $a 161 p. : $b col. ill. ; $c 28 cm.
1562 500 $a "Platforms: Nintendo GameCube, Macintosh, PC, PlayStation 2 computer entertainment system, Xbox"--P. [4] of cover.
1563 650 0 $a Video games.
1564 710 2 $a Kaizen Media Group.
1565 856 42 $3 Publisher description $u http://www.loc.gov/catdir/description/random052/2003112666.html
1570 As can be seen, the first record in the result set is from the
1571 Index Data test database, and the second from the Library of
1572 Congress database. The result-set continues alternating records
1573 round-robin style until the point where one of the databases'
1574 records are exhausted.
1577 This example uses only two back-end databases; more may be used.
1578 There is no limitation imposed on the number of databases that may
1579 be metasearched in this way: issues of resource usage and
1580 administrative complexity dictate the practical limits.
1583 What happens when one of the databases doesn't respond? By default,
1584 the entire multi-database search fails, and the appropriate
1585 diagnostic is returned to the client. This is usually appropriate
1586 during development, when technicians need maximum information, but
1587 can be inconvenient in deployment, when users typically don't want
1588 to be bothered with problems of this kind and prefer just to get
1589 the records from the databases that are available. To obtain this
1590 latter behavior add an empty
1591 <literal><hideunavailable></literal>
1593 <literal>multi</literal> filter:
1595 <screen><![CDATA[ <filter type="multi">
1597 </filter>]]></screen>
1599 Under this regime, an error is reported to the client only if
1600 <emphasis>all</emphasis> the databases in a multi-database search
1606 <section id="multidb.what">
1607 <title>What's going on?</title>
1609 <title>Lark's vomit</title>
1611 This section goes into a level of technical detail that is
1612 probably not necessary in order to configure and use Metaproxy.
1613 It is provided only for those who like to know how things work.
1614 You should feel free to skip on to the next section if this one
1615 doesn't seem like fun.
1619 Hold on tight - this may get a little hairy.
1622 In the general course of things, a Z39.50 Init request may carry
1623 with it an otherInfo packet of type <literal>VAL_PROXY</literal>,
1624 whose value indicates the address of a Z39.50 server to which the
1625 ultimate connection is to be made. (This otherInfo packet is
1626 supported by YAZ-based Z39.50 clients and servers, but has not yet
1627 been ratified by the Maintenance Agency and so is not widely used
1628 in non-Index Data software. We're working on it.)
1629 The <literal>VAL_PROXY</literal> packet functions
1630 analogously to the absoluteURI-style Request-URI used with the GET
1631 method when a web browser asks a proxy to forward its request: see
1633 <ulink url="http://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html#sec5.1.2"
1634 >Request-URI</ulink>
1636 <ulink url="http://www.w3.org/Protocols/rfc2616/rfc2616.html"
1637 >the HTTP 1.1 specification</ulink>.
1640 Within Metaproxy, Search requests that are part of the same
1641 session as an Init request that carries a
1642 <literal>VAL_PROXY</literal> otherInfo are also annotated with the
1643 same information. The role of the <literal>virt_db</literal>
1644 filter is to rewrite this otherInfo packet dependent on the
1645 virtual database that the client wants to search.
1648 When Metaproxy receives a Z39.50 Init request from a client, it
1649 doesn't immediately forward that request to the back-end server.
1650 Why not? Because it doesn't know <emphasis>which</emphasis>
1651 back-end server to forward it to until the client sends a Search
1652 request that specifies the database that it wants to search in.
1653 Instead, it just treasures the Init request up in its heart; and,
1654 later, the first time the client does a search on one of the
1655 specified virtual databases, a connection is forged to the
1656 appropriate server and the Init request is forwarded to it. If,
1657 later in the session, the same client searches in a different
1658 virtual database, then a connection is forged to the server that
1659 hosts it, and the same cached Init request is forwarded there,
1663 All of this clever Init-delaying is done by the
1664 <literal>frontend_net</literal> filter. The
1665 <literal>virt_db</literal> filter knows nothing about it; in
1666 fact, because the Init request that is received from the client
1667 doesn't get forwarded until a Search request is received, the
1668 <literal>virt_db</literal> filter (and the
1669 <literal>z3950_client</literal> filter behind it) doesn't even get
1670 invoked at Init time. The <emphasis>only</emphasis> thing that a
1671 <literal>virt_db</literal> filter ever does is rewrite the
1672 <literal>VAL_PROXY</literal> otherInfo in the requests that pass
1676 It is possible for a <literal>virt_db</literal> filter to contain
1678 <literal><target></literal>
1679 elements. What does this mean? Only that the filter will add
1680 multiple <literal>VAL_PROXY</literal> otherInfo packets to the
1681 Search requests that pass through it. That's because the virtual
1682 DB filter is dumb, and does exactly what it's told - no more, no
1684 If a Search request with multiple <literal>VAL_PROXY</literal>
1685 otherInfo packets reaches a <literal>z3950_client</literal>
1686 filter, this is an error. That filter doesn't know how to deal
1687 with multiple targets, so it will either just pick one and search
1688 in it, or (better) fail with an error message.
1691 The <literal>multi</literal> filter comes to the rescue! This is
1692 the only filter that knows how to deal with multiple
1693 <literal>VAL_PROXY</literal> otherInfo packets, and it does so by
1694 making multiple copies of the entire Search request: one for each
1695 <literal>VAL_PROXY</literal>. Each of these new copies is then
1696 passed down through the remaining filters in the route. (The
1697 copies are handled in parallel though the
1698 spawning of new threads.) Since the copies each have only one
1699 <literal>VAL_PROXY</literal> otherInfo, they can be handled by the
1700 <literal>z3950_client</literal> filter, which happily deals with
1701 each one individually. When the results of the individual
1702 searches come back up to the <literal>multi</literal> filter, it
1703 merges them into a single Search response, which is what
1704 eventually makes it back to the client.
1709 <section id="multidb.picture">
1710 <title>A picture is worth a thousand words (but only five hundred on 64-bit architectures)</title>
1714 <imagedata fileref="multi.pdf" format="PDF" scale="50"/>
1717 <imagedata fileref="multi.png" format="PNG"/>
1720 <!-- Fall back if none of the images can be used -->
1722 [Here there should be a diagram showing the progress of
1723 packages through the filters during a simple virtual-database
1724 search and a multi-database search, but is seems that your
1725 tool chain has not been able to include the diagram in this
1729 <!-- ### This used to work with an older version of DocBook
1731 <para>Caption: progress of packages through filters.</para>
1734 </inlinemediaobject>
1740 <chapter id="sru-server">
1741 <title>Combined SRU webservice and Z39.50 server configuration</title>
1743 Metaproxy can act as
1744 <ulink url="&url.sru;">SRU</ulink> and
1745 <ulink url="&url.srw;">SRW</ulink>
1746 web service server, which translates web service requests to
1747 <ulink url="&url.z39.50;">ANSI/NISO Z39.50</ulink> packages and
1748 sends them off to common available targets.
1751 A typical setup for this operation needs a filter route including the
1755 <table id="sru-server-table-config" frame="top">
1756 <title>SRU/Z39.50 Server Filter Route Configuration</title>
1760 <entry>Filter</entry>
1761 <entry>Importance</entry>
1762 <entry>Purpose</entry>
1768 <entry><literal>frontend_net</literal></entry>
1769 <entry>required</entry>
1770 <entry>Accepting HTTP connections and passing them to following
1771 filters. Since this filter also accepts Z39.50 connections, the
1772 server works as SRU and Z39.50 server on the same port.</entry>
1775 <entry><literal>sru_z3950</literal></entry>
1776 <entry>required</entry>
1777 <entry>Accepting SRU GET/POST/SOAP explain and
1778 searchRetrieve requests for the the configured databases.
1779 Explain requests are directly served from the static XML configuration.
1780 SearchRetrieve requests are
1781 transformed to Z39.50 search and present packages.
1782 All other HTTP and Z39.50 packages are passed unaltered.</entry>
1785 <entry><literal>http_file</literal></entry>
1786 <entry>optional</entry>
1787 <entry>Serving HTTP requests from the filesystem. This is only
1788 needed if the server should serve XSLT stylesheets, static HTML
1789 files or Java Script for thin browser based clients.
1790 Z39.50 packages are passed unaltered.</entry>
1793 <entry><literal>cql_rpn</literal></entry>
1794 <entry>required</entry>
1795 <entry>Usually, Z39.50 servers do not talk CQL, hence the
1796 translation of the CQL query language to RPN is mandatory in
1797 most cases. Affects only Z39.50 search packages.</entry>
1800 <entry><literal>record_transform</literal></entry>
1801 <entry>optional</entry>
1802 <entry>Some Z39.50 backend targets can not present XML record
1803 syntaxes in common wanted element sets. using this filter, one
1804 can transform binary MARC records to MARCXML records, and
1805 further transform those to any needed XML schema/format by XSLT
1806 transformations. Changes only Z39.50 present packages.</entry>
1809 <entry><literal>session_shared</literal></entry>
1810 <entry>optional</entry>
1811 <entry>The stateless nature of web services requires frequent
1812 re-searching of the same targets for display of paged result set
1813 records. This might be an unacceptable burden for the accessed
1814 backend Z39.50 targets, and this mosule can be added for
1815 efficient backend target resource pooling.</entry>
1818 <entry><literal>z3950_client</literal></entry>
1819 <entry>required</entry>
1820 <entry>Finally, a Z39.50 package sink is needed in the filter
1821 chain to provide the response packages. The Z39.50 client module
1822 is used to access external targets over the network, but any
1823 coming local Z39.50 package sink could be used instead of.</entry>
1826 <entry><literal>bounce</literal></entry>
1827 <entry>required</entry>
1828 <entry>Any Metaproxy package arriving here did not do so by
1829 purpose, and is bounced back with connection closure. this
1830 prevents inifinite package hanging inside the SRU server.</entry>
1836 A typical minimal example <ulink url="&url.sru;">SRU</ulink> and
1837 <ulink url="&url.srw;">SRW</ulink> server configuration file is found
1838 in the tarball distribution at
1839 <literal>etc/config-sru-to-z3950.xml</literal>.
1842 Off course, any other metaproxy modules can be integrated into a
1843 SRU server solution, including, but not limited to, load balancing,
1844 multiple target querying
1845 (see <xref linkend="multidb"/>), and complex RPN query rewrites.
1852 <chapter id="extensions">
1853 <title>Writing extensions for Metaproxy</title>
1854 <para>### To be written</para>
1860 <chapter id="classes">
1861 <title>Classes in the Metaproxy source code</title>
1865 <title>Introductory notes</title>
1867 <emphasis>Stop! Do not read this!</emphasis>
1868 You won't enjoy it at all. You should just skip ahead to
1869 <xref linkend="reference"/>,
1871 <!-- The remainder of this paragraph is lifted verbatim from
1872 Douglas Adams' _Hitch Hiker's Guide to the Galaxy_, chapter 8 -->
1873 you things you really need to know, like the fact that the
1874 fabulously beautiful planet Bethselamin is now so worried about
1875 the cumulative erosion by ten billion visiting tourists a year
1876 that any net imbalance between the amount you eat and the amount
1877 you excrete whilst on the planet is surgically removed from your
1878 bodyweight when you leave: so every time you go to the lavatory it
1879 is vitally important to get a receipt.
1882 This chapter contains documentation of the Metaproxy source code, and is
1883 of interest only to maintainers and developers. If you need to
1884 change Metaproxy's behavior or write a new filter, then you will most
1885 likely find this chapter helpful. Otherwise it's a waste of your
1886 good time. Seriously: go and watch a film or something.
1887 <citetitle>This is Spinal Tap</citetitle> is particularly good.
1890 Still here? OK, let's continue.
1893 In general, classes seem to be named big-endianly, so that
1894 <literal>FactoryFilter</literal> is not a filter that filters
1895 factories, but a factory that produces filters; and
1896 <literal>FactoryStatic</literal> is a factory for the statically
1897 registered filters (as opposed to those that are dynamically
1902 <section id="individual.classes">
1903 <title>Individual classes</title>
1905 The classes making up the Metaproxy application are here listed by
1906 class-name, with the names of the source files that define them in
1911 <title><literal>mp::FactoryFilter</literal>
1912 (<filename>factory_filter.cpp</filename>)</title>
1914 A factory class that exists primarily to provide the
1915 <literal>create()</literal> method, which takes the name of a
1916 filter class as its argument and returns a new filter of that
1917 type. To enable this, the factory must first be populated by
1918 calling <literal>add_creator()</literal> for static filters (this
1919 is done by the <literal>FactoryStatic</literal> class, see below)
1920 and <literal>add_creator_dyn()</literal> for filters loaded
1926 <title><literal>mp::FactoryStatic</literal>
1927 (<filename>factory_static.cpp</filename>)</title>
1929 A subclass of <literal>FactoryFilter</literal> which is
1930 responsible for registering all the statically defined filter
1931 types. It does this by knowing about all those filters'
1932 structures, which are listed in its constructor. Merely
1933 instantiating this class registers all the static classes. It is
1934 for the benefit of this class that <literal>struct
1935 metaproxy_1_filter_struct</literal> exists, and that all the filter
1936 classes provide a static object of that type.
1941 <title><literal>mp::filter::Base</literal>
1942 (<filename>filter.cpp</filename>)</title>
1944 The virtual base class of all filters. The filter API is, on the
1945 surface at least, extremely simple: two methods.
1946 <literal>configure()</literal> is passed an XML DOM tree representing
1947 that part of the configuration file that pertains to this filter
1948 instance, and is expected to walk that tree extracting relevant
1949 information. And <literal>process()</literal> processes a
1950 package (see below). That surface simplicity is a bit
1951 misleading, as <literal>process()</literal> needs to know a lot
1952 about the <literal>Package</literal> class in order to do
1958 <title><literal>mp::filter::AuthSimple</literal>,
1959 <literal>Backend_test</literal>, etc.
1960 (<filename>filter_auth_simple.cpp</filename>,
1961 <filename>filter_backend_test.cpp</filename>, etc.)</title>
1963 Individual filters. Each of these is implemented by a header and
1964 a source file, named <filename>filter_*.hpp</filename> and
1965 <filename>filter_*.cpp</filename> respectively. All the header
1966 files should be pretty much identical, in that they declare the
1967 class, including a private <literal>Rep</literal> class and a
1968 member pointer to it, and the two public methods.
1971 The source file for each filter needs to supply:
1976 A definition of the private <literal>Rep</literal> class.
1981 Some boilerplate constructors and destructors.
1986 A <literal>configure()</literal> method that uses the
1987 appropriate XML fragment.
1992 Most important, the <literal>process()</literal> method that
1993 does all the actual work.
2000 <title><literal>mp::Package</literal>
2001 (<filename>package.cpp</filename>)</title>
2003 Represents a package on its way through the series of filters
2004 that make up a route. This is essentially a Z39.50 or SRU APDU
2005 together with information about where it came from, which is
2006 modified as it passes through the various filters.
2011 <title><literal>mp::Pipe</literal>
2012 (<filename>pipe.cpp</filename>)</title>
2014 This class provides a compatibility layer so that we have an IPC
2015 mechanism that works the same under Unix and Windows. It's not
2016 particularly exciting.
2021 <title><literal>mp::RouterChain</literal>
2022 (<filename>router_chain.cpp</filename>)</title>
2029 <title><literal>mp::RouterFleXML</literal>
2030 (<filename>router_flexml.cpp</filename>)</title>
2037 <title><literal>mp::Session</literal>
2038 (<filename>session.cpp</filename>)</title>
2045 <title><literal>mp::ThreadPoolSocketObserver</literal>
2046 (<filename>thread_pool_observer.cpp</filename>)</title>
2053 <title><literal>mp::util</literal>
2054 (<filename>util.cpp</filename>)</title>
2056 A namespace of various small utility functions and classes,
2057 collected together for convenience. Most importantly, includes
2058 the <literal>mp::util::odr</literal> class, a wrapper for YAZ's
2064 <title><literal>mp::xml</literal>
2065 (<filename>xmlutil.cpp</filename>)</title>
2067 A namespace of various XML utility functions and classes,
2068 collected together for convenience.
2074 <section id="other.source.files">
2075 <title>Other Source Files</title>
2077 In addition to the Metaproxy source files that define the classes
2078 described above, there are a few additional files which are
2079 briefly described here:
2083 <term><literal>metaproxy_prog.cpp</literal></term>
2086 The main function of the <command>metaproxy</command> program.
2091 <term><literal>ex_router_flexml.cpp</literal></term>
2094 Identical to <literal>metaproxy_prog.cpp</literal>: it's not clear why.
2099 <term><literal>test_*.cpp</literal></term>
2102 Unit-tests for various modules.
2108 ### Still to be described:
2109 <literal>ex_filter_frontend_net.cpp</literal>,
2110 <literal>filter_dl.cpp</literal>,
2111 <literal>plainfile.cpp</literal>,
2112 <literal>tstdl.cpp</literal>.
2118 <reference id="reference">
2119 <title>Reference</title>
2122 The material in this chapter is drawn directly from the individual
2123 manual entries. In particular, the Metaproxy invocation section is
2124 available using <command>man metaproxy</command>, and the section
2125 on each individual filter is available using the name of the filter
2126 as the argument to the <command>man</command> command.
2133 <!-- Keep this comment at the end of the file
2138 sgml-minimize-attributes:nil
2139 sgml-always-quote-attributes:t
2142 sgml-parent-document: nil
2143 sgml-local-catalogs: nil
2144 sgml-namecase-general:t