1 <?xml version="1.0" standalone="no"?>
2 <!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.1//EN"
3 "http://www.oasis-open.org/docbook/xml/4.1/docbookx.dtd"
5 <!ENTITY % local SYSTEM "local.ent">
7 <!ENTITY manref SYSTEM "manref.xml">
8 <!ENTITY progref SYSTEM "progref.xml">
9 <!ENTITY gpl2 SYSTEM "gpl-2.0.xml">
10 <!ENTITY % idcommon SYSTEM "common/common.ent">
12 <!-- Next line allows imagedata/@format="PDF" and is taken from
13 http://lists.oasis-open.org/archives/docbook/200303/msg00163.html
15 <!ENTITY % local.notation.class "| PDF">
16 <!-- Next line is necessary for some XML parsers, for reasons I
17 don't understand. I got this from
18 http://lists.oasis-open.org/archives/docbook/200303/msg00180.html
20 <!NOTATION PDF SYSTEM "PDF">
24 <title>Metaproxy - User's Guide and Reference</title>
27 <firstname>Adam</firstname><surname>Dickmeiss</surname>
30 <firstname>Marc</firstname><surname>Cromme</surname>
33 <firstname>Mike</firstname><surname>Taylor</surname>
36 <releaseinfo>&version;</releaseinfo>
38 <year>2005-2009</year>
39 <holder>Index Data ApS</holder>
43 This manual is part of Metaproxy version &version;.
46 Metaproxy is a universal router, proxy and encapsulated
47 metasearcher for information retrieval protocols. It accepts,
48 processes, interprets and redirects requests from IR clients using
49 standard protocols such as the binary
50 <ulink url="&url.z39.50;">ANSI/NISO Z39.50</ulink>
51 and the information search and retrieval
52 web service <ulink url="&url.sru;">SRU</ulink>
53 as well as functioning as a limited
54 <ulink url="&url.http;">HTTP</ulink> server.
57 Metaproxy is configured by an XML file which
58 specifies how the software should function in terms of routes that
59 the request packets can take through the proxy, each step on a
60 route being an instantiation of a filter. Filters come in many
61 types, one for each operation: accepting Z39.50 packets, logging,
62 query transformation, multiplexing, etc. Further filter-types can
63 be added as loadable modules to extend Metaproxy functionality,
67 Metaproxy is covered by the GNU General Public License version 2.
72 <imagedata fileref="common/id.png" format="PNG"/>
75 <imagedata fileref="common/id.eps" format="EPS"/>
82 <chapter id="introduction">
83 <title>Introduction</title>
86 <ulink url="&url.metaproxy;">Metaproxy</ulink>
87 is a stand alone program that acts as a universal router, proxy and
88 encapsulated metasearcher for information retrieval protocols such
89 as <ulink url="&url.z39.50;">Z39.50</ulink> and
90 <ulink url="&url.sru;">SRU</ulink>.
91 To clients, it acts as a server of these protocols: it can be searched,
92 records can be retrieved from it, etc.
93 To servers, it acts as a client: it searches in them,
94 retrieves records from them, etc. it satisfies its clients'
95 requests by transforming them, multiplexing them, forwarding them
96 on to zero or more servers, merging the results, transforming
97 them, and delivering them back to the client. In addition, it
98 acts as a simple <ulink url="&url.http;">HTTP</ulink> server; support
99 for further protocols can be added in a modular fashion, through the
100 creation of new filters.
105 Fish, bananas, cold pyjamas,
106 Mutton, beef and trout!
107 - attributed to Cole Porter.
110 Metaproxy is a more capable alternative to
111 <ulink url="&url.yazproxy;">YAZ Proxy</ulink>,
112 being more powerful, flexible, configurable and extensible. Among
113 its many advantages over the older, more pedestrian work are
114 support for multiplexing (encapsulated metasearching), routing by
115 database name, authentication and authorization and serving local
116 files via HTTP. Equally significant, its modular architecture
117 facilitites the creation of pluggable modules implementing further
121 This manual will describe how to install Metaproxy
122 before giving an overview of its architecture, then discussing the
123 key concept of a filter in some depth and giving an overview of
124 the various filter types, then discussing the configuration file
125 format. After this come several optional chapters which may be
126 freely skipped: a detailed discussion of virtual databases and
127 multi-database searching, some notes on writing extensions
128 (additional filter types) and a high-level description of the
129 source code. Finally comes the reference guide, which contains
130 instructions for invoking the <command>metaproxy</command>
131 program, and detailed information on each type of filter,
136 <chapter id="installation">
137 <title>Installation</title>
139 Metaproxy depends on the following tools/libraries:
141 <varlistentry><term><ulink url="&url.yazplusplus;">YAZ++</ulink></term>
144 This is a C++ library based on <ulink url="&url.yaz;">YAZ</ulink>.
148 <varlistentry><term><ulink url="&url.libxslt;">Libxslt</ulink></term>
150 <para>This is an XSLT processor - based on
151 <ulink url="&url.libxml2;">Libxml2</ulink>. Both Libxml2 and
152 Libxslt must be installed with the development components
153 (header files, etc.) as well as the run-time libraries.
157 <varlistentry><term><ulink url="&url.boost;">Boost</ulink></term>
160 The popular C++ library. Initial versions of Metaproxy
161 was built with 1.32 but this is no longer supported.
162 Metaproxy is known to work with Boost version 1.33 through 1.38.
169 In order to compile Metaproxy a modern C++ compiler is
170 required. Boost, in particular, requires the C++ compiler
171 to facilitate the newest features. Refer to Boost
172 <ulink url="&url.boost.compilers.status;">Compiler Status</ulink>
173 for more information.
176 We have successfully built Metaproxy using the compilers
177 <ulink url="&url.gcc;">GCC</ulink> version 4.0 and
178 <ulink url="&url.vstudio;">Microsoft Visual Studio</ulink> 2003/2005.
181 <section id="installation.unix">
182 <title>Installation on Unix (from Source)</title>
184 Here is a quick step-by-step guide on how to compile all the
185 tools that Metaproxy uses. Only few systems have none of the required
186 tools binary packages. If, for example, Libxml2/libxslt are already
187 installed as development packages use those (and omit compilation).
190 <section id="libxml2.fromsource">
191 <title>Libxml2/libxslt</title>
196 gunzip -c libxml2-version.tar.gz|tar xf -
204 gunzip -c libxslt-version.tar.gz|tar xf -
212 <section id="yaz.fromsource">
213 <title>YAZ/YAZ++</title>
215 gunzip -c yaz-version.tar.gz|tar xf -
223 gunzip -c yazpp-version.tar.gz|tar xf -
232 <title id="boost.fromsource">Boost</title>
234 Metaproxy needs components thread and test from
238 gunzip -c boost-version.tar.gz|tar xf -
240 ./configure --with-libraries=thread,test --with-toolset=gcc
246 However, under the hood bjam is used. You can invoke that with
249 ./bjam --toolset=gcc --with-thread --with-test stage
252 Replace <literal>stage</literal> with <literal>clean</literal> /
253 <literal>install</literal> to perform clean and install respectively.
256 Add <literal>--prefix=DIR</literal> to install Boost in other
257 prefix than <literal>/usr/local</literal>.
260 <section id="metaproxy.fromsource">
261 <title>Metaproxy</title>
263 gunzip -c metaproxy-version.tar.gz|tar xf -
271 You may have to tell configure where Boost is installed by supplying
272 options <literal>--with-boost</literal> and <literal>--with-boost-toolset</literal>.
273 The former sets the PREFIX for Boost (same as --prefix for Boost above).
274 The latter the compiler toolset (eg. gcc34).
277 Pass <literal>--help</literal> to configure to get a list of
283 <section id="installation.debian">
284 <title>Installation on Debian GNU/Linux</title>
286 All dependencies for Metaproxy are available as
287 <ulink url="&url.debian;">Debian</ulink>
288 packages for the sarge (stable in 2005) and etch (testing in 2005)
292 The procedures for Debian based systems, such as
293 <ulink url="&url.ubuntu;">Ubuntu</ulink> is probably similar
296 There is currently no official Debian package for YAZ++.
297 And the official Debian package for YAZ is probably too old.
298 But Index Data bulds "new" versions of those for Debian (i386 only).
301 Update the <filename>/etc/apt/sources.list</filename>
302 to include the Index Data repository.
303 See YAZ' <ulink url="&url.yaz.download.debian;">Download Debian</ulink>
304 for more information.
307 apt-get install libxslt1-dev
308 apt-get install libyazpp2-dev
309 apt-get install libboost-dev
310 apt-get install libboost-thread-dev
311 apt-get install libboost-test-dev
314 With these packages installed, the usual configure + make
315 procedure can be used for Metaproxy as outlined in
316 <xref linkend="installation.unix"/>.
320 <section id="installation.rpm">
321 <title>Installation on RPM based Linux Systems</title>
323 All external dependencies for Metaproxy are available as
324 RPM packages, either from your distribution site, or from the
325 <ulink url="http://fr.rpmfind.net/">RPMfind</ulink> site.
328 For example, an installation of the requires Boost C++ development
329 libraries on RedHat Fedora C4 and C5 can be done like this:
331 wget ftp://fr.rpmfind.net/wlinux/fedora/core/updates/testing/4/SRPMS/boost-1.33.0-3.fc4.src.rpm
332 sudo rpmbuild --buildroot src/ --rebuild -p fc4/boost-1.33.0-3.fc4.src.rpm
333 sudo rpm -U /usr/src/redhat/RPMS/i386/boost-*rpm
337 The <ulink url="&url.yaz;">YAZ</ulink> library is needed to
338 compile &metaproxy;, see there
339 for more information on available RPM packages.
342 There is currently no official RPM package for YAZ++.
343 See the <ulink url="&url.yazplusplus;">YAZ++</ulink> pages
344 for more information on a Unix tarball install.
347 With these packages installed, the usual configure + make
348 procedure can be used for Metaproxy as outlined in
349 <xref linkend="installation.unix"/>.
353 <section id="installation.windows">
354 <title>Installation on Windows</title>
356 Metaproxy can be compiled with Microsoft
357 <ulink url="&url.vstudio;">Visual Studio</ulink>.
358 Versions 2003 (C 7.1), 2005 (C 8.0) and 2008 (C 9.0) is known to work.
360 <section id="installation.windows.boost">
363 For Windows, it's easiest to get the precompiled Boost
364 package from <ulink url="&url.boost.windows.download;">here</ulink>.
365 Several versions of the Boost libraries may be selected when
366 installing Boost for windows. Please choose at least the
367 <emphasis>multithreaded</emphasis> (non-DLL) version because
368 the Metaproxy makefile uses that.
371 For more information about installing Boost refer to the
372 <ulink url="&url.boost.getting.started;">getting started</ulink>
377 <section id="installation.windows.libxslt">
378 <title>Libxslt</title>
380 <ulink url="&url.libxslt;">Libxslt</ulink> can be downloaded
382 <ulink url="&url.libxml2.download.win32;">here</ulink>.
385 Libxslt has other dependencies, but these can all be downloaded
386 from the same site. Get the following package:
387 iconv, zlib, libxml2, libxslt.
391 <section id="installation.windows.yaz">
394 <ulink url="&url.yaz;">YAZ</ulink> can be downloaded
396 <ulink url="&url.yaz.download.win32;">here</ulink>.
400 <section id="installation.windows.yazplusplus">
403 Get <ulink url="&url.yazplusplus;">YAZ++</ulink> as well.
404 Version 1.1.0 or later is required.
407 YAZ++ includes NMAKE makefiles, similar to those found in the
412 <section id="installation.windows.metaproxy">
413 <title>Metaproxy</title>
415 Metaproxy is shipped with NMAKE makefiles as well - similar
416 to those found in the YAZ++/YAZ packages. Adjust this Makefile
417 to point to the proper locations of Boost, Libxslt, Libxml2,
418 zlib, iconv, yaz and yazpp.
422 <varlistentry><term><literal>DEBUG</literal></term>
424 If set to 1, the software is
425 compiled with debugging libraries (code generation is
426 multi-threaded debug DLL).
427 If set to 0, the software is compiled with release libraries
428 (code generation is multi-threaded DLL).
433 <term><literal>BOOST</literal></term>
436 Boost install location
442 <term><literal>BOOST_VERSION</literal></term>
445 Boost version (replace . with _).
451 <term><literal>BOOST_TOOLSET</literal></term>
460 <term><literal>LIBXSLT_DIR</literal>,
461 <literal>LIBXML2_DIR</literal> ..</term>
464 Specify the locations of Libxslt, libiconv, libxml2 and
473 After successful compilation you'll find
474 <literal>metaproxy.exe</literal> in the
475 <literal>bin</literal> directory.
483 <chapter id="yazproxy-comparison">
484 <title>YAZ Proxy Comparison</title>
486 The table below lists facilities either supported by either
487 <ulink url="&url.yazproxy;">YAZ Proxy</ulink> or Metaproxy.
489 <table id="yazproxy-comparison-table">
490 <title>Metaproxy / YAZ Proxy comparison</title>
494 <entry>Facility</entry>
495 <entry>Metaproxy</entry>
496 <entry>YAZ Proxy</entry>
501 <entry>Z39.50 server</entry>
502 <entry>Using filter <literal>frontend_net</literal></entry>
503 <entry>Supported</entry>
506 <entry>SRU server</entry>
507 <entry>Supported with filter <literal>sru_z3950</literal></entry>
508 <entry>Supported</entry>
511 <entry>Z39.50 client</entry>
512 <entry>Supported with filter <literal>z3950_client</literal></entry>
513 <entry>Supported</entry>
516 <entry>SRU client</entry>
517 <entry>Unsupported</entry>
518 <entry>Unsupported</entry>
521 <entry>Connection reuse</entry>
522 <entry>Supported with filter <literal>session_shared</literal></entry>
523 <entry>Supported</entry>
526 <entry>Connection share</entry>
527 <entry>Supported with filter <literal>session_shared</literal></entry>
528 <entry>Unsupported</entry>
531 <entry>Result set reuse</entry>
532 <entry>Supported with filter <literal>session_shared</literal></entry>
533 <entry>Within one Z39.50 session / HTTP keep-alive</entry>
536 <entry>Record cache</entry>
537 <entry>Unsupported</entry>
538 <entry>Supported for last result set within one Z39.50/HTTP-keep alive session</entry>
541 <entry>Z39.50 Virtual database, i.e. select any Z39.50 target for database</entry>
542 <entry>Supported with filter <literal>virt_db</literal></entry>
543 <entry>Unsupported</entry>
546 <entry>SRU Virtual database, i.e. select any Z39.50 target for path</entry>
547 <entry>Supported with filter <literal>virt_db</literal>,
548 <literal>sru_z3950</literal></entry>
549 <entry>Supported</entry>
552 <entry>Multi target search</entry>
553 <entry>Supported with filter <literal>multi</literal> (round-robin)</entry>
554 <entry>Unsupported</entry>
557 <entry>Retrieval and search limits</entry>
558 <entry>Unsupported</entry>
559 <entry>Supported</entry>
562 <entry>Bandwidth limits</entry>
563 <entry>Unsupported</entry>
564 <entry>Supported</entry>
567 <entry>Connect limits</entry>
568 <entry>Unsupported</entry>
569 <entry>Supported</entry>
572 <entry>Retrieval sanity check and conversions</entry>
573 <entry>Supported using filter <literal>record_transform</literal></entry>
574 <entry>Supported</entry>
577 <entry>Query check</entry>
579 Supported in a limited way using <literal>query_rewrite</literal>
581 <entry>Supported</entry>
584 <entry>Query rewrite</entry>
585 <entry>Supported with <literal>query_rewrite</literal></entry>
586 <entry>Unsupported</entry>
589 <entry>Session invalidate for -1 hits</entry>
590 <entry>Unsupported</entry>
591 <entry>Supported</entry>
594 <entry>Architecture</entry>
595 <entry>Multi-threaded + select for networked modules such as
596 <literal>frontend_net</literal>)</entry>
597 <entry>Single-threaded using select</entry>
601 <entry>Extensability</entry>
602 <entry>Most functionality implemented as loadable modules</entry>
603 <entry>Unsupported and experimental</entry>
607 <entry><ulink url="&url.usemarcon;">USEMARCON</ulink></entry>
608 <entry>Unsupported</entry>
609 <entry>Supported</entry>
613 <entry>Portability</entry>
615 Requires YAZ, YAZ++ and modern C++ compiler supporting
616 <ulink url="&url.boost;">Boost</ulink>.
619 Requires YAZ and YAZ++.
620 STL is not required so pretty much any C++ compiler out there should work.
629 <chapter id="architecture">
630 <title>The Metaproxy Architecture</title>
632 The Metaproxy architecture is based on three concepts:
633 the <emphasis>package</emphasis>,
634 the <emphasis>route</emphasis>
635 and the <emphasis>filter</emphasis>.
639 <term>Packages</term>
642 A package is request or response, encoded in some protocol,
643 issued by a client, making its way through Metaproxy, send to or
644 received from a server, or sent back to the client.
647 The core of a package is the protocol unit - for example, a
648 Z39.50 Init Request or Search Response, or an SRU searchRetrieve
649 URL or Explain Response. In addition to this core, a package
650 also carries some extra information added and used by Metaproxy
654 In general, packages are doctored as they pass through
655 Metaproxy. For example, when the proxy performs authentication
656 and authorization on a Z39.50 Init request, it removes the
657 authentication credentials from the package so that they are not
658 passed onto the back-end server; and when search-response
659 packages are obtained from multiple servers, they are merged
660 into a single unified package that makes its way back to the
669 Packages make their way through routes, which can be thought of
670 as programs that operate on the package data-type. Each
671 incoming package initially makes its way through a default
672 route, but may be switched to a different route based on various
673 considerations. Routes are made up of sequences of filters (see
682 Filters provide the individual instructions within a route, and
683 effect the necessary transformations on packages. A particular
684 configuration of Metaproxy is essentially a set of filters,
685 described by configuration details and arranged in order in one
686 or more routes. There are many kinds of filter - about a dozen
687 at the time of writing with more appearing all the time - each
688 performing a specific function and configured by different
692 The word ``filter'' is sometimes used rather loosely, in two
693 different ways: it may be used to mean a particular
694 <emphasis>type</emphasis> of filter, as when we speak of ``the
695 auth_simple filter'' or ``the multi filter''; or it may be used
696 to be a specific <emphasis>instance</emphasis> of a filter
697 within a Metaproxy configuration. For example, a single
698 configuration will often contain multiple instances of the
699 <literal>z3950_client</literal> filter. In
700 operational terms, of these is a separate filter. In practice,
701 context always make it clear which sense of the word ``filter''
705 Extensibility of Metaproxy is primarily through the creation of
706 plugins that provide new filters. The filter API is small and
707 conceptually simple, but there are many details to master. See
709 <link linkend="filters">Filters</link>.
715 Since packages are created and handled by the system itself, and
716 routes are conceptually simple, most of the remainder of this
717 document concentrates on filters. After a brief overview of the
718 filter types follows, along with some thoughts on possible future
725 <chapter id="filters">
726 <title>Filters</title>
729 <section id="filters-introductory-notes">
730 <title>Introductory notes</title>
732 It's useful to think of Metaproxy as an interpreter providing a small
733 number of primitives and operations, but operating on a very
734 complex data type, namely the ``package''.
737 A package represents a Z39.50 or SRU/W request (whether for Init,
738 Search, Scan, etc.) together with information about where it came
739 from. Packages are created by front-end filters such as
740 <literal>frontend_net</literal> (see below), which reads them from
741 the network; other front-end filters are possible. They then pass
742 along a route consisting of a sequence of filters, each of which
743 transforms the package and may also have side-effects such as
744 generating logging. Eventually, the route will yield a response,
745 which is sent back to the origin.
748 There are many kinds of filter: some that are defined statically
749 as part of Metaproxy, and others may be provided by third parties
750 and dynamically loaded. They all conform to the same simple API
751 of essentially two methods: <function>configure()</function> is
752 called at startup time, and is passed an XML DOM tree representing that
753 part of the configuration file that pertains to this filter
754 instance: it is expected to walk that tree extracting relevant
755 information; and <function>process()</function> is called every
756 time the filter has to processes a package.
759 While all filters provide the same API, there are different modes
760 of functionality. Some filters are sources: they create
762 (<literal>frontend_net</literal>);
763 others are sinks: they consume packages and return a result
764 (<literal>backend_test</literal>,
765 <literal>bounce</literal>,
766 <literal>http_file</literal>,
767 <literal>z3950_client</literal>);
768 the others are true filters, that read, process and pass on the
769 packages they are fed
770 (<literal>auth_simple</literal>,
771 <literal>log</literal>,
772 <literal>multi</literal>,
773 <literal>query_rewrite</literal>,
774 <literal>record_transform</literal>,
775 <literal>session_shared</literal>,
776 <literal>sru_z3950</literal>,
777 <literal>template</literal>,
778 <literal>virt_db</literal>).
783 <section id="overview.filter.types">
784 <title>Overview of filter types</title>
786 We now briefly consider each of the types of filter supported by
787 the core Metaproxy binary. This overview is intended to give a
788 flavor of the available functionality; more detailed information
789 about each type of filter is included below in
790 <xref linkend="reference"/>.
793 The filters are here named by the string that is used as the
794 <literal>type</literal> attribute of a
795 <literal><filter></literal> element in the configuration
796 file to request them, with the name of the class that implements
797 them in parentheses. (The classname is not needed for normal
798 configuration and use of Metaproxy; it is useful only to
802 The filters are here listed in alphabetical order:
809 New virt_db-alike that does inteligent peer choice, explain merging,
810 adds FD&N to explain. Keeps init responses (like "virt_db Classic"),
811 makes routing choices based on local explain knowledge. Ref IDDI
814 Filter to convert Explain Classic to ZeeRex.
816 CQL2PQF (which needs augmented ZeeRex) - MARC for Talis.
820 Figure out what additional information we need in:
821 ZeeRex (check against D3.1)
822 Init request (e.g. loop detection)
823 Query package (e.g. number of hops)
824 Query response (e.g. record source)
828 <section id="auth_simple">
829 <title><literal>auth_simple</literal>
830 (mp::filter::AuthSimple)</title>
832 Simple authentication and authorization. The configuration
833 specifies the name of a file that is the user register, which
834 lists <varname>username</varname>:<varname>password</varname>
835 pairs, one per line, colon separated. When a session begins, it
836 is rejected unless username and passsword are supplied, and match
837 a pair in the register. The configuration file may also specific
838 the name of another file that is the target register: this lists
839 lists <varname>username</varname>:<varname>dbname</varname>,<varname>dbname</varname>...
840 sets, one per line, with multiple database names separated by
841 commas. When a search is processed, it is rejected unless the
842 database to be searched is one of those listed as available to
847 <section id="backend_test">
848 <title><literal>backend_test</literal>
849 (mp::filter::Backend_test)</title>
851 A partial sink that provides dummy responses in the manner of the
852 <literal>yaz-ztest</literal> Z39.50 server. This is useful only
853 for testing. Seriously, you don't need this. Pretend you didn't
854 even read this section.
858 <section id="bounce">
859 <title><literal>bounce</literal>
860 (mp::filter::Bounce)</title>
862 A sink that swallows <emphasis>all packages</emphasis>,
863 and returns them almost unprocessed.
864 It never sends any package of any type further down the row, but
865 sets Z39.50 packages to Z_Close, and HTTP_Request packages to
866 HTTP_Response err code 400 packages, and adds a suitable bounce
868 The bounce filter is usually added at end of each filter chain route
869 to prevent infinite hanging of for example HTTP
870 requests packages when only the Z39.50 client partial sink
871 filter is found in the
876 <section id="cql_rpn">
877 <title><literal>cql_rpn</literal>
878 (mp::filter::CQLtoRPN)</title>
880 A query language transforming filter which catches Z39.50
881 <literal>searchRequest</literal>
882 packages containing <literal>CQL</literal> queries, transforms
883 those to <literal>RPN</literal> queries,
884 and sends the <literal>searchRequests</literal> on to the next
885 filters. It is among other things useful in a SRU context.
889 <section id="frontend_net">
890 <title><literal>frontend_net</literal>
891 (mp::filter::FrontendNet)</title>
893 A source that accepts Z39.50 connections from a port
894 specified in the configuration, reads protocol units, and
895 feeds them into the next filter in the route. When the result is
896 received, it is returned to the original origin.
900 <section id="http_file">
901 <title><literal>http_file</literal>
902 (mp::filter::HttpFile)</title>
904 A partial sink which swallows only
905 <literal>HTTP_Request</literal> packages, and
906 returns the contents of files from the local
907 filesystem in response to HTTP requests.
908 It lets Z39.50 packages and all other forthcoming package types
911 does mean that Metaproxy is also a Web-server in its spare time. So
912 far it does not contain either an email-reader or a Lisp
913 interpreter, but that day is surely coming.)
917 <section id="load_balance">
918 <title><literal>load_balance</literal>
919 (mp::filter::LoadBalance)</title>
921 Performs load balancing for incoming Z39.50 init requests.
922 It is used together with the <literal>virt_db</literal> filter,
923 but unlike the <literal>multi</literal> filter it does send an
924 entire session to only one of the virtual backends. The
925 <literal>load_balance</literal> filter is assuming that
926 all backend targets have equal content, and chooses the backend
927 with least load cost for a new session.
930 This filter is experimental and yet not mature for heavy load
938 <title><literal>log</literal>
939 (mp::filter::Log)</title>
941 Writes logging information to standard output, and passes on
942 the package unchanged. A log file name can be specified, as well
943 as multiple different logging formats.
948 <title><literal>multi</literal>
949 (mp::filter::Multi)</title>
951 Performs multi-database searching.
953 <link linkend="multidb">the extended discussion</link>
954 of virtual databases and multi-database searching below.
958 <section id="query_rewrite">
959 <title><literal>query_rewrite</literal>
960 (mp::filter::QueryRewrite)</title>
962 Rewrites Z39.50 <literal>Type-1</literal>
963 and <literal>Type-101</literal> (``<literal>RPN</literal>'')
965 three-step process: the query is transliterated from Z39.50
966 packet structures into an XML representation; that XML
967 representation is transformed by an XSLT stylesheet; and the
968 resulting XML is transliterated back into the Z39.50 packet
974 <section id="record_transform">
975 <title><literal>record_transform</literal>
976 (mp::filter::RecordTransform)</title>
978 This filter acts only on Z3950 present requests, and let all
979 other types of packages and requests pass untouched. It's use is
980 twofold: blocking Z3950 present requests, which the backend
981 server does not understand and can not honor, and transforming
982 the present syntax and elementset name according to the rules
983 specified, to fetch only existing record formats, and transform
984 them on the fly to requested record syntaxes.
988 <section id="session_shared">
989 <title><literal>session_shared</literal>
990 (mp::filter::SessionShared)</title>
992 This filter implements global sharing of
993 result sets (i.e. between threads and therefore between
994 clients), yielding performance improvements by clever resource
999 <section id="sru_z3950">
1000 <title><literal>sru_z3950</literal>
1001 (mp::filter::SRUtoZ3950)</title>
1003 This filter transforms valid
1004 SRU GET/POST/SOAP searchRetrieve requests to Z3950 init, search,
1005 and present requests, and wraps the
1006 received hit counts and XML records into suitable SRU response
1008 The <literal>sru_z3950</literal> filter processes also SRU
1009 GET/POST/SOAP explain requests, returning
1010 either the absolute minimum required by the standard, or a full
1011 pre-defined ZeeReX explain record.
1013 <ulink url="&url.zeerex.explain;">ZeeReX Explain</ulink>
1014 standard pages and the
1015 <ulink url="&url.sru.explain;">SRU Explain</ulink> pages
1016 for more information on the correct explain syntax.
1017 SRU scan requests are not supported yet.
1021 <section id="template">
1022 <title><literal>template</literal>
1023 (mp::filter::Template)</title>
1025 Does nothing at all, merely passing the packet on. (Maybe it
1026 should be called <literal>nop</literal> or
1027 <literal>passthrough</literal>?) This exists not to be used, but
1028 to be copied - to become the skeleton of new filters as they are
1029 written. As with <literal>backend_test</literal>, this is not
1030 intended for civilians.
1034 <section id="virt_db">
1035 <title><literal>virt_db</literal>
1036 (mp::filter::VirtualDB)</title>
1038 Performs virtual database selection: based on the name of the
1039 database in the search request, a server is selected, and its
1040 address added to the request in a <literal>VAL_PROXY</literal>
1041 otherInfo packet. It will subsequently be used by a
1042 <literal>z3950_client</literal> filter.
1044 <link linkend="multidb">the extended discussion</link>
1045 of virtual databases and multi-database searching below.
1049 <section id="z3950_client">
1050 <title><literal>z3950_client</literal>
1051 (mp::filter::Z3950Client)</title>
1053 A partial sink which swallows only Z39.50 packages.
1054 It performs Z39.50 searching and retrieval by proxying the
1055 packages that are passed to it. Init requests are sent to the
1056 address specified in the <literal>VAL_PROXY</literal> otherInfo
1057 attached to the request: this may have been specified by client,
1058 or generated by a <literal>virt_db</literal> filter earlier in
1059 the route. Subsequent requests are sent to the same address,
1060 which is remembered at Init time in a Session object.
1061 HTTP_Request packages and all other forthcoming package types
1062 are passed untouched.
1067 <section id="zeerex_explain">
1068 <title><literal>zeerex_explain</literal>
1069 (mp::filter::ZeerexExplain)</title>
1071 This filter acts as a sink for
1072 Z39.50 explain requests, returning a static ZeeReX
1073 Explain XML record from the config section. All other packages
1076 <ulink url="&url.zeerex.explain;">ZeeReX Explain</ulink>
1078 for more information on the correct explain syntax.
1082 This filter is not yet completed.
1091 <section id="future.directions">
1092 <title>Future directions</title>
1094 Some other filters that do not yet exist, but which would be
1095 useful, are briefly described. These may be added in future
1096 releases (or may be created by third parties, as loadable
1102 <term><literal>frontend_cli</literal> (source)</term>
1105 Command-line interface for generating requests.
1110 <term><literal>sru_client</literal> (sink)</term>
1113 SRU/GET and SRU/SOAP searching and retrieval.
1118 <term><literal>opensearch_client</literal> (sink)</term>
1121 A9 OpenSearch searching and retrieval.
1131 <chapter id="configuration">
1132 <title>Configuration: the Metaproxy configuration file format</title>
1135 <section id="configuration-introductory-notes">
1136 <title>Introductory notes</title>
1138 If Metaproxy is an interpreter providing operations on packages, then
1139 its configuration file can be thought of as a program for that
1140 interpreter. Configuration is by means of a single XML file, the name
1141 of which is supplied as the sole command-line argument to the
1142 <command>metaproxy</command> program. (See
1143 <xref linkend="reference"/> below for more information on invoking
1148 <section id="overview.xml.structure">
1149 <title>Overview of the config file XML structure</title>
1151 All elements and attributes are in the namespace
1152 <ulink url="http://indexdata.com/metaproxy"/>.
1153 This is most easily achieved by setting the default namespace on
1154 the top-level element, as here:
1157 <metaproxy xmlns="http://indexdata.com/metaproxy" version="1.0">
1160 The top-level element is <metaproxy>. This contains
1161 a <dlpath> element,
1162 a <start> element,
1163 a <filters> element and
1164 a <routes> element, in that order. <dlpath> and
1165 <filters> are optional; the other two are mandatory.
1166 All four are non-repeatable.
1169 The <dlpath;> element contains a text element which
1170 specifies the location of filter modules. This is only needed
1171 if Metaproxy must load 3rd party filters (most filters with Metaproxy
1172 are built into the Metaproxy application).
1175 The <start> element is empty, but carries a
1176 <literal>route</literal> attribute, whose value is the name of
1177 route at which to start running - analogous to the name of the
1178 start production in a formal grammar.
1181 If present, <filters> contains zero or more <filter>
1182 elements. Each filter carries a <literal>type</literal> attribute
1183 which specifies what kind of filter is being defined
1184 (<literal>frontend_net</literal>, <literal>log</literal>, etc.)
1185 and contain various elements that provide suitable configuration
1186 for a filter of its type. The filter-specific elements are
1188 <xref linkend="reference"/>.
1189 Filters defined in this part of the file must carry an
1190 <literal>id</literal> attribute so that they can be referenced
1194 <routes> contains one or more <route> elements, each
1195 of which must carry an <literal>id</literal> element. One of the
1196 routes must have the ID value that was specified as the start
1197 route in the <start> element's <literal>route</literal>
1198 attribute. Each route contains zero or more <filter>
1199 elements. These are of two types. They may be empty, but carry a
1200 <literal>refid</literal> attribute whose value is the same as the
1201 <literal>id</literal> of a filter previously defined in the
1202 <filters> section. Alternatively, a route within a filter
1203 may omit the <literal>refid</literal> attribute, but contain
1204 configuration elements similar to those used for filters defined
1205 in the <filters> section. (In other words, each filter in a
1206 route may be included either by reference or by physical
1212 <section id="example.configuration">
1213 <title>An example configuration</title>
1215 The following is a small, but complete, Metaproxy configuration
1216 file (included in the distribution as
1217 <literal>metaproxy/etc/config1.xml</literal>).
1218 This file defines a very simple configuration that simply proxies
1219 to whatever back-end server the client requests, but logs each
1220 request and response. This can be useful for debugging complex
1221 client-server dialogues.
1223 <screen><![CDATA[<?xml version="1.0"?>
1224 <metaproxy xmlns="http://indexdata.com/metaproxy" version="1.0">
1225 <dlpath>/usr/lib/metaproxy/modules</dlpath>
1226 <start route="start"/>
1228 <filter id="frontend" type="frontend_net">
1231 <filter id="backend" type="z3950_client">
1236 <filter refid="frontend"/>
1237 <filter type="log"/>
1238 <filter refid="backend"/>
1239 <filter type="bounce"/>
1245 It works by defining a single route, called
1246 <literal>start</literal>, which consists of a sequence of four
1247 filters. The first and last of these are included by reference:
1248 their <literal><filter></literal> elements have
1249 <literal>refid</literal> attributes that refer to filters defined
1250 within the prior <literal><filters></literal> section. The
1251 middle filter is included inline in the route.
1254 The four filters in the route are as follows: first, a
1255 <literal>frontend_net</literal> filter accepts Z39.50 requests
1256 from any host on port 9000; then these requests are passed through
1257 a <literal>log</literal> filter that emits a message for each
1258 request; they are then fed into a <literal>z3950_client</literal>
1259 filter, which forwards all Z39.50 requests to the client-specified
1260 back-end Z39.509 server. Those Z39.50 packages are returned by the
1261 <literal>z3950_client</literal> filter, with the response data
1262 filled by the external Z39.50 server targeted.
1263 All non-Z39.50 packages are passed through to the
1264 <literal>bounce</literal> filter, which definitely bounces
1265 everything, including fish, bananas, cold pyjamas,
1266 mutton, beef and trout packages.
1267 When the response arrives, it is handed
1268 back to the <literal>log</literal> filter, which emits another
1269 message; and then to the <literal>frontend_net</literal> filter,
1270 which returns the response to the client.
1274 <section id="config-file-modularity">
1275 <title>Config file modularity</title>
1277 Metaproxy XML configuration snippets can be reused by other
1278 filters using the <literal>XInclude</literal> standard, as seen in
1279 the <literal>/etc/config-sru-to-z3950.xml</literal> example SRU
1282 <filter id="sru" type="sru_z3950">
1283 <database name="Default">
1284 <xi:include xmlns:xi="http://www.w3.org/2001/XInclude"
1285 href="explain.xml"/>
1292 <section id="config-file-syntax-check">
1293 <title>Config file syntax checking</title>
1295 The distribution contains RelaxNG Compact and XML syntax checking
1296 files, as well as XML Schema files. These are found in the
1299 xml/schema/metaproxy.rnc
1300 xml/schema/metaproxy.rng
1301 xml/schema/metaproxy.xsd
1303 and can be used to verify or debug the XML structure of
1304 configuration files. For example, using the utility
1305 <filename>xmllint</filename>, syntax checking is done like this:
1307 xmllint --noout --schema xml/schema/metaproxy.xsd etc/config-local.xml
1308 xmllint --noout --relaxng xml/schema/metaproxy.rng etc/config-local.xml
1310 (A recent version of <literal>libxml2</literal> is required, as
1311 support for XML Schemas is a relatively recent addition.)
1314 You can of course use any other RelaxNG or XML Schema compliant tool
1322 <chapter id="multidb">
1323 <title>Virtual databases and multi-database searching</title>
1326 <section id="multidb-introductory-notes">
1327 <title>Introductory notes</title>
1329 Two of Metaproxy's filters are concerned with multiple-database
1330 operations. Of these, <literal>virt_db</literal> can work alone
1331 to control the routing of searches to one of a number of servers,
1332 while <literal>multi</literal> can work together with
1333 <literal>virt_db</literal> to perform multi-database searching, merging
1334 the results into a unified result-set - ``metasearch in a box''.
1337 The interaction between
1338 these two filters is necessarily complex: it reflects the real,
1339 irreducible complexity of multi-database searching in a protocol such
1340 as Z39.50 that separates initialization from searching, and in
1341 which the database to be searched is not known at initialization
1345 It's possible to use these filters without understanding the
1346 details of their functioning and the interaction between them; the
1347 next two sections of this chapter are ``HOW-TO'' guides for doing
1348 just that. However, debugging complex configurations will require
1349 a deeper understanding, which the last two sections of this
1350 chapters attempt to provide.
1355 <section id="multidb.virt_db">
1356 <title>Virtual databases with the <literal>virt_db</literal> filter</title>
1358 Working alone, the purpose of the
1359 <literal>virt_db</literal>
1360 filter is to route search requests to one of a selection of
1361 back-end databases. In this way, a single Z39.50 endpoint
1362 (running Metaproxy) can provide access to several different
1363 underlying services, including those that would otherwise be
1364 inaccessible due to firewalls. In many useful configurations, the
1365 back-end databases are local to the Metaproxy installation, but
1366 the software does not enforce this, and any valid Z39.50 servers
1367 may be used as back-ends.
1370 For example, a <literal>virt_db</literal>
1371 filter could be set up so that searches in the virtual database
1372 ``lc'' are forwarded to the Library of Congress bibliographic
1373 catalogue server, and searches in the virtual database ``marc''
1374 are forwarded to the toy database of MARC records that Index Data
1375 hosts for testing purposes. A <literal>virt_db</literal>
1376 configuration to make this switch would look like this:
1378 <screen><![CDATA[<filter type="virt_db">
1380 <database>lc</database>
1381 <target>z3950.loc.gov:7090/voyager</target>
1384 <database>marc</database>
1385 <target>indexdata.com/marc</target>
1387 </filter>]]></screen>
1389 As well as being useful in it own right, this filter also provides
1390 the foundation for multi-database searching.
1395 <section id="multidb.multi">
1396 <title>Multi-database search with the <literal>multi</literal> filter</title>
1398 To arrange for Metaproxy to broadcast searches to multiple back-end
1399 servers, the configuration needs to include two components: a
1400 <literal>virt_db</literal>
1401 filter that specifies multiple
1402 <literal><target></literal>
1403 elements, and a subsequent
1404 <literal>multi</literal>
1405 filter. Here, for example, is a complete configuration that
1406 broadcasts searches to both the Library of Congress catalogue and
1407 Index Data's tiny testing database of MARC records:
1409 <screen><![CDATA[<?xml version="1.0"?>
1410 <metaproxy xmlns="http://indexdata.com/metaproxy" version="1.0">
1411 <start route="start"/>
1414 <filter type="frontend_net">
1415 <threads>10</threads>
1418 <filter type="virt_db">
1420 <database>lc</database>
1421 <target>z3950.loc.gov:7090/voyager</target>
1424 <database>marc</database>
1425 <target>indexdata.com/marc</target>
1428 <database>all</database>
1429 <target>z3950.loc.gov:7090/voyager</target>
1430 <target>indexdata.com/marc</target>
1433 <filter type="multi"/>
1434 <filter type="z3950_client">
1435 <timeout>30</timeout>
1437 <filter type="bounce"/>
1440 </metaproxy>]]></screen>
1443 <literal>virt_db</literal>
1444 filter that specifies multiple
1445 <literal><target></literal>
1446 elements but without a subsequent
1447 <literal>multi</literal>
1448 filter yields surprising and undesirable results, as will be
1449 described below. Don't do that.)
1452 Metaproxy can be invoked with this configuration as follows:
1454 <screen>../src/metaproxy --config config-simple-multi.xml</screen>
1456 And thereafter, Z39.50 clients can connect to the running server
1457 (on port 9000, as specified in the configuration) and search in
1458 any of the databases
1459 <literal>lc</literal> (the Library of Congress catalogue),
1460 <literal>marc</literal> (Index Data's test database of MARC records)
1462 <literal>all</literal> (both of these). As an example, a session
1463 using the YAZ command-line client <literal>yaz-client</literal> is
1464 here included (edited for brevity and clarity):
1466 <screen><![CDATA[$ yaz-client @:9000
1470 Search was a success.
1471 Number of hits: 10000, setno 1
1475 Search was a success.
1476 Number of hits: 10, setno 3
1480 Search was a success.
1481 Number of hits: 10010, setno 4
1484 [marc]Record type: USmarc
1487 005 00000000000000.0
1488 008 910710c19910701nju 00010 eng
1492 100 10 $a Jack Collins
1493 245 10 $a How to program a computer
1499 [VOYAGER]Record type: USmarc
1501 005 20041229102447.0
1502 008 030910s2004 caua 000 0 eng
1503 035 $a (DLC) 2003112666
1504 906 $a 7 $b cbc $c orignew $d 4 $e epcn $f 20 $g y-gencatlg
1505 925 0 $a acquire $b 1 shelf copy $x policy default
1506 955 $a pc10 2003-09-10 $a pv12 2004-06-23 to SSCD; $h sj05 2004-11-30 $e sj05 2004-11-30 to Shelf.
1509 040 $a DLC $c DLC $d DLC
1510 050 00 $a MLCM 2004/03312 (G)
1511 245 10 $a 007, everything or nothing : $b Prima's official strategy guide / $c created by Kaizen Media Group.
1512 246 3 $a Double-O-seven, everything or nothing
1513 246 30 $a Prima's official strategy guide
1514 260 $a Roseville, CA : $b Prima Games, $c c2004.
1515 300 $a 161 p. : $b col. ill. ; $c 28 cm.
1516 500 $a "Platforms: Nintendo GameCube, Macintosh, PC, PlayStation 2 computer entertainment system, Xbox"--P. [4] of cover.
1517 650 0 $a Video games.
1518 710 2 $a Kaizen Media Group.
1519 856 42 $3 Publisher description $u http://www.loc.gov/catdir/description/random052/2003112666.html
1524 As can be seen, the first record in the result set is from the
1525 Index Data test database, and the second from the Library of
1526 Congress database. The result-set continues alternating records
1527 round-robin style until the point where one of the databases'
1528 records are exhausted.
1531 This example uses only two back-end databases; more may be used.
1532 There is no limitation imposed on the number of databases that may
1533 be metasearched in this way: issues of resource usage and
1534 administrative complexity dictate the practical limits.
1537 What happens when one of the databases doesn't respond? By default,
1538 the entire multi-database search fails, and the appropriate
1539 diagnostic is returned to the client. This is usually appropriate
1540 during development, when technicians need maximum information, but
1541 can be inconvenient in deployment, when users typically don't want
1542 to be bothered with problems of this kind and prefer just to get
1543 the records from the databases that are available. To obtain this
1544 latter behavior add an empty
1545 <literal><hideunavailable></literal>
1547 <literal>multi</literal> filter:
1549 <screen><![CDATA[ <filter type="multi">
1551 </filter>]]></screen>
1553 Under this regime, an error is reported to the client only if
1554 <emphasis>all</emphasis> the databases in a multi-database search
1560 <section id="multidb.what">
1561 <title>What's going on?</title>
1563 <title>Lark's vomit</title>
1565 This section goes into a level of technical detail that is
1566 probably not necessary in order to configure and use Metaproxy.
1567 It is provided only for those who like to know how things work.
1568 You should feel free to skip on to the next section if this one
1569 doesn't seem like fun.
1573 Hold on tight - this may get a little hairy.
1576 In the general course of things, a Z39.50 Init request may carry
1577 with it an otherInfo packet of type <literal>VAL_PROXY</literal>,
1578 whose value indicates the address of a Z39.50 server to which the
1579 ultimate connection is to be made. (This otherInfo packet is
1580 supported by YAZ-based Z39.50 clients and servers, but has not yet
1581 been ratified by the Maintenance Agency and so is not widely used
1582 in non-Index Data software. We're working on it.)
1583 The <literal>VAL_PROXY</literal> packet functions
1584 analogously to the absoluteURI-style Request-URI used with the GET
1585 method when a web browser asks a proxy to forward its request: see
1587 <ulink url="http://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html#sec5.1.2"
1588 >Request-URI</ulink>
1590 <ulink url="http://www.w3.org/Protocols/rfc2616/rfc2616.html"
1591 >the HTTP 1.1 specification</ulink>.
1594 Within Metaproxy, Search requests that are part of the same
1595 session as an Init request that carries a
1596 <literal>VAL_PROXY</literal> otherInfo are also annotated with the
1597 same information. The role of the <literal>virt_db</literal>
1598 filter is to rewrite this otherInfo packet dependent on the
1599 virtual database that the client wants to search.
1602 When Metaproxy receives a Z39.50 Init request from a client, it
1603 doesn't immediately forward that request to the back-end server.
1604 Why not? Because it doesn't know <emphasis>which</emphasis>
1605 back-end server to forward it to until the client sends a Search
1606 request that specifies the database that it wants to search in.
1607 Instead, it just treasures the Init request up in its heart; and,
1608 later, the first time the client does a search on one of the
1609 specified virtual databases, a connection is forged to the
1610 appropriate server and the Init request is forwarded to it. If,
1611 later in the session, the same client searches in a different
1612 virtual database, then a connection is forged to the server that
1613 hosts it, and the same cached Init request is forwarded there,
1617 All of this clever Init-delaying is done by the
1618 <literal>frontend_net</literal> filter. The
1619 <literal>virt_db</literal> filter knows nothing about it; in
1620 fact, because the Init request that is received from the client
1621 doesn't get forwarded until a Search request is received, the
1622 <literal>virt_db</literal> filter (and the
1623 <literal>z3950_client</literal> filter behind it) doesn't even get
1624 invoked at Init time. The <emphasis>only</emphasis> thing that a
1625 <literal>virt_db</literal> filter ever does is rewrite the
1626 <literal>VAL_PROXY</literal> otherInfo in the requests that pass
1630 It is possible for a <literal>virt_db</literal> filter to contain
1632 <literal><target></literal>
1633 elements. What does this mean? Only that the filter will add
1634 multiple <literal>VAL_PROXY</literal> otherInfo packets to the
1635 Search requests that pass through it. That's because the virtual
1636 DB filter is dumb, and does exactly what it's told - no more, no
1638 If a Search request with multiple <literal>VAL_PROXY</literal>
1639 otherInfo packets reaches a <literal>z3950_client</literal>
1640 filter, this is an error. That filter doesn't know how to deal
1641 with multiple targets, so it will either just pick one and search
1642 in it, or (better) fail with an error message.
1645 The <literal>multi</literal> filter comes to the rescue! This is
1646 the only filter that knows how to deal with multiple
1647 <literal>VAL_PROXY</literal> otherInfo packets, and it does so by
1648 making multiple copies of the entire Search request: one for each
1649 <literal>VAL_PROXY</literal>. Each of these new copies is then
1650 passed down through the remaining filters in the route. (The
1651 copies are handled in parallel though the
1652 spawning of new threads.) Since the copies each have only one
1653 <literal>VAL_PROXY</literal> otherInfo, they can be handled by the
1654 <literal>z3950_client</literal> filter, which happily deals with
1655 each one individually. When the results of the individual
1656 searches come back up to the <literal>multi</literal> filter, it
1657 merges them into a single Search response, which is what
1658 eventually makes it back to the client.
1663 <section id="multidb.picture">
1664 <title>A picture is worth a thousand words (but only five hundred on 64-bit architectures)</title>
1668 <imagedata fileref="multi.pdf" format="PDF" scale="50"/>
1671 <imagedata fileref="multi.png" format="PNG"/>
1674 <!-- Fall back if none of the images can be used -->
1676 [Here there should be a diagram showing the progress of
1677 packages through the filters during a simple virtual-database
1678 search and a multi-database search, but is seems that your
1679 tool chain has not been able to include the diagram in this
1683 <!-- ### This used to work with an older version of DocBook
1685 <para>Caption: progress of packages through filters.</para>
1688 </inlinemediaobject>
1694 <chapter id="sru-server">
1695 <title>Combined SRU webservice and Z39.50 server configuration</title>
1697 Metaproxy can act as
1698 <ulink url="&url.sru;">SRU</ulink> and
1699 web service server, which translates web service requests to
1700 <ulink url="&url.z39.50;">ANSI/NISO Z39.50</ulink> packages and
1701 sends them off to common available targets.
1704 A typical setup for this operation needs a filter route including the
1708 <table id="sru-server-table-config" frame="top">
1709 <title>SRU/Z39.50 Server Filter Route Configuration</title>
1713 <entry>Filter</entry>
1714 <entry>Importance</entry>
1715 <entry>Purpose</entry>
1721 <entry><literal>frontend_net</literal></entry>
1722 <entry>required</entry>
1723 <entry>Accepting HTTP connections and passing them to following
1724 filters. Since this filter also accepts Z39.50 connections, the
1725 server works as SRU and Z39.50 server on the same port.</entry>
1728 <entry><literal>sru_z3950</literal></entry>
1729 <entry>required</entry>
1730 <entry>Accepting SRU GET/POST/SOAP explain and
1731 searchRetrieve requests for the the configured databases.
1732 Explain requests are directly served from the static XML configuration.
1733 SearchRetrieve requests are
1734 transformed to Z39.50 search and present packages.
1735 All other HTTP and Z39.50 packages are passed unaltered.</entry>
1738 <entry><literal>http_file</literal></entry>
1739 <entry>optional</entry>
1740 <entry>Serving HTTP requests from the filesystem. This is only
1741 needed if the server should serve XSLT stylesheets, static HTML
1742 files or Java Script for thin browser based clients.
1743 Z39.50 packages are passed unaltered.</entry>
1746 <entry><literal>cql_rpn</literal></entry>
1747 <entry>required</entry>
1748 <entry>Usually, Z39.50 servers do not talk CQL, hence the
1749 translation of the CQL query language to RPN is mandatory in
1750 most cases. Affects only Z39.50 search packages.</entry>
1753 <entry><literal>record_transform</literal></entry>
1754 <entry>optional</entry>
1755 <entry>Some Z39.50 backend targets can not present XML record
1756 syntaxes in common wanted element sets. using this filter, one
1757 can transform binary MARC records to MARCXML records, and
1758 further transform those to any needed XML schema/format by XSLT
1759 transformations. Changes only Z39.50 present packages.</entry>
1762 <entry><literal>session_shared</literal></entry>
1763 <entry>optional</entry>
1764 <entry>The stateless nature of web services requires frequent
1765 re-searching of the same targets for display of paged result set
1766 records. This might be an unacceptable burden for the accessed
1767 backend Z39.50 targets, and this mosule can be added for
1768 efficient backend target resource pooling.</entry>
1771 <entry><literal>z3950_client</literal></entry>
1772 <entry>required</entry>
1773 <entry>Finally, a Z39.50 package sink is needed in the filter
1774 chain to provide the response packages. The Z39.50 client module
1775 is used to access external targets over the network, but any
1776 coming local Z39.50 package sink could be used instead of.</entry>
1779 <entry><literal>bounce</literal></entry>
1780 <entry>required</entry>
1781 <entry>Any Metaproxy package arriving here did not do so by
1782 purpose, and is bounced back with connection closure. this
1783 prevents inifinite package hanging inside the SRU server.</entry>
1789 A typical minimal example <ulink url="&url.sru;">SRU</ulink>
1790 server configuration file is found in the tarball distribution at
1791 <literal>etc/config-sru-to-z3950.xml</literal>.
1794 Off course, any other metaproxy modules can be integrated into a
1795 SRU server solution, including, but not limited to, load balancing,
1796 multiple target querying
1797 (see <xref linkend="multidb"/>), and complex RPN query rewrites.
1804 <chapter id="extensions">
1805 <title>Writing extensions for Metaproxy</title>
1806 <para>### To be written</para>
1812 <chapter id="classes">
1813 <title>Classes in the Metaproxy source code</title>
1816 <section id="classes-introductory-notes">
1817 <title>Introductory notes</title>
1819 <emphasis>Stop! Do not read this!</emphasis>
1820 You won't enjoy it at all. You should just skip ahead to
1821 <xref linkend="reference"/>,
1823 <!-- The remainder of this paragraph is lifted verbatim from
1824 Douglas Adams' _Hitch Hiker's Guide to the Galaxy_, chapter 8 -->
1825 you things you really need to know, like the fact that the
1826 fabulously beautiful planet Bethselamin is now so worried about
1827 the cumulative erosion by ten billion visiting tourists a year
1828 that any net imbalance between the amount you eat and the amount
1829 you excrete whilst on the planet is surgically removed from your
1830 bodyweight when you leave: so every time you go to the lavatory it
1831 is vitally important to get a receipt.
1834 This chapter contains documentation of the Metaproxy source code, and is
1835 of interest only to maintainers and developers. If you need to
1836 change Metaproxy's behavior or write a new filter, then you will most
1837 likely find this chapter helpful. Otherwise it's a waste of your
1838 good time. Seriously: go and watch a film or something.
1839 <citetitle>This is Spinal Tap</citetitle> is particularly good.
1842 Still here? OK, let's continue.
1845 In general, classes seem to be named big-endianly, so that
1846 <literal>FactoryFilter</literal> is not a filter that filters
1847 factories, but a factory that produces filters; and
1848 <literal>FactoryStatic</literal> is a factory for the statically
1849 registered filters (as opposed to those that are dynamically
1854 <section id="individual.classes">
1855 <title>Individual classes</title>
1857 The classes making up the Metaproxy application are here listed by
1858 class-name, with the names of the source files that define them in
1862 <section id="class-FactoryFilter">
1863 <title><literal>mp::FactoryFilter</literal>
1864 (<filename>factory_filter.cpp</filename>)</title>
1866 A factory class that exists primarily to provide the
1867 <literal>create()</literal> method, which takes the name of a
1868 filter class as its argument and returns a new filter of that
1869 type. To enable this, the factory must first be populated by
1870 calling <literal>add_creator()</literal> for static filters (this
1871 is done by the <literal>FactoryStatic</literal> class, see below)
1872 and <literal>add_creator_dyn()</literal> for filters loaded
1877 <section id="class-FactoryStatic">
1878 <title><literal>mp::FactoryStatic</literal>
1879 (<filename>factory_static.cpp</filename>)</title>
1881 A subclass of <literal>FactoryFilter</literal> which is
1882 responsible for registering all the statically defined filter
1883 types. It does this by knowing about all those filters'
1884 structures, which are listed in its constructor. Merely
1885 instantiating this class registers all the static classes. It is
1886 for the benefit of this class that <literal>struct
1887 metaproxy_1_filter_struct</literal> exists, and that all the filter
1888 classes provide a static object of that type.
1892 <section id="class-filter-Base">
1893 <title><literal>mp::filter::Base</literal>
1894 (<filename>filter.cpp</filename>)</title>
1896 The virtual base class of all filters. The filter API is, on the
1897 surface at least, extremely simple: two methods.
1898 <literal>configure()</literal> is passed an XML DOM tree representing
1899 that part of the configuration file that pertains to this filter
1900 instance, and is expected to walk that tree extracting relevant
1901 information. And <literal>process()</literal> processes a
1902 package (see below). That surface simplicity is a bit
1903 misleading, as <literal>process()</literal> needs to know a lot
1904 about the <literal>Package</literal> class in order to do
1909 <section id="class-AuthSimple">
1910 <title><literal>mp::filter::AuthSimple</literal>,
1911 <literal>Backend_test</literal>, etc.
1912 (<filename>filter_auth_simple.cpp</filename>,
1913 <filename>filter_backend_test.cpp</filename>, etc.)</title>
1915 Individual filters. Each of these is implemented by a header and
1916 a source file, named <filename>filter_*.hpp</filename> and
1917 <filename>filter_*.cpp</filename> respectively. All the header
1918 files should be pretty much identical, in that they declare the
1919 class, including a private <literal>Rep</literal> class and a
1920 member pointer to it, and the two public methods.
1923 The source file for each filter needs to supply:
1928 A definition of the private <literal>Rep</literal> class.
1933 Some boilerplate constructors and destructors.
1938 A <literal>configure()</literal> method that uses the
1939 appropriate XML fragment.
1944 Most important, the <literal>process()</literal> method that
1945 does all the actual work.
1951 <section id="class-Package">
1952 <title><literal>mp::Package</literal>
1953 (<filename>package.cpp</filename>)</title>
1955 Represents a package on its way through the series of filters
1956 that make up a route. This is essentially a Z39.50 or SRU APDU
1957 together with information about where it came from, which is
1958 modified as it passes through the various filters.
1962 <section id="class-Pipe">
1963 <title><literal>mp::Pipe</literal>
1964 (<filename>pipe.cpp</filename>)</title>
1966 This class provides a compatibility layer so that we have an IPC
1967 mechanism that works the same under Unix and Windows. It's not
1968 particularly exciting.
1972 <section id="class-RouterChain">
1973 <title><literal>mp::RouterChain</literal>
1974 (<filename>router_chain.cpp</filename>)</title>
1980 <section id="class-RouterFleXML">
1981 <title><literal>mp::RouterFleXML</literal>
1982 (<filename>router_flexml.cpp</filename>)</title>
1988 <section id="class-Session">
1989 <title><literal>mp::Session</literal>
1990 (<filename>session.cpp</filename>)</title>
1996 <section id="class-ThreadPoolSocketObserver">
1997 <title><literal>mp::ThreadPoolSocketObserver</literal>
1998 (<filename>thread_pool_observer.cpp</filename>)</title>
2004 <section id="class-util">
2005 <title><literal>mp::util</literal>
2006 (<filename>util.cpp</filename>)</title>
2008 A namespace of various small utility functions and classes,
2009 collected together for convenience. Most importantly, includes
2010 the <literal>mp::util::odr</literal> class, a wrapper for YAZ's
2015 <section id="class-xml">
2016 <title><literal>mp::xml</literal>
2017 (<filename>xmlutil.cpp</filename>)</title>
2019 A namespace of various XML utility functions and classes,
2020 collected together for convenience.
2026 <section id="other.source.files">
2027 <title>Other Source Files</title>
2029 In addition to the Metaproxy source files that define the classes
2030 described above, there are a few additional files which are
2031 briefly described here:
2035 <term><literal>metaproxy_prog.cpp</literal></term>
2038 The main function of the <command>metaproxy</command> program.
2043 <term><literal>ex_router_flexml.cpp</literal></term>
2046 Identical to <literal>metaproxy_prog.cpp</literal>: it's not clear why.
2051 <term><literal>test_*.cpp</literal></term>
2054 Unit-tests for various modules.
2060 ### Still to be described:
2061 <literal>ex_filter_frontend_net.cpp</literal>,
2062 <literal>filter_dl.cpp</literal>,
2063 <literal>plainfile.cpp</literal>,
2064 <literal>tstdl.cpp</literal>.
2070 <reference id="reference">
2071 <title>Reference</title>
2072 <partintro id="reference-introduction">
2074 The material in this chapter is drawn directly from the individual
2075 manual entries. In particular, the Metaproxy invocation section is
2076 available using <command>man metaproxy</command>, and the section
2077 on each individual filter is available using the name of the filter
2078 as the argument to the <command>man</command> command.
2084 <appendix id="license">
2085 <title>License</title>
2088 Metaproxy, Copyright © 1995-2009 Index Data ApS.
2092 Metaproxy is free software; you can redistribute it and/or modify it under
2093 the terms of the GNU General Public License as published by the Free
2094 Software Foundation; either version 2, or (at your option) any later
2099 Metaproxy is distributed in the hope that it will be useful, but WITHOUT ANY
2100 WARRANTY; without even the implied warranty of MERCHANTABILITY or
2101 FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
2106 You should have received a copy of the GNU General Public License
2107 along with Metaproxy; see the file LICENSE. If not, write to the
2108 Free Software Foundation,
2109 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
2117 <!-- Keep this comment at the end of the file
2122 sgml-minimize-attributes:nil
2123 sgml-always-quote-attributes:t
2126 sgml-parent-document: nil
2127 sgml-local-catalogs: nil
2128 sgml-namecase-general:t