1 <?xml version="1.0" standalone="no"?>
2 <!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.1//EN"
3 "http://www.oasis-open.org/docbook/xml/4.1/docbookx.dtd"
5 <!ENTITY local SYSTEM "local.ent">
6 <!ENTITY manref SYSTEM "manref.xml">
7 <!ENTITY progref SYSTEM "progref.xml">
8 <!ENTITY % common SYSTEM "common/common.ent">
10 <!-- Next line allows imagedata/@format="PDF" and is taken from
11 http://lists.oasis-open.org/archives/docbook/200303/msg00163.html
13 <!ENTITY % local.notation.class "| PDF">
14 <!-- Next line is necessary for some XML parsers, for reasons I
15 don't understand. I got this from
16 http://lists.oasis-open.org/archives/docbook/200303/msg00180.html
18 <!NOTATION PDF SYSTEM "PDF">
20 <!-- $Id: book.xml,v 1.33 2006-06-02 10:23:49 adam Exp $ -->
23 <title>Metaproxy - User's Guide and Reference</title>
25 <firstname>Mike</firstname><surname>Taylor</surname>
28 <firstname>Adam</firstname><surname>Dickmeiss</surname>
32 <holder>Index Data ApS</holder>
36 Metaproxy is a universal router, proxy and encapsulated
37 metasearcher for information retrieval protocols. It accepts,
38 processes, interprets and redirects requests from IR clients using
39 standard protocols such as
40 <ulink url="&url.z39.50;">ANSI/NISO Z39.50</ulink>
41 (and in the future <ulink url="&url.sru;">SRU</ulink>
42 and <ulink url="&url.srw;">SRW</ulink>), as
43 well as functioning as a limited
44 <ulink url="&url.http;">HTTP</ulink> server.
45 Metaproxy is configured by an XML file which
46 specifies how the software should function in terms of routes that
47 the request packets can take through the proxy, each step on a
48 route being an instantiation of a filter. Filters come in many
49 types, one for each operation: accepting Z39.50 packets, logging,
50 query transformation, multiplexing, etc. Further filter-types can
51 be added as loadable modules to extend Metaproxy functionality,
55 The terms under which Metaproxy will be distributed have yet to be
56 established, but it will not necessarily be open source; so users
57 should not at this stage redistribute the code without explicit
58 written permission from the copyright holders, Index Data ApS.
63 <imagedata fileref="common/id.png" format="PNG"/>
66 <imagedata fileref="common/id.eps" format="EPS"/>
73 <chapter id="introduction">
74 <title>Introduction</title>
78 <ulink url="&url.metaproxy;">Metaproxy</ulink>
79 is a standalone program that acts as a universal router, proxy and
80 encapsulated metasearcher for information retrieval protocols such
81 as <ulink url="&url.z39.50;">Z39.50</ulink>, and in the future
82 <ulink url="&url.sru;">SRU</ulink> and <ulink url="&url.srw;">SRW</ulink>.
83 To clients, it acts as a server of these protocols: it can be searched,
84 records can be retrieved from it, etc.
85 To servers, it acts as a client: it searches in them,
86 retrieves records from them, etc. it satisfies its clients'
87 requests by transforming them, multiplexing them, forwarding them
88 on to zero or more servers, merging the results, transforming
89 them, and delivering them back to the client. In addition, it
90 acts as a simple <ulink url="&url.http;">HTTP</ulink> server; support
91 for further protocols can be added in a modular fashion, through the
92 creation of new filters.
97 Fish, bananas, cold pyjamas,
98 Mutton, beef and trout!
99 - attributed to Cole Porter.
102 Metaproxy is a more capable alternative to
103 <ulink url="&url.yazproxy;">YAZ Proxy</ulink>,
104 being more powerful, flexible, configurable and extensible. Among
105 its many advantages over the older, more pedestrian work are
106 support for multiplexing (encapsulated metasearching), routing by
107 database name, authentication and authorisation and serving local
108 files via HTTP. Equally significant, its modular architecture
109 facilitites the creation of pluggable modules implementing further
113 This manual will briefly describe Metaproxy's licensing situation
114 before giving an overview of its architecture, then discussing the
115 key concept of a filter in some depth and giving an overview of
116 the various filter types, then discussing the configuration file
117 format. After this come several optional chapters which may be
118 freely skipped: a detailed discussion of virtual databases and
119 multi-database searching, some notes on writing extensions
120 (additional filter types) and a high-level description of the
121 source code. Finally comes the reference guide, which contains
122 instructions for invoking the <command>metaproxy</command>
123 program, and detailed information on each type of filter,
130 <chapter id="licence">
131 <title>The Metaproxy Licence</title>
133 <emphasis role="strong">
134 No decision has yet been made on the terms under which
135 Metaproxy will be distributed.
137 It is possible that, unlike
138 other Index Data products, metaproxy may not be released under a
139 free-software licence such as the GNU GPL. Until a decision is
140 made and a public statement made, then, and unless it has been
141 delivered to you other specific terms, please treat Metaproxy as
142 though it were proprietary software.
143 The code should not be redistributed without explicit
144 written permission from the copyright holders, Index Data ApS.
148 <chapter id="installation">
149 <title>Installation</title>
151 Metaproxy depends on the following tools/libraries:
153 <varlistentry><term><ulink url="&url.yazplusplus;">YAZ++</ulink></term>
156 This is a C++ library based on <ulink url="&url.yaz;">YAZ</ulink>.
160 <varlistentry><term><ulink url="&url.libxslt;">Libxslt</ulink></term>
162 <para>This is an XSLT processor - based on
163 <ulink url="&url.libxml2;">Libxml2</ulink>. Both Libxml2 and
164 Libxslt must be installed with the development components
165 (header files, etc.) as well as the run-time libraries.
169 <varlistentry><term><ulink url="&url.boost;">Boost</ulink></term>
172 The popular C++ library. Initial versions of Metaproxy
173 was built with 1.33.0. Version 1.33.1 works too.
180 In order to compile Metaproxy a modern C++ compiler is
181 required. Boost, in particular, requires the C++ compiler
182 to facilitate the newest features. Refer to Boost
183 <ulink url="&url.boost.compilers.status;">Compiler Status</ulink>
184 for more information.
187 We have succesfully built Metaproxy using the compilers
188 <ulink url="&url.gcc;">GCC</ulink> version 4.0 and
189 <ulink url="&url.vstudio;">Microsoft Visual Studio</ulink> 2003/2005.
192 <section id="installation.unix">
193 <title>Installation on Unix (from Source)</title>
195 Here is a quick step-by-step guide on how to compile all the
196 tools that Metaproxy uses. Only few systems have none of the required
197 tools binary packages. If, for example, Libxml2/libxslt are already
198 installed as development packages use those (and omit compilation).
205 gunzip -c libxml2-version.tar.gz|tar xf -
213 gunzip -c libxslt-version.tar.gz|tar xf -
224 gunzip -c yaz-version.tar.gz|tar xf -
232 gunzip -c yazpp-version.tar.gz|tar xf -
243 gunzip -c boost-version.tar.gz|tar xf -
254 gunzip -c metaproxy-version.tar.gz|tar xf -
263 <section id="installation.debian">
264 <title>Installation on Debian GNU/Linux</title>
266 All dependencies for Metaproxy are available as
267 <ulink url="&url.debian;">Debian</ulink>
268 packages for the sarge (stable in 2005) and etch (testing in 2005)
272 The procedures for Debian based systems, such as
273 <ulink url="&url.ubuntu;">Ubuntu</ulink> is probably similar
276 There is currently no official Debian package for YAZ++.
277 And the Debian package for YAZ is probably too old.
278 Update the <filename>/etc/apt/sources.list</filename>
279 to include the Index Data repository.
280 See YAZ' <ulink url="&url.yaz.download.debian;">Download Debian</ulink>
281 for more information.
284 apt-get install libxslt1-dev
285 apt-get install libyazpp-dev
286 apt-get install libboost-dev
287 apt-get install libboost-thread-dev
288 apt-get install libboost-date-time-dev
289 apt-get install libboost-program-options-dev
290 apt-get install libboost-test-dev
293 With these packages installed, the usual configure + make
294 procedure can be used for Metaproxy as outlined in
295 <xref linkend="installation.unix"/>.
299 <section id="installation.windows">
300 <title>Installation on Windows</title>
302 Metaproxy can be compiled with Microsoft
303 <ulink url="&url.vstudio;">Visual Studio</ulink>.
304 Version 2003 (C 7.1) and 2005 (C 8.0) is known to work.
306 <section id="installation.windows.boost">
309 Get Boost from its <ulink url="&url.boost;">home page</ulink>.
310 You also need Boost Jam (an alternative to make).
311 That's also available from the Boost home page.
312 The files to be downloaded are called something like:
313 <filename>boost_1_33-1.exe</filename>
315 <filename>boost-jam-3.1.12-1-ntx86.zip</filename>.
316 Unpack Boost Jam first. Put <filename>bjam.exe</filename>
317 in your system path. Make a command prompt and ensure
318 it can be found automatically. If not check the PATH.
319 The Boost .exe is a self-extracting exe with
320 complete source for Boost. Compile that source with
321 Boost Jam (An alternative to Make).
322 The compilation takes a while.
323 For Visual Studio 2003, use
325 bjam "-sTOOLS=vc-7_1"
327 Here <literal>vc-7_1</literal> refers to a "Toolset" (compiler system).
328 For Visual Studio 2005, use
330 bjam "-sTOOLS=vc-8_0"
332 To install the libraries in a common place, use
334 bjam "-sTOOLS=vc-7_1" install
336 (or vc-8_0 for VS 2005).
339 By default, the Boost build process installs the resulting
340 libraries + header files in
341 <literal>\boost\lib</literal>, <literal>\boost\include</literal>.
344 For more informatation about installing Boost refer to the
345 <ulink url="&url.boost.getting.started;">getting started</ulink>
350 <section id="installation.windows.libxslt">
351 <title>Libxslt</title>
353 <ulink url="&url.libxslt;">Libxslt</ulink> can be downloaded
355 <ulink url="&url.libxml2.download.win32;">here</ulink>.
358 Libxslt has other dependencies, but thes can all be downloaded
359 from the same site. Get the following:
360 iconv, zlib, libxml2, libxslt.
364 <section id="installation.windows.yaz">
367 <ulink url="&url.yaz;">YAZ</ulink> can be downloaded
369 <ulink url="&url.yaz.download.win32;">here</ulink>.
373 <section id="installation.windows.yazplusplus">
376 Get <ulink url="&url.yazplusplus;">YAZ++</ulink> as well.
377 Version 1.0 or later is required. For now get it from
379 <ulink url="&url.snapshot.download;">Snapshot area</ulink>.
382 YAZ++ includes NMAKE makefiles, similar to those found in the
387 <section id="installation.windows.metaproxy">
388 <title>Metaproxy</title>
390 Metaproxy is shipped with NMAKE makfiles as well - similar
391 to those found in the YAZ++/YAZ packages. Adjust this Makefile
392 to point to the proper locations of Boost, Libxslt, Libxml2,
393 zlib, iconv, yaz and yazpp.
397 <varlistentry><term><literal>DEBUG</literal></term>
399 If set to 1, the software is
400 compiled with debugging libraries (code generation is
401 multi-threaded debug DLL).
402 If set to 0, the software is compiled with release libraries
403 (code generation is multi-threaded DLL).
408 <term><literal>BOOST</literal></term>
411 Boost install location
417 <term><literal>BOOST_VERSION</literal></term>
420 Boost version (replace . with _).
426 <term><literal>BOOST_TOOLSET</literal></term>
435 <term><literal>LIBXSLT_DIR</literal>,
436 <literal>LIBXML2_DIR</literal> ..</term>
439 Specify the locations of Libxslt, libiconv, libxml2 and
448 After succesful compilation you'll find
449 <literal>metaproxy.exe</literal> in the
450 <literal>bin</literal> directory.
458 <chapter id="architecture">
459 <title>The Metaproxy Architecture</title>
461 The Metaproxy architecture is based on three concepts:
462 the <emphasis>package</emphasis>,
463 the <emphasis>route</emphasis>
464 and the <emphasis>filter</emphasis>.
468 <term>Packages</term>
471 A package is request or response, encoded in some protocol,
472 issued by a client, making its way through Metaproxy, send to or
473 received from a server, or sent back to the client.
476 The core of a package is the protocol unit - for example, a
477 Z39.50 Init Request or Search Response, or an SRU searchRetrieve
478 URL or Explain Response. In addition to this core, a package
479 also carries some extra information added and used by Metaproxy
483 In general, packages are doctored as they pass through
484 Metaproxy. For example, when the proxy performs authentication
485 and authorisation on a Z39.50 Init request, it removes the
486 authentication credentials from the package so that they are not
487 passed onto the back-end server; and when search-response
488 packages are obtained from multiple servers, they are merged
489 into a single unified package that makes its way back to the
498 Packages make their way through routes, which can be thought of
499 as programs that operate on the package data-type. Each
500 incoming package initially makes its way through a default
501 route, but may be switched to a different route based on various
502 considerations. Routes are made up of sequences of filters (see
511 Filters provide the individual instructions within a route, and
512 effect the necessary transformations on packages. A particular
513 configuration of Metaproxy is essentially a set of filters,
514 described by configuration details and arranged in order in one
515 or more routes. There are many kinds of filter - about a dozen
516 at the time of writing with more appearing all the time - each
517 performing a specific function and configured by different
521 The word ``filter'' is sometimes used rather loosely, in two
522 different ways: it may be used to mean a particular
523 <emphasis>type</emphasis> of filter, as when we speak of ``the
524 auth_simplefilter'' or ``the multi filter''; or it may be used
525 to be a specific <emphasis>instance</emphasis> of a filter
526 within a Metaproxy configuration. For example, a single
527 configuration will often contain multiple instances of the
528 <literal>z3950_client</literal> filter. In
529 operational terms, of these is a separate filter. In practice,
530 context always make it clear which sense of the word ``filter''
534 Extensibility of Metaproxy is primarily through the creation of
535 plugins that provide new filters. The filter API is small and
536 conceptually simple, but there are many details to master. See
538 <link linkend="extensions">extensions</link>.
544 Since packages are created and handled by the system itself, and
545 routes are conceptually simple, most of the remainder of this
546 document concentrates on filters. After a brief overview of the
547 filter types follows, along with some thoughts on possible future
554 <chapter id="filters">
555 <title>Filters</title>
559 <title>Introductory notes</title>
561 It's useful to think of Metaproxy as an interpreter providing a small
562 number of primitives and operations, but operating on a very
563 complex data type, namely the ``package''.
566 A package represents a Z39.50 or SRU/W request (whether for Init,
567 Search, Scan, etc.) together with information about where it came
568 from. Packages are created by front-end filters such as
569 <literal>frontend_net</literal> (see below), which reads them from
570 the network; other front-end filters are possible. They then pass
571 along a route consisting of a sequence of filters, each of which
572 transforms the package and may also have side-effects such as
573 generating logging. Eventually, the route will yield a response,
574 which is sent back to the origin.
577 There are many kinds of filter: some that are defined statically
578 as part of Metaproxy, and others may be provided by third parties
579 and dynamically loaded. They all conform to the same simple API
580 of essentially two methods: <function>configure()</function> is
581 called at startup time, and is passed a DOM tree representing that
582 part of the configuration file that pertains to this filter
583 instance: it is expected to walk that tree extracting relevant
584 information; and <function>process()</function> is called every
585 time the filter has to processes a package.
588 While all filters provide the same API, there are different modes
589 of functionality. Some filters are sources: they create
591 (<literal>frontend_net</literal>);
592 others are sinks: they consume packages and return a result
593 (<literal>z3950_client</literal>,
594 <literal>backend_test</literal>,
595 <literal>http_file</literal>);
596 the others are true filters, that read, process and pass on the
597 packages they are fed
598 (<literal>auth_simple</literal>,
599 <literal>log</literal>,
600 <literal>multi</literal>,
601 <literal>query_rewrite</literal>,
602 <literal>session_shared</literal>,
603 <literal>template</literal>,
604 <literal>virt_db</literal>).
609 <section id="overview.filter.types">
610 <title>Overview of filter types</title>
612 We now briefly consider each of the types of filter supported by
613 the core Metaproxy binary. This overview is intended to give a
614 flavour of the available functionality; more detailed information
615 about each type of filter is included below in
616 <link linkend="filterref"
617 >the reference guide to Metaproxy filters</link>.
620 The filters are here named by the string that is used as the
621 <literal>type</literal> attribute of a
622 <literal><filter></literal> element in the configuration
623 file to request them, with the name of the class that implements
624 them in parentheses. (The classname is not needed for normal
625 configuration and use of Metaproxy; it is useful only to
629 The filters are here listed in alphabetical order:
633 <title><literal>auth_simple</literal>
634 (mp::filter::AuthSimple)</title>
636 Simple authentication and authorisation. The configuration
637 specifies the name of a file that is the user register, which
638 lists <varname>username</varname>:<varname>password</varname>
639 pairs, one per line, colon separated. When a session begins, it
640 is rejected unless username and passsword are supplied, and match
641 a pair in the register. The configuration file may also specific
642 the name of another file that is the target register: this lists
643 lists <varname>username</varname>:<varname>dbname</varname>,<varname>dbname</varname>...
644 sets, one per line, with multiple database names separated by
645 commas. When a search is processed, it is rejected unless the
646 database to be searched is one of those listed as available to
652 <title><literal>backend_test</literal>
653 (mp::filter::Backend_test)</title>
655 A sink that provides dummy responses in the manner of the
656 <literal>yaz-ztest</literal> Z39.50 server. This is useful only
657 for testing. Seriously, you don't need this. Pretend you didn't
658 even read this section.
663 <title><literal>frontend_net</literal>
664 (mp::filter::FrontendNet)</title>
666 A source that accepts Z39.50 connections from a port
667 specified in the configuration, reads protocol units, and
668 feeds them into the next filter in the route. When the result is
669 revceived, it is returned to the original origin.
674 <title><literal>http_file</literal>
675 (mp::filter::HttpFile)</title>
677 A sink that returns the contents of files from the local
678 filesystem in response to HTTP requests. (Yes, Virginia, this
679 does mean that Metaproxy is also a Web-server in its spare time. So
680 far it does not contain either an email-reader or a Lisp
681 interpreter, but that day is surely coming.)
686 <title><literal>log</literal>
687 (mp::filter::Log)</title>
689 Writes logging information to standard output, and passes on
690 the package unchanged.
695 <title><literal>multi</literal>
696 (mp::filter::Multi)</title>
698 Performs multi-database searching.
700 <link linkend="multidb">the extended discussion</link>
701 of virtual databases and multi-database searching below.
706 <title><literal>query_rewrite</literal>
707 (mp::filter::QueryRewrite)</title>
709 Rewrites Z39.50 Type-1 and Type-101 (``RPN'') queries by a
710 three-step process: the query is transliterated from Z39.50
711 packet structures into an XML representation; that XML
712 representation is transformed by an XSLT stylesheet; and the
713 resulting XML is transliterated back into the Z39.50 packet
719 <title><literal>session_shared</literal>
720 (mp::filter::SessionShared)</title>
722 When this is finished, it will implement global sharing of
723 result sets (i.e. between threads and therefore between
724 clients), yielding performance improvements especially when
725 incoming requests are from a stateless environment such as a
726 web-server, in which the client process representing a session
727 might be any one of many. However:
731 This filter is not yet completed.
737 <title><literal>template</literal>
738 (mp::filter::Template)</title>
740 Does nothing at all, merely passing the packet on. (Maybe it
741 should be called <literal>nop</literal> or
742 <literal>passthrough</literal>?) This exists not to be used, but
743 to be copied - to become the skeleton of new filters as they are
744 written. As with <literal>backend_test</literal>, this is not
745 intended for civilians.
750 <title><literal>virt_db</literal>
751 (mp::filter::Virt_db)</title>
753 Performs virtual database selection: based on the name of the
754 database in the search request, a server is selected, and its
755 address added to the request in a <literal>VAL_PROXY</literal>
756 otherInfo packet. It will subsequently be used by a
757 <literal>z3950_client</literal> filter.
759 <link linkend="multidb">the extended discussion</link>
760 of virtual databases and multi-database searching below.
765 <title><literal>z3950_client</literal>
766 (mp::filter::Z3950Client)</title>
768 Performs Z39.50 searching and retrieval by proxying the
769 packages that are passed to it. Init requests are sent to the
770 address specified in the <literal>VAL_PROXY</literal> otherInfo
771 attached to the request: this may have been specified by client,
772 or generated by a <literal>virt_db</literal> filter earlier in
773 the route. Subsequent requests are sent to the same address,
774 which is remembered at Init time in a Session object.
780 <section id="future.directions">
781 <title>Future directions</title>
783 Some other filters that do not yet exist, but which would be
784 useful, are briefly described. These may be added in future
785 releases (or may be created by third parties, as loadable
791 <term><literal>frontend_cli</literal> (source)</term>
794 Command-line interface for generating requests.
799 <term><literal>frontend_sru</literal> (source)</term>
802 Receive SRU (and perhaps SRW) requests.
807 <term><literal>sru2z3950</literal> (filter)</term>
810 Translate SRU requests into Z39.50 requests.
815 <term><literal>sru_client</literal> (sink)</term>
818 SRU searching and retrieval.
823 <term><literal>srw_client</literal> (sink)</term>
826 SRW searching and retrieval.
831 <term><literal>opensearch_client</literal> (sink)</term>
834 A9 OpenSearch searching and retrieval.
844 <chapter id="configuration">
845 <title>Configuration: the Metaproxy configuration file format</title>
849 <title>Introductory notes</title>
851 If Metaproxy is an interpreter providing operations on packages, then
852 its configuration file can be thought of as a program for that
853 interpreter. Configuration is by means of a single file, the name
854 of which is supplied as the sole command-line argument to the
855 <command>metaproxy</command> program. (See
856 <link linkend="progref">the reference guide</link>
857 below for more information on invoking Metaproxy.)
860 The configuration files are written in XML. (But that's just an
861 implementation detail - they could just as well have been written
862 in YAML or Lisp-like S-expressions, or in a custom syntax.)
865 Since XML has been chosen, an XML schema,
866 <filename>config.xsd</filename>, is provided for validating
867 configuration files. This file is supplied in the
868 <filename>etc</filename> directory of the Metaproxy distribution. It
869 can be used by (among other tools) the <command>xmllint</command>
870 program supplied as part of the <literal>libxml2</literal>
874 xmllint --noout --schema etc/config.xsd my-config-file.xml
877 (A recent version of <literal>libxml2</literal> is required, as
878 support for XML Schemas is a relatively recent addition.)
882 <section id="overview.xml.structure">
883 <title>Overview of XML structure</title>
885 All elements and attributes are in the namespace
886 <ulink url="http://indexdata.dk/yp2/config/1"/>.
887 This is most easily achieved by setting the default namespace on
888 the top-level element, as here:
891 <yp2 xmlns="http://indexdata.dk/yp2/config/1">
894 The top-level element is <yp2>. This contains a
895 <start> element, a <filters> element and a
896 <routes> element, in that order. <filters> is
897 optional; the other two are mandatory. All three are
901 The <start> element is empty, but carries a
902 <literal>route</literal> attribute, whose value is the name of
903 route at which to start running - analogous to the name of the
904 start production in a formal grammar.
907 If present, <filters> contains zero or more <filter>
908 elements. Each filter carries a <literal>type</literal> attribute
909 which specifies what kind of filter is being defined
910 (<literal>frontend_net</literal>, <literal>log</literal>, etc.)
911 and contain various elements that provide suitable configuration
912 for a filter of its type. The filter-specific elements are
914 <link linkend="filterref">the reference guide below</link>.
915 Filters defined in this part of the file must carry an
916 <literal>id</literal> attribute so that they can be referenced
920 <routes> contains one or more <route> elements, each
921 of which must carry an <literal>id</literal> element. One of the
922 routes must have the ID value that was specified as the start
923 route in the <start> element's <literal>route</literal>
924 attribute. Each route contains zero or more <filter>
925 elements. These are of two types. They may be empty, but carry a
926 <literal>refid</literal> attribute whose value is the same as the
927 <literal>id</literal> of a filter previously defined in the
928 <filters> section. Alternatively, a route within a filter
929 may omit the <literal>refid</literal> attribute, but contain
930 configuration elements similar to those used for filters defined
931 in the <filters> section. (In other words, each filter in a
932 route may be included either by reference or by physical
938 <section id="example.configuration">
939 <title>An example configuration</title>
941 The following is a small, but complete, Metaproxy configuration
942 file (included in the distribution as
943 <literal>metaproxy/etc/config0.xml</literal>).
944 This file defines a very simple configuration that simply proxies
945 to whatever back-end server the client requests, but logs each
946 request and response. This can be useful for debugging complex
947 client-server dialogues.
949 <screen><![CDATA[<?xml version="1.0"?>
950 <yp2 xmlns="http://indexdata.dk/yp2/config/1">
951 <start route="start"/>
953 <filter id="frontend" type="frontend_net">
956 <filter id="backend" type="z3950_client">
961 <filter refid="frontend"/>
963 <filter refid="backend"/>
969 It works by defining a single route, called
970 <literal>start</literal>, which consists of a sequence of three
971 filters. The first and last of these are included by reference:
972 their <literal><filter></literal> elements have
973 <literal>refid</literal> attributes that refer to filters defined
974 within the prior <literal><filters></literal> section. The
975 middle filter is included inline in the route.
978 The three filters in the route are as follows: first, a
979 <literal>frontend_net</literal> filter accepts Z39.50 requests
980 from any host on port 9000; then these requests are passed through
981 a <literal>log</literal> filter that emits a message for each
982 request; they are then fed into a <literal>z3950_client</literal>
983 filter, which forwards the requests to the client-specified
984 back-end Z39.509 server. When the response arrives, it is handed
985 back to the <literal>log</literal> filter, which emits another
986 message; and then to the front-end filter, which returns the
987 response to the client.
994 <chapter id="multidb">
995 <title>Virtual databases and multi-database searching</title>
999 <title>Introductory notes</title>
1001 Two of Metaproxy's filters are concerned with multiple-database
1002 operations. Of these, <literal>virt_db</literal> can work alone
1003 to control the routing of searches to one of a number of servers,
1004 while <literal>multi</literal> can work together with
1005 <literal>virt_db</literal> to perform multi-database searching, merging
1006 the results into a unified result-set - ``metasearch in a box''.
1009 The interaction between
1010 these two filters is necessarily complex: it reflects the real,
1011 irreducible complexity of multi-database searching in a protocol such
1012 as Z39.50 that separates initialisation from searching, and in
1013 which the database to be searched is not known at initialisation
1017 It's possible to use these filters without understanding the
1018 details of their functioning and the interaction between them; the
1019 next two sections of this chapter are ``HOWTO'' guides for doing
1020 just that. However, debugging complex configurations will require
1021 a deeper understanding, which the last two sections of this
1022 chapters attempt to provide.
1027 <section id="multidb.virt_db">
1028 <title>Virtual databases with the <literal>virt_db</literal> filter</title>
1030 Working alone, the purpose of the
1031 <literal>virt_db</literal>
1032 filter is to route search requests to one of a selection of
1033 back-end databases. In this way, a single Z39.50 endpoint
1034 (running Metaproxy) can provide access to several different
1035 underlying services, including those that would otherwise be
1036 inaccessible due to firewalls. In many useful configurations, the
1037 back-end databases are local to the Metaproxy installation, but
1038 the software does not enforce this, and any valid Z39.50 servers
1039 may be used as back-ends.
1042 For example, a <literal>virt_db</literal>
1043 filter could be set up so that searches in the virtual database
1044 ``lc'' are forwarded to the Library of Congress bibliographic
1045 catalogue server, and searches in the virtual database ``marc''
1046 are forwarded to the toy database of MARC records that Index Data
1047 hosts for testing purposes. A <literal>virt_db</literal>
1048 configuration to make this switch would look like this:
1050 <screen><![CDATA[<filter type="virt_db">
1052 <database>lc</database>
1053 <target>z3950.loc.gov:7090/voyager</target>
1056 <database>marc</database>
1057 <target>indexdata.dk/marc</target>
1059 </filter>]]></screen>
1061 As well as being useful in it own right, this filter also provides
1062 the foundation for multi-database searching.
1067 <section id="multidb.multi">
1068 <title>Multi-database search with the <literal>multi</literal> filter</title>
1070 To arrange for Metaproxy to broadcast searches to multiple back-end
1071 servers, the configuration needs to include two components: a
1072 <literal>virt_db</literal>
1073 filter that specifies multiple
1074 <literal><target></literal>
1075 elements, and a subsequent
1076 <literal>multi</literal>
1077 filter. Here, for example, is a complete configuration that
1078 broadcasts searches to both the Library of Congress catalogue and
1079 Index Data's tiny testing database of MARC records:
1081 <screen><![CDATA[<?xml version="1.0"?>
1082 <yp2 xmlns="http://indexdata.dk/yp2/config/1">
1083 <start route="start"/>
1086 <filter type="frontend_net">
1087 <threads>10</threads>
1090 <filter type="virt_db">
1092 <database>lc</database>
1093 <target>z3950.loc.gov:7090/voyager</target>
1096 <database>marc</database>
1097 <target>indexdata.dk/marc</target>
1100 <database>all</database>
1101 <target>z3950.loc.gov:7090/voyager</target>
1102 <target>indexdata.dk/marc</target>
1105 <filter type="multi"/>
1106 <filter type="z3950_client">
1107 <timeout>30</timeout>
1114 <literal>virt_db</literal>
1115 filter that specifies multiple
1116 <literal><target></literal>
1117 elements but without a subsequent
1118 <literal>multi</literal>
1119 filter yields surprising and undesirable results, as will be
1120 described below. Don't do that.)
1123 Metaproxy can be invoked with this configuration as follows:
1125 <screen>../src/metaproxy --config config-simple-multi.xml</screen>
1127 And thereafter, Z39.50 clients can connect to the running server
1128 (on port 9000, as specified in the configuration) and search in
1129 any of the databases
1130 <literal>lc</literal> (the Library of Congress catalogue),
1131 <literal>marc</literal> (Index Data's test database of MARC records)
1133 <literal>all</literal> (both of these). As an example, a session
1134 using the YAZ command-line client <literal>yaz-client</literal> is
1135 here included (edited for brevity and clarity):
1137 <screen><![CDATA[$ yaz-client @:9000
1141 Search was a success.
1142 Number of hits: 10000, setno 1
1146 Search was a success.
1147 Number of hits: 10, setno 3
1151 Search was a success.
1152 Number of hits: 10010, setno 4
1155 [marc]Record type: USmarc
1158 005 00000000000000.0
1159 008 910710c19910701nju 00010 eng
1163 100 10 $a Jack Collins
1164 245 10 $a How to program a computer
1170 [VOYAGER]Record type: USmarc
1172 005 20041229102447.0
1173 008 030910s2004 caua 000 0 eng
1174 035 $a (DLC) 2003112666
1175 906 $a 7 $b cbc $c orignew $d 4 $e epcn $f 20 $g y-gencatlg
1176 925 0 $a acquire $b 1 shelf copy $x policy default
1177 955 $a pc10 2003-09-10 $a pv12 2004-06-23 to SSCD; $h sj05 2004-11-30 $e sj05 2004-11-30 to Shelf.
1180 040 $a DLC $c DLC $d DLC
1181 050 00 $a MLCM 2004/03312 (G)
1182 245 10 $a 007, everything or nothing : $b Prima's official strategy guide / $c created by Kaizen Media Group.
1183 246 3 $a Double-O-seven, everything or nothing
1184 246 30 $a Prima's official strategy guide
1185 260 $a Roseville, CA : $b Prima Games, $c c2004.
1186 300 $a 161 p. : $b col. ill. ; $c 28 cm.
1187 500 $a "Platforms: Nintendo GameCube, Macintosh, PC, PlayStation 2 computer entertainment system, Xbox"--P. [4] of cover.
1188 650 0 $a Video games.
1189 710 2 $a Kaizen Media Group.
1190 856 42 $3 Publisher description $u http://www.loc.gov/catdir/description/random052/2003112666.html
1195 As can be seen, the first record in the result set is from the
1196 Index Data test database, and the second from the Library of
1197 Congress database. The result-set continues alternating records
1198 round-robin style until the point where one of the databases'
1199 records are exhausted.
1202 This example uses only two back-end databases; more may be used.
1203 There is no limitation imposed on the number of databases that may
1204 be metasearched in this way: issues of resource usage and
1205 administrative complexity dictate the practical limits.
1208 What happens when one of the databases doesn't respond? By default,
1209 the entire multi-database search fails, and the appropriate
1210 diagnostic is returned to the client. This is usually appropriate
1211 during development, when technicians need maximum information, but
1212 can be inconvenient in deployment, when users typically don't want
1213 to be bothered with problems of this kind and prefer just to get
1214 the records from the databases that are available. To obtain this
1215 latter behaviour add an empty
1216 <literal><hideunavailable></literal>
1218 <literal>multi</literal> filter:
1220 <screen><![CDATA[ <filter type="multi">
1222 </filter>]]></screen>
1224 Under this regime, an error is reported to the client only if
1225 <emphasis>all</emphasis> the databases in a multi-database search
1231 <section id="multidb.what">
1232 <title>What's going on?</title>
1234 <title>Lark's vomit</title>
1236 This section goes into a level of technical detail that is
1237 probably not necessary in order to configure and use Metaproxy.
1238 It is provided only for those who like to know how things work.
1239 You should feel free to skip on to the next section if this one
1240 doesn't seem like fun.
1244 Hold on tight - this may get a little hairy.
1247 In the general course of things, a Z39.50 Init request may carry
1248 with it an otherInfo packet of type <literal>VAL_PROXY</literal>,
1249 whose value indicates the address of a Z39.50 server to which the
1250 ultimate connection is to be made. (This otherInfo packet is
1251 supported by YAZ-based Z39.50 clients and servers, but has not yet
1252 been ratified by the Maintenance Agency and so is not widely used
1253 in non-Index Data software. We're working on it.)
1254 The <literal>VAL_PROXY</literal> packet functions
1255 analogously to the absoluteURI-style Request-URI used with the GET
1256 method when a web browser asks a proxy to forward its request: see
1258 <ulink url="http://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html#sec5.1.2"
1259 >Request-URI</ulink>
1261 <ulink url="http://www.w3.org/Protocols/rfc2616/rfc2616.html"
1262 >the HTTP 1.1 specification</ulink>.
1265 Within Metaproxy, Search requests that are part of the same
1266 session as an Init request that carries a
1267 <literal>VAL_PROXY</literal> otherInfo are also annotated with the
1268 same information. The role of the <literal>virt_db</literal>
1269 filter is to rewrite this otherInfo packet dependent on the
1270 virtual database that the client wants to search.
1273 When Metaproxy receives a Z39.50 Init request from a client, it
1274 doesn't immediately forward that request to the back-end server.
1275 Why not? Because it doesn't know <emphasis>which</emphasis>
1276 back-end server to forward it to until the client sends a Search
1277 request that specifies the database that it wants to search in.
1278 Instead, it just treasures the Init request up in its heart; and,
1279 later, the first time the client does a search on one of the
1280 specified virtual databases, a connection is forged to the
1281 appropriate server and the Init request is forwarded to it. If,
1282 later in the session, the same client searches in a different
1283 virtual database, then a connection is forged to the server that
1284 hosts it, and the same cached Init request is forwarded there,
1288 All of this clever Init-delaying is done by the
1289 <literal>frontend_net</literal> filter. The
1290 <literal>virt_db</literal> filter knows nothing about it; in
1291 fact, because the Init request that is received from the client
1292 doesn't get forwarded until a Search request is received, the
1293 <literal>virt_db</literal> filter (and the
1294 <literal>z3950_client</literal> filter behind it) doesn't even get
1295 invoked at Init time. The <emphasis>only</emphasis> thing that a
1296 <literal>virt_db</literal> filter ever does is rewrite the
1297 <literal>VAL_PROXY</literal> otherInfo in the requests that pass
1301 It is possible for a <literal>virt_db</literal> filter to contain
1303 <literal><target></literal>
1304 elements. What does this mean? Only that the filter will add
1305 multiple <literal>VAL_PROXY</literal> otherInfo packets to the
1306 Search requests that pass through it. That's because the virtual
1307 DB filter is dumb, and does exactly what it's told - no more, no
1309 If a Search request with multiple <literal>VAL_PROXY</literal>
1310 otherInfo packets reaches a <literal>z3950_client</literal>
1311 filter, this is an error. That filter doesn't know how to deal
1312 with multiple targets, so it will either just pick one and search
1313 in it, or (better) fail with an error message.
1316 The <literal>multi</literal> filter comes to the rescue! This is
1317 the only filter that knows how to deal with multiple
1318 <literal>VAL_PROXY</literal> otherInfo packets, and it does so by
1319 making multiple copies of the entire Search request: one for each
1320 <literal>VAL_PROXY</literal>. Each of these new copies is then
1321 passed down through the remaining filters in the route. (The
1322 copies are handled in parallel though the
1323 spawning of new threads.) Since the copies each have only one
1324 <literal>VAL_PROXY</literal> otherInfo, they can be handled by the
1325 <literal>z3950_client</literal> filter, which happily deals with
1326 each one individually. When the results of the individual
1327 searches come back up to the <literal>multi</literal> filter, it
1328 merges them into a single Search response, which is what
1329 eventually makes it back to the client.
1334 <section id="multidb.picture">
1335 <title>A picture is worth a thousand words (but only five hundred on 64-bit architectures)</title>
1339 <imagedata fileref="multi.pdf" format="PDF" scale="50"/>
1342 <imagedata fileref="multi.png" format="PNG"/>
1345 <!-- Fall back if none of the images can be used -->
1347 [Here there should be a diagram showing the progress of
1348 packages through the filters during a simple virtual-database
1349 search and a multi-database search, but is seems that your
1350 toolchain has not been able to include the diagram in this
1351 document. This is because of LaTeX suckage. Time to move to
1352 OpenOffice. Yes, really.]
1355 <!-- ### This used to work with an older version of DocBook
1357 <para>Caption: progress of packages through filters.</para>
1360 </inlinemediaobject>
1367 <chapter id="extensions">
1368 <title>Writing extensions for Metaproxy</title>
1369 <para>### To be written</para>
1375 <chapter id="classes">
1376 <title>Classes in the Metaproxy source code</title>
1380 <title>Introductory notes</title>
1382 <emphasis>Stop! Do not read this!</emphasis>
1383 You won't enjoy it at all. You should just skip ahead to
1384 <link linkend="refguide">the reference guide</link>,
1386 <!-- The remainder of this paragraph is lifted verbatim from
1387 Douglas Adams' _Hitch Hiker's Guide to the Galaxy_, chapter 8 -->
1388 you things you really need to know, like the fact that the
1389 fabulously beautiful planet Bethselamin is now so worried about
1390 the cumulative erosion by ten billion visiting tourists a year
1391 that any net imbalance between the amount you eat and the amount
1392 you excrete whilst on the planet is surgically removed from your
1393 bodyweight when you leave: so every time you go to the lavatory it
1394 is vitally important to get a receipt.
1397 This chapter contains documentation of the Metaproxy source code, and is
1398 of interest only to maintainers and developers. If you need to
1399 change Metaproxy's behaviour or write a new filter, then you will most
1400 likely find this chapter helpful. Otherwise it's a waste of your
1401 good time. Seriously: go and watch a film or something.
1402 <citetitle>This is Spinal Tap</citetitle> is particularly good.
1405 Still here? OK, let's continue.
1408 In general, classes seem to be named big-endianly, so that
1409 <literal>FactoryFilter</literal> is not a filter that filters
1410 factories, but a factory that produces filters; and
1411 <literal>FactoryStatic</literal> is a factory for the statically
1412 registered filters (as opposed to those that are dynamically
1417 <section id="individual.classes">
1418 <title>Individual classes</title>
1420 The classes making up the Metaproxy application are here listed by
1421 class-name, with the names of the source files that define them in
1426 <title><literal>mp::FactoryFilter</literal>
1427 (<filename>factory_filter.cpp</filename>)</title>
1429 A factory class that exists primarily to provide the
1430 <literal>create()</literal> method, which takes the name of a
1431 filter class as its argument and returns a new filter of that
1432 type. To enable this, the factory must first be populated by
1433 calling <literal>add_creator()</literal> for static filters (this
1434 is done by the <literal>FactoryStatic</literal> class, see below)
1435 and <literal>add_creator_dyn()</literal> for filters loaded
1441 <title><literal>mp::FactoryStatic</literal>
1442 (<filename>factory_static.cpp</filename>)</title>
1444 A subclass of <literal>FactoryFilter</literal> which is
1445 responsible for registering all the statically defined filter
1446 types. It does this by knowing about all those filters'
1447 structures, which are listed in its constructor. Merely
1448 instantiating this class registers all the static classes. It is
1449 for the benefit of this class that <literal>struct
1450 metaproxy_1_filter_struct</literal> exists, and that all the filter
1451 classes provide a static object of that type.
1456 <title><literal>mp::filter::Base</literal>
1457 (<filename>filter.cpp</filename>)</title>
1459 The virtual base class of all filters. The filter API is, on the
1460 surface at least, extremely simple: two methods.
1461 <literal>configure()</literal> is passed a DOM tree representing
1462 that part of the configuration file that pertains to this filter
1463 instance, and is expected to walk that tree extracting relevant
1464 information. And <literal>process()</literal> processes a
1465 package (see below). That surface simplicitly is a bit
1466 misleading, as <literal>process()</literal> needs to know a lot
1467 about the <literal>Package</literal> class in order to do
1473 <title><literal>mp::filter::AuthSimple</literal>,
1474 <literal>Backend_test</literal>, etc.
1475 (<filename>filter_auth_simple.cpp</filename>,
1476 <filename>filter_backend_test.cpp</filename>, etc.)</title>
1478 Individual filters. Each of these is implemented by a header and
1479 a source file, named <filename>filter_*.hpp</filename> and
1480 <filename>filter_*.cpp</filename> respectively. All the header
1481 files should be pretty much identical, in that they declare the
1482 class, including a private <literal>Rep</literal> class and a
1483 member pointer to it, and the two public methods. The only extra
1484 information in any filter header is additional private types and
1485 members (which should really all be in the <literal>Rep</literal>
1486 anyway) and private methods (which should also remain known only
1487 to the source file, but C++'s brain-damaged design requires this
1488 dirty laundry to be exhibited in public. Thanks, Bjarne!)
1491 The source file for each filter needs to supply:
1496 A definition of the private <literal>Rep</literal> class.
1501 Some boilerplate constructors and destructors.
1506 A <literal>configure()</literal> method that uses the
1507 appropriate XML fragment.
1512 Most important, the <literal>process()</literal> method that
1513 does all the actual work.
1520 <title><literal>mp::Package</literal>
1521 (<filename>package.cpp</filename>)</title>
1523 Represents a package on its way through the series of filters
1524 that make up a route. This is essentially a Z39.50 or SRU APDU
1525 together with information about where it came from, which is
1526 modified as it passes through the various filters.
1531 <title><literal>mp::Pipe</literal>
1532 (<filename>pipe.cpp</filename>)</title>
1534 This class provides a compatibility layer so that we have an IPC
1535 mechanism that works the same under Unix and Windows. It's not
1536 particularly exciting.
1541 <title><literal>mp::RouterChain</literal>
1542 (<filename>router_chain.cpp</filename>)</title>
1549 <title><literal>mp::RouterFleXML</literal>
1550 (<filename>router_flexml.cpp</filename>)</title>
1557 <title><literal>mp::Session</literal>
1558 (<filename>session.cpp</filename>)</title>
1565 <title><literal>mp::ThreadPoolSocketObserver</literal>
1566 (<filename>thread_pool_observer.cpp</filename>)</title>
1573 <title><literal>mp::util</literal>
1574 (<filename>util.cpp</filename>)</title>
1576 A namespace of various small utility functions and classes,
1577 collected together for convenience. Most importantly, includes
1578 the <literal>mp::util::odr</literal> class, a wrapper for YAZ's
1584 <title><literal>mp::xml</literal>
1585 (<filename>xmlutil.cpp</filename>)</title>
1587 A namespace of various XML utility functions and classes,
1588 collected together for convenience.
1594 <section id="other.source.files">
1595 <title>Other Source Files</title>
1597 In addition to the Metaproxy source files that define the classes
1598 described above, there are a few additional files which are
1599 briefly described here:
1603 <term><literal>metaproxy_prog.cpp</literal></term>
1606 The main function of the <command>metaproxy</command> program.
1611 <term><literal>ex_router_flexml.cpp</literal></term>
1614 Identical to <literal>metaproxy_prog.cpp</literal>: it's not clear why.
1619 <term><literal>test_*.cpp</literal></term>
1622 Unit-tests for various modules.
1628 ### Still to be described:
1629 <literal>ex_filter_frontend_net.cpp</literal>,
1630 <literal>filter_dl.cpp</literal>,
1631 <literal>plainfile.cpp</literal>,
1632 <literal>tstdl.cpp</literal>.
1639 <chapter id="refguide">
1640 <title>Reference guide</title>
1642 The material in this chapter is drawn directly from the individual
1643 manual entries. In particular, the Metaproxy invocation section is
1644 available using <command>man metaproxy</command>, and the section
1645 on each individual filter is available using the name of the filter
1646 as the argument to the <command>man</command> command.
1650 <section id="progref">
1651 <title>Metaproxy invocation</title>
1656 <section id="filterref">
1657 <title>Reference guide to Metaproxy filters</title>
1663 <!-- Keep this comment at the end of the file
1668 sgml-minimize-attributes:nil
1669 sgml-always-quote-attributes:t
1672 sgml-parent-document: nil
1673 sgml-local-catalogs: nil
1674 sgml-namecase-general:t