1 <!-- $Id: book.xml,v 1.27 2006-04-27 19:49:35 adam Exp $ -->
3 <title>Metaproxy - User's Guide and Reference</title>
5 <firstname>Mike</firstname><surname>Taylor</surname>
8 <firstname>Adam</firstname><surname>Dickmeiss</surname>
12 <holder>Index Data ApS</holder>
16 Metaproxy is a universal router, proxy and encapsulated
17 metasearcher for information retrieval protocols. It accepts,
18 processes, interprets and redirects requests from IR clients using
19 standard protocols such as
20 <ulink url="&url.z39.50;">ANSI/NISO Z39.50</ulink>
21 (and in the future <ulink url="&url.sru;">SRU</ulink>
22 and <ulink url="&url.srw;">SRW</ulink>), as
23 well as functioning as a limited
24 <ulink url="&url.http;">HTTP</ulink> server.
25 Metaproxy is configured by an XML file which
26 specifies how the software should function in terms of routes that
27 the request packets can take through the proxy, each step on a
28 route being an instantiation of a filter. Filters come in many
29 types, one for each operation: accepting Z39.50 packets, logging,
30 query transformation, multiplexing, etc. Further filter-types can
31 be added as loadable modules to extend Metaproxy functionality,
35 The terms under which Metaproxy will be distributed have yet to be
36 established, but it will not necessarily be open source; so users
37 should not at this stage redistribute the code without explicit
38 written permission from the copyright holders, Index Data ApS.
43 <imagedata fileref="common/id.png" format="PNG"/>
46 <imagedata fileref="common/id.eps" format="EPS"/>
53 <chapter id="introduction">
54 <title>Introduction</title>
58 <ulink url="&url.metaproxy;">Metaproxy</ulink>
59 is a standalone program that acts as a universal router, proxy and
60 encapsulated metasearcher for information retrieval protocols such
61 as <ulink url="&url.z39.50;">Z39.50</ulink>, and in the future
62 <ulink url="&url.sru;">SRU</ulink> and <ulink url="&url.srw;">SRW</ulink>.
63 To clients, it acts as a server of these protocols: it can be searched,
64 records can be retrieved from it, etc.
65 To servers, it acts as a client: it searches in them,
66 retrieves records from them, etc. it satisfies its clients'
67 requests by transforming them, multiplexing them, forwarding them
68 on to zero or more servers, merging the results, transforming
69 them, and delivering them back to the client. In addition, it
70 acts as a simple <ulink url="&url.http;">HTTP</ulink> server; support
71 for further protocols can be added in a modular fashion, through the
72 creation of new filters.
77 Fish, bananas, cold pyjamas,
78 Mutton, beef and trout!
79 - attributed to Cole Porter.
82 Metaproxy is a more capable alternative to
83 <ulink url="&url.yazproxy;">YAZ Proxy</ulink>,
84 being more powerful, flexible, configurable and extensible. Among
85 its many advantages over the older, more pedestrian work are
86 support for multiplexing (encapsulated metasearching), routing by
87 database name, authentication and authorisation and serving local
88 files via HTTP. Equally significant, its modular architecture
89 facilitites the creation of pluggable modules implementing further
93 This manual will briefly describe Metaproxy's licensing situation
94 before giving an overview of its architecture, then discussing the
95 key concept of a filter in some depth and giving an overview of
96 the various filter types, then discussing the configuration file
97 format. After this come several optional chapters which may be
98 freely skipped: a detailed discussion of virtual databases and
99 multi-database searching, some notes on writing extensions
100 (additional filter types) and a high-level description of the
101 source code. Finally comes the reference guide, which contains
102 instructions for invoking the <command>metaproxy</command>
103 program, and detailed information on each type of filter,
110 <chapter id="licence">
111 <title>The Metaproxy Licence</title>
113 <emphasis role="strong">
114 No decision has yet been made on the terms under which
115 Metaproxy will be distributed.
117 It is possible that, unlike
118 other Index Data products, metaproxy may not be released under a
119 free-software licence such as the GNU GPL. Until a decision is
120 made and a public statement made, then, and unless it has been
121 delivered to you other specific terms, please treat Metaproxy as
122 though it were proprietary software.
123 The code should not be redistributed without explicit
124 written permission from the copyright holders, Index Data ApS.
128 <chapter id="installation">
129 <title>Installation</title>
131 Metaproxy depends on the following tools/libraries:
133 <varlistentry><term><ulink url="&url.yazplusplus;">YAZ++</ulink></term>
136 This is a C++ library based on <ulink url="&url.yaz;">YAZ</ulink>.
140 <varlistentry><term><ulink url="&url.libxslt;">Libxslt</ulink></term>
142 <para>This is an XSLT processor - based on
143 <ulink url="&url.libxml2;">Libxml2</ulink>. Both Libxml2 and
144 Libxslt must be installed with the development components
145 (header files, etc.) as well as the run-time libraries.
149 <varlistentry><term><ulink url="&url.boost;">Boost</ulink></term>
152 The popular C++ library. Initial versions of Metaproxy
153 was built with 1.33.0. Version 1.33.1 works too.
160 In order to compile Metaproxy a modern C++ compiler is
161 required. Boost, in particular, requires the C++ compiler
162 to facilitate the newest features. Refer to Boost
163 <ulink url="&url.boost.compilers.status;">Compiler Status</ulink>
164 for more information.
167 We have succesfully built Metaproxy using the compilers
168 <ulink url="&url.gcc;">GCC</ulink> version 4.0 and
169 <ulink url="&url.vstudio;">Microsoft Visual Studio</ulink> 2003/2005.
172 <section id="installation.unix">
173 <title>Installation on Unix (from Source)</title>
175 Here is a quick step-by-step guide on how to compile all the
176 tools that Metaproxy uses. Only few systems have none of the required
177 tools binary packages. If, for example, Libxml2/libxslt are already
178 installed as development packages use those (and omit compilation).
185 gunzip -c libxml2-version.tar.gz|tar xf -
193 gunzip -c libxslt-version.tar.gz|tar xf -
204 gunzip -c yaz-version.tar.gz|tar xf -
212 gunzip -c yazpp-version.tar.gz|tar xf -
223 gunzip -c boost-version.tar.gz|tar xf -
234 gunzip -c metaproxy-version.tar.gz|tar xf -
243 <section id="installation.debian">
244 <title>Installation on Debian GNU/Linux</title>
246 All dependencies for Metaproxy are available as
247 <ulink url="&url.debian;">Debian</ulink>
248 packages for the sarge (stable in 2005) and etch (testing in 2005)
252 The procedures for Debian based systems, such as
253 <ulink url="&url.ubuntu;">Ubuntu</ulink> is probably similar
256 There is currently no official Debian package for YAZ++.
257 And the Debian package for YAZ is probably too old.
258 Update the <filename>/etc/apt/sources.list</filename>
259 to include the Index Data repository.
260 See YAZ' <ulink url="&url.yaz.download.debian;">Download Debian</ulink>
261 for more information.
264 apt-get install libxslt1-dev
265 apt-get install libyazpp-dev
266 apt-get install libboost-dev
267 apt-get install libboost-thread-dev
268 apt-get install libboost-date-time-dev
269 apt-get install libboost-program-options-dev
270 apt-get install libboost-test-dev
273 With these packages installed, the usual configure + make
274 procedure can be used for Metaproxy as outlined in
275 <xref linkend="installation.unix"/>.
279 <section id="installation.windows">
280 <title>Installation on Windows</title>
282 Compilation of Metaproxy can be done using
283 Microsoft <ulink url="&url.vstudio;">Visual Studio</ulink>.
284 We know Version 2003 works. We expect Version 2005 to
287 <section id="installation.windows.boost">
290 Get Boost from its <ulink url="&url.boost;">home page</ulink>.
291 You also need Boost Jam (an alternative to make).
292 That's also available from the Boost home page.
293 The files to be downloaded are called something like:
294 <filename>boost_1_33-1.exe</filename>
296 <filename>boost-jam-3.1.12-1-ntx86.zip</filename>.
297 Unpack Boost Jam first. Put <filename>bjam.exe</filename>
298 in your system path. Make a command prompt and ensure
299 it can be found automatically. If not check the PATH.
300 The Boost .exe is a self-extracting exe with
301 complete source for Boost. Compile that source with
302 Boost Jam (An alternative to Make).
303 The compilation takes a while.
304 By default, the Boost build process puts the resulting
305 libraries + header files in
306 <literal>\boost\lib</literal>, <literal>\boost\include</literal>.
309 For more informatation about installing Boost refer to the
310 <ulink url="&url.boost.getting.started;">getting started</ulink>
315 <section id="installation.windows.libxslt">
316 <title>Libxslt</title>
318 <ulink url="&url.libxslt;">Libxslt</ulink> can be downloaded
320 <ulink url="&url.libxml2.download.win32;">here</ulink>.
323 Libxslt has other dependencies, but thes can all be downloaded
324 from the same site. Get the following:
325 iconv, zlib, libxml2, libxslt.
329 <section id="installation.windows.yaz">
332 <ulink url="&url.yaz;">YAZ</ulink> can be downloaded
334 <ulink url="&url.yaz.download.win32;">here</ulink>.
338 <section id="installation.windows.yazplusplus">
341 Get <ulink url="&url.yazplusplus;">YAZ++</ulink> as well.
342 Version 1.0 or later is required. For now get it from
344 <ulink url="&url.snapshot.download;">Snapshot area</ulink>.
347 YAZ++ includes NMAKE makefiles, similar to those found in the
352 <section id="installation.windows.metaproxy">
353 <title>Metaproxy</title>
355 Metaproxy is shipped with NMAKE makfiles as well - similar
356 to those found in the YAZ++/YAZ packages. Adjust this Makefile
357 to point to the proper locations of Boost, Libxslt, Libxml2,
358 zlib, iconv, yaz and yazpp.
361 After succesful compilation you'll find
362 <literal>metaproxy.exe</literal> in the
363 <literal>bin</literal> directory.
370 <chapter id="architecture">
371 <title>The Metaproxy Architecture</title>
373 The Metaproxy architecture is based on three concepts:
374 the <emphasis>package</emphasis>,
375 the <emphasis>route</emphasis>
376 and the <emphasis>filter</emphasis>.
380 <term>Packages</term>
383 A package is request or response, encoded in some protocol,
384 issued by a client, making its way through Metaproxy, send to or
385 received from a server, or sent back to the client.
388 The core of a package is the protocol unit - for example, a
389 Z39.50 Init Request or Search Response, or an SRU searchRetrieve
390 URL or Explain Response. In addition to this core, a package
391 also carries some extra information added and used by Metaproxy
395 In general, packages are doctored as they pass through
396 Metaproxy. For example, when the proxy performs authentication
397 and authorisation on a Z39.50 Init request, it removes the
398 authentication credentials from the package so that they are not
399 passed onto the back-end server; and when search-response
400 packages are obtained from multiple servers, they are merged
401 into a single unified package that makes its way back to the
410 Packages make their way through routes, which can be thought of
411 as programs that operate on the package data-type. Each
412 incoming package initially makes its way through a default
413 route, but may be switched to a different route based on various
414 considerations. Routes are made up of sequences of filters (see
423 Filters provide the individual instructions within a route, and
424 effect the necessary transformations on packages. A particular
425 configuration of Metaproxy is essentially a set of filters,
426 described by configuration details and arranged in order in one
427 or more routes. There are many kinds of filter - about a dozen
428 at the time of writing with more appearing all the time - each
429 performing a specific function and configured by different
433 The word ``filter'' is sometimes used rather loosely, in two
434 different ways: it may be used to mean a particular
435 <emphasis>type</emphasis> of filter, as when we speak of ``the
436 auth_simplefilter'' or ``the multi filter''; or it may be used
437 to be a specific <emphasis>instance</emphasis> of a filter
438 within a Metaproxy configuration. For example, a single
439 configuration will often contain multiple instances of the
440 <literal>z3950_client</literal> filter. In
441 operational terms, of these is a separate filter. In practice,
442 context always make it clear which sense of the word ``filter''
446 Extensibility of Metaproxy is primarily through the creation of
447 plugins that provide new filters. The filter API is small and
448 conceptually simple, but there are many details to master. See
450 <link linkend="extensions">extensions</link>.
456 Since packages are created and handled by the system itself, and
457 routes are conceptually simple, most of the remainder of this
458 document concentrates on filters. After a brief overview of the
459 filter types follows, along with some thoughts on possible future
466 <chapter id="filters">
467 <title>Filters</title>
471 <title>Introductory notes</title>
473 It's useful to think of Metaproxy as an interpreter providing a small
474 number of primitives and operations, but operating on a very
475 complex data type, namely the ``package''.
478 A package represents a Z39.50 or SRU/W request (whether for Init,
479 Search, Scan, etc.) together with information about where it came
480 from. Packages are created by front-end filters such as
481 <literal>frontend_net</literal> (see below), which reads them from
482 the network; other front-end filters are possible. They then pass
483 along a route consisting of a sequence of filters, each of which
484 transforms the package and may also have side-effects such as
485 generating logging. Eventually, the route will yield a response,
486 which is sent back to the origin.
489 There are many kinds of filter: some that are defined statically
490 as part of Metaproxy, and others may be provided by third parties
491 and dynamically loaded. They all conform to the same simple API
492 of essentially two methods: <function>configure()</function> is
493 called at startup time, and is passed a DOM tree representing that
494 part of the configuration file that pertains to this filter
495 instance: it is expected to walk that tree extracting relevant
496 information; and <function>process()</function> is called every
497 time the filter has to processes a package.
500 While all filters provide the same API, there are different modes
501 of functionality. Some filters are sources: they create
503 (<literal>frontend_net</literal>);
504 others are sinks: they consume packages and return a result
505 (<literal>z3950_client</literal>,
506 <literal>backend_test</literal>,
507 <literal>http_file</literal>);
508 the others are true filters, that read, process and pass on the
509 packages they are fed
510 (<literal>auth_simple</literal>,
511 <literal>log</literal>,
512 <literal>multi</literal>,
513 <literal>query_rewrite</literal>,
514 <literal>session_shared</literal>,
515 <literal>template</literal>,
516 <literal>virt_db</literal>).
521 <section id="overview.filter.types">
522 <title>Overview of filter types</title>
524 We now briefly consider each of the types of filter supported by
525 the core Metaproxy binary. This overview is intended to give a
526 flavour of the available functionality; more detailed information
527 about each type of filter is included below in
528 <link linkend="filterref"
529 >the reference guide to Metaproxy filters</link>.
532 The filters are here named by the string that is used as the
533 <literal>type</literal> attribute of a
534 <literal><filter></literal> element in the configuration
535 file to request them, with the name of the class that implements
536 them in parentheses. (The classname is not needed for normal
537 configuration and use of Metaproxy; it is useful only to
541 The filters are here listed in alphabetical order:
545 <title><literal>auth_simple</literal>
546 (mp::filter::AuthSimple)</title>
548 Simple authentication and authorisation. The configuration
549 specifies the name of a file that is the user register, which
550 lists <varname>username</varname>:<varname>password</varname>
551 pairs, one per line, colon separated. When a session begins, it
552 is rejected unless username and passsword are supplied, and match
553 a pair in the register. The configuration file may also specific
554 the name of another file that is the target register: this lists
555 lists <varname>username</varname>:<varname>dbname</varname>,<varname>dbname</varname>...
556 sets, one per line, with multiple database names separated by
557 commas. When a search is processed, it is rejected unless the
558 database to be searched is one of those listed as available to
564 <title><literal>backend_test</literal>
565 (mp::filter::Backend_test)</title>
567 A sink that provides dummy responses in the manner of the
568 <literal>yaz-ztest</literal> Z39.50 server. This is useful only
569 for testing. Seriously, you don't need this. Pretend you didn't
570 even read this section.
575 <title><literal>frontend_net</literal>
576 (mp::filter::FrontendNet)</title>
578 A source that accepts Z39.50 connections from a port
579 specified in the configuration, reads protocol units, and
580 feeds them into the next filter in the route. When the result is
581 revceived, it is returned to the original origin.
586 <title><literal>http_file</literal>
587 (mp::filter::HttpFile)</title>
589 A sink that returns the contents of files from the local
590 filesystem in response to HTTP requests. (Yes, Virginia, this
591 does mean that Metaproxy is also a Web-server in its spare time. So
592 far it does not contain either an email-reader or a Lisp
593 interpreter, but that day is surely coming.)
598 <title><literal>log</literal>
599 (mp::filter::Log)</title>
601 Writes logging information to standard output, and passes on
602 the package unchanged.
607 <title><literal>multi</literal>
608 (mp::filter::Multi)</title>
610 Performs multi-database searching.
612 <link linkend="multidb">the extended discussion</link>
613 of virtual databases and multi-database searching below.
618 <title><literal>query_rewrite</literal>
619 (mp::filter::QueryRewrite)</title>
621 Rewrites Z39.50 Type-1 and Type-101 (``RPN'') queries by a
622 three-step process: the query is transliterated from Z39.50
623 packet structures into an XML representation; that XML
624 representation is transformed by an XSLT stylesheet; and the
625 resulting XML is transliterated back into the Z39.50 packet
631 <title><literal>session_shared</literal>
632 (mp::filter::SessionShared)</title>
634 When this is finished, it will implement global sharing of
635 result sets (i.e. between threads and therefore between
636 clients), yielding performance improvements especially when
637 incoming requests are from a stateless environment such as a
638 web-server, in which the client process representing a session
639 might be any one of many. However:
643 This filter is not yet completed.
649 <title><literal>template</literal>
650 (mp::filter::Template)</title>
652 Does nothing at all, merely passing the packet on. (Maybe it
653 should be called <literal>nop</literal> or
654 <literal>passthrough</literal>?) This exists not to be used, but
655 to be copied - to become the skeleton of new filters as they are
656 written. As with <literal>backend_test</literal>, this is not
657 intended for civilians.
662 <title><literal>virt_db</literal>
663 (mp::filter::Virt_db)</title>
665 Performs virtual database selection: based on the name of the
666 database in the search request, a server is selected, and its
667 address added to the request in a <literal>VAL_PROXY</literal>
668 otherInfo packet. It will subsequently be used by a
669 <literal>z3950_client</literal> filter.
671 <link linkend="multidb">the extended discussion</link>
672 of virtual databases and multi-database searching below.
677 <title><literal>z3950_client</literal>
678 (mp::filter::Z3950Client)</title>
680 Performs Z39.50 searching and retrieval by proxying the
681 packages that are passed to it. Init requests are sent to the
682 address specified in the <literal>VAL_PROXY</literal> otherInfo
683 attached to the request: this may have been specified by client,
684 or generated by a <literal>virt_db</literal> filter earlier in
685 the route. Subsequent requests are sent to the same address,
686 which is remembered at Init time in a Session object.
692 <section id="future.directions">
693 <title>Future directions</title>
695 Some other filters that do not yet exist, but which would be
696 useful, are briefly described. These may be added in future
697 releases (or may be created by third parties, as loadable
703 <term><literal>frontend_cli</literal> (source)</term>
706 Command-line interface for generating requests.
711 <term><literal>frontend_sru</literal> (source)</term>
714 Receive SRU (and perhaps SRW) requests.
719 <term><literal>sru2z3950</literal> (filter)</term>
722 Translate SRU requests into Z39.50 requests.
727 <term><literal>sru_client</literal> (sink)</term>
730 SRU searching and retrieval.
735 <term><literal>srw_client</literal> (sink)</term>
738 SRW searching and retrieval.
743 <term><literal>opensearch_client</literal> (sink)</term>
746 A9 OpenSearch searching and retrieval.
756 <chapter id="configuration">
757 <title>Configuration: the Metaproxy configuration file format</title>
761 <title>Introductory notes</title>
763 If Metaproxy is an interpreter providing operations on packages, then
764 its configuration file can be thought of as a program for that
765 interpreter. Configuration is by means of a single file, the name
766 of which is supplied as the sole command-line argument to the
767 <command>metaproxy</command> program. (See
768 <link linkend="progref">the reference guide</link>
769 below for more information on invoking Metaproxy.)
772 The configuration files are written in XML. (But that's just an
773 implementation detail - they could just as well have been written
774 in YAML or Lisp-like S-expressions, or in a custom syntax.)
777 Since XML has been chosen, an XML schema,
778 <filename>config.xsd</filename>, is provided for validating
779 configuration files. This file is supplied in the
780 <filename>etc</filename> directory of the Metaproxy distribution. It
781 can be used by (among other tools) the <command>xmllint</command>
782 program supplied as part of the <literal>libxml2</literal>
786 xmllint --noout --schema etc/config.xsd my-config-file.xml
789 (A recent version of <literal>libxml2</literal> is required, as
790 support for XML Schemas is a relatively recent addition.)
794 <section id="overview.xml.structure">
795 <title>Overview of XML structure</title>
797 All elements and attributes are in the namespace
798 <ulink url="http://indexdata.dk/yp2/config/1"/>.
799 This is most easily achieved by setting the default namespace on
800 the top-level element, as here:
803 <yp2 xmlns="http://indexdata.dk/yp2/config/1">
806 The top-level element is <yp2>. This contains a
807 <start> element, a <filters> element and a
808 <routes> element, in that order. <filters> is
809 optional; the other two are mandatory. All three are
813 The <start> element is empty, but carries a
814 <literal>route</literal> attribute, whose value is the name of
815 route at which to start running - analogous to the name of the
816 start production in a formal grammar.
819 If present, <filters> contains zero or more <filter>
820 elements. Each filter carries a <literal>type</literal> attribute
821 which specifies what kind of filter is being defined
822 (<literal>frontend_net</literal>, <literal>log</literal>, etc.)
823 and contain various elements that provide suitable configuration
824 for a filter of its type. The filter-specific elements are
826 <link linkend="filterref">the reference guide below</link>.
827 Filters defined in this part of the file must carry an
828 <literal>id</literal> attribute so that they can be referenced
832 <routes> contains one or more <route> elements, each
833 of which must carry an <literal>id</literal> element. One of the
834 routes must have the ID value that was specified as the start
835 route in the <start> element's <literal>route</literal>
836 attribute. Each route contains zero or more <filter>
837 elements. These are of two types. They may be empty, but carry a
838 <literal>refid</literal> attribute whose value is the same as the
839 <literal>id</literal> of a filter previously defined in the
840 <filters> section. Alternatively, a route within a filter
841 may omit the <literal>refid</literal> attribute, but contain
842 configuration elements similar to those used for filters defined
843 in the <filters> section. (In other words, each filter in a
844 route may be included either by reference or by physical
850 <section id="example.configuration">
851 <title>An example configuration</title>
853 The following is a small, but complete, Metaproxy configuration
854 file (included in the distribution as
855 <literal>metaproxy/etc/config0.xml</literal>).
856 This file defines a very simple configuration that simply proxies
857 to whatever back-end server the client requests, but logs each
858 request and response. This can be useful for debugging complex
859 client-server dialogues.
861 <screen><![CDATA[<?xml version="1.0"?>
862 <yp2 xmlns="http://indexdata.dk/yp2/config/1">
863 <start route="start"/>
865 <filter id="frontend" type="frontend_net">
868 <filter id="backend" type="z3950_client">
873 <filter refid="frontend"/>
875 <filter refid="backend"/>
881 It works by defining a single route, called
882 <literal>start</literal>, which consists of a sequence of three
883 filters. The first and last of these are included by reference:
884 their <literal><filter></literal> elements have
885 <literal>refid</literal> attributes that refer to filters defined
886 within the prior <literal><filters></literal> section. The
887 middle filter is included inline in the route.
890 The three filters in the route are as follows: first, a
891 <literal>frontend_net</literal> filter accepts Z39.50 requests
892 from any host on port 9000; then these requests are passed through
893 a <literal>log</literal> filter that emits a message for each
894 request; they are then fed into a <literal>z3950_client</literal>
895 filter, which forwards the requests to the client-specified
896 back-end Z39.509 server. When the response arrives, it is handed
897 back to the <literal>log</literal> filter, which emits another
898 message; and then to the front-end filter, which returns the
899 response to the client.
906 <chapter id="multidb">
907 <title>Virtual databases and multi-database searching</title>
911 <title>Introductory notes</title>
913 Two of Metaproxy's filters are concerned with multiple-database
914 operations. Of these, <literal>virt_db</literal> can work alone
915 to control the routing of searches to one of a number of servers,
916 while <literal>multi</literal> can work together with
917 <literal>virt_db</literal> to perform multi-database searching, merging
918 the results into a unified result-set - ``metasearch in a box''.
921 The interaction between
922 these two filters is necessarily complex: it reflects the real,
923 irreducible complexity of multi-database searching in a protocol such
924 as Z39.50 that separates initialisation from searching, and in
925 which the database to be searched is not known at initialisation
929 It's possible to use these filters without understanding the
930 details of their functioning and the interaction between them; the
931 next two sections of this chapter are ``HOWTO'' guides for doing
932 just that. However, debugging complex configurations will require
933 a deeper understanding, which the last two sections of this
934 chapters attempt to provide.
939 <section id="multidb.virt_db">
940 <title>Virtual databases with the <literal>virt_db</literal> filter</title>
942 Working alone, the purpose of the
943 <literal>virt_db</literal>
944 filter is to route search requests to one of a selection of
945 back-end databases. In this way, a single Z39.50 endpoint
946 (running Metaproxy) can provide access to several different
947 underlying services, including those that would otherwise be
948 inaccessible due to firewalls. In many useful configurations, the
949 back-end databases are local to the Metaproxy installation, but
950 the software does not enforce this, and any valid Z39.50 servers
951 may be used as back-ends.
954 For example, a <literal>virt_db</literal>
955 filter could be set up so that searches in the virtual database
956 ``lc'' are forwarded to the Library of Congress bibliographic
957 catalogue server, and searches in the virtual database ``marc''
958 are forwarded to the toy database of MARC records that Index Data
959 hosts for testing purposes. A <literal>virt_db</literal>
960 configuration to make this switch would look like this:
962 <screen><![CDATA[<filter type="virt_db">
964 <database>lc</database>
965 <target>z3950.loc.gov:7090/voyager</target>
968 <database>marc</database>
969 <target>indexdata.dk/marc</target>
971 </filter>]]></screen>
973 As well as being useful in it own right, this filter also provides
974 the foundation for multi-database searching.
979 <section id="multidb.multi">
980 <title>Multi-database search with the <literal>multi</literal> filter</title>
982 To arrange for Metaproxy to broadcast searches to multiple back-end
983 servers, the configuration needs to include two components: a
984 <literal>virt_db</literal>
985 filter that specifies multiple
986 <literal><target></literal>
987 elements, and a subsequent
988 <literal>multi</literal>
989 filter. Here, for example, is a complete configuration that
990 broadcasts searches to both the Library of Congress catalogue and
991 Index Data's tiny testing database of MARC records:
993 <screen><![CDATA[<?xml version="1.0"?>
994 <yp2 xmlns="http://indexdata.dk/yp2/config/1">
995 <start route="start"/>
998 <filter type="frontend_net">
999 <threads>10</threads>
1002 <filter type="virt_db">
1004 <database>lc</database>
1005 <target>z3950.loc.gov:7090/voyager</target>
1008 <database>marc</database>
1009 <target>indexdata.dk/marc</target>
1012 <database>all</database>
1013 <target>z3950.loc.gov:7090/voyager</target>
1014 <target>indexdata.dk/marc</target>
1017 <filter type="multi"/>
1018 <filter type="z3950_client">
1019 <timeout>30</timeout>
1026 <literal>virt_db</literal>
1027 filter that specifies multiple
1028 <literal><target></literal>
1029 elements but without a subsequent
1030 <literal>multi</literal>
1031 filter yields surprising and undesirable results, as will be
1032 described below. Don't do that.)
1035 Metaproxy can be invoked with this configuration as follows:
1037 <screen>../src/metaproxy --config config-simple-multi.xml</screen>
1039 And thereafter, Z39.50 clients can connect to the running server
1040 (on port 9000, as specified in the configuration) and search in
1041 any of the databases
1042 <literal>lc</literal> (the Library of Congress catalogue),
1043 <literal>marc</literal> (Index Data's test database of MARC records)
1045 <literal>all</literal> (both of these). As an example, a session
1046 using the YAZ command-line client <literal>yaz-client</literal> is
1047 here included (edited for brevity and clarity):
1049 <screen><![CDATA[$ yaz-client @:9000
1053 Search was a success.
1054 Number of hits: 10000, setno 1
1058 Search was a success.
1059 Number of hits: 10, setno 3
1063 Search was a success.
1064 Number of hits: 10010, setno 4
1067 [marc]Record type: USmarc
1070 005 00000000000000.0
1071 008 910710c19910701nju 00010 eng
1075 100 10 $a Jack Collins
1076 245 10 $a How to program a computer
1082 [VOYAGER]Record type: USmarc
1084 005 20041229102447.0
1085 008 030910s2004 caua 000 0 eng
1086 035 $a (DLC) 2003112666
1087 906 $a 7 $b cbc $c orignew $d 4 $e epcn $f 20 $g y-gencatlg
1088 925 0 $a acquire $b 1 shelf copy $x policy default
1089 955 $a pc10 2003-09-10 $a pv12 2004-06-23 to SSCD; $h sj05 2004-11-30 $e sj05 2004-11-30 to Shelf.
1092 040 $a DLC $c DLC $d DLC
1093 050 00 $a MLCM 2004/03312 (G)
1094 245 10 $a 007, everything or nothing : $b Prima's official strategy guide / $c created by Kaizen Media Group.
1095 246 3 $a Double-O-seven, everything or nothing
1096 246 30 $a Prima's official strategy guide
1097 260 $a Roseville, CA : $b Prima Games, $c c2004.
1098 300 $a 161 p. : $b col. ill. ; $c 28 cm.
1099 500 $a "Platforms: Nintendo GameCube, Macintosh, PC, PlayStation 2 computer entertainment system, Xbox"--P. [4] of cover.
1100 650 0 $a Video games.
1101 710 2 $a Kaizen Media Group.
1102 856 42 $3 Publisher description $u http://www.loc.gov/catdir/description/random052/2003112666.html
1107 As can be seen, the first record in the result set is from the
1108 Index Data test database, and the second from the Library of
1109 Congress database. The result-set continues alternating records
1110 round-robin style until the point where one of the databases'
1111 records are exhausted.
1114 This example uses only two back-end databases; more may be used.
1115 There is no limitation imposed on the number of databases that may
1116 be metasearched in this way: issues of resource usage and
1117 administrative complexity dictate the practical limits.
1122 <section id="multidb.what">
1123 <title>What's going on?</title>
1125 <title>Lark's vomit</title>
1127 This section goes into a level of technical detail that is
1128 probably not necessary in order to configure and use Metaproxy.
1129 It is provided only for those who like to know how things work.
1130 You should feel free to skip on to the next section if this one
1131 doesn't seem like fun.
1135 Hold on tight - this may get a little hairy.
1138 In the general course of things, a Z39.50 Init request may carry
1139 with it an otherInfo packet of type <literal>VAL_PROXY</literal>,
1140 whose value indicates the address of a Z39.50 server to which the
1141 ultimate connection is to be made. (This otherInfo packet is
1142 supported by YAZ-based Z39.50 clients and servers, but has not yet
1143 been ratified by the Maintenance Agency and so is not widely used
1144 in non-Index Data software. We're working on it.)
1145 The <literal>VAL_PROXY</literal> packet functions
1146 analogously to the absoluteURI-style Request-URI used with the GET
1147 method when a web browser asks a proxy to forward its request: see
1149 <ulink url="http://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html#sec5.1.2"
1150 >Request-URI</ulink>
1152 <ulink url="http://www.w3.org/Protocols/rfc2616/rfc2616.html"
1153 >the HTTP 1.1 specification</ulink>.
1156 The role of the <literal>virt_db</literal> filter is to rewrite
1157 this otherInfo packet dependent on the virtual database that the
1158 client wants to search.
1161 When Metaproxy receives a Z39.50 Init request from a client, it
1162 doesn't immediately forward that request to the back-end server.
1163 Why not? Because it doesn't know <emphasis>which</emphasis>
1164 back-end server to forward it to until the client sends a search
1165 request that specifies the database that it wants to search in.
1166 Instead, it just treasures the Init request up in its heart; and,
1167 later, the first time the client does a search on one of the
1168 specified virtual databases, a connection is forged to the
1169 appropriate server and the Init request is forwarded to it. If,
1170 later in the session, the same client searches in a different
1171 virtual database, then a connection is forged to the server that
1172 hosts it, and the same cached Init request is forwarded there,
1176 All of this clever Init-delaying is done by the
1177 <literal>frontend_net</literal> filter. The
1178 <literal>virt_db</literal> filter knows nothing about it; in
1179 fact, because the Init request that is received from the client
1180 doesn't get forwarded until a Search request is received, the
1181 <literal>virt_db</literal> filter (and the
1182 <literal>z3950_client</literal> filter behind it) doesn't even get
1183 invoked at Init time. The <emphasis>only</emphasis> thing that a
1184 <literal>virt_db</literal> filter ever does is rewrite the
1185 <literal>VAL_PROXY</literal> otherInfo in the requests that pass
1189 ### Describe the use of multiple <literal>VAL_PROXY</literal>
1190 otherInfos, added by <literal>virt_db</literal> and used by
1191 <literal>multi</literal>.
1196 <section id="multidb.picture">
1197 <title>A picture is worth a thousand words (but only five hundred on 64-bit architectures)</title>
1201 <imagedata fileref="multi.pdf" format="PDF" scale="50"/>
1204 <imagedata fileref="multi.png" format="PNG"/>
1207 <!-- Fall back if none of the images can be used -->
1209 [Here there should be a diagram showing the progress of
1210 packages through the filters during a simple virtual-database
1211 search and a multi-database search, but is seems that your
1212 toolchain has not been able to include the diagram in this
1213 document. This is because of LaTeX suckage. Time to move to
1214 OpenOffice. Yes, really.]
1217 <!-- ### This used to work with an older version of DocBook
1219 <para>Caption: progress of packages through filters.</para>
1222 </inlinemediaobject>
1229 <chapter id="extensions">
1230 <title>Writing extensions for Metaproxy</title>
1231 <para>### To be written</para>
1237 <chapter id="classes">
1238 <title>Classes in the Metaproxy source code</title>
1242 <title>Introductory notes</title>
1244 <emphasis>Stop! Do not read this!</emphasis>
1245 You won't enjoy it at all. You should just skip ahead to
1246 <link linkend="refguide">the reference guide</link>,
1248 <!-- The remainder of this paragraph is lifted verbatim from
1249 Douglas Adams' _Hitch Hiker's Guide to the Galaxy_, chapter 8 -->
1250 you things you really need to know, like the fact that the
1251 fabulously beautiful planet Bethselamin is now so worried about
1252 the cumulative erosion by ten billion visiting tourists a year
1253 that any net imbalance between the amount you eat and the amount
1254 you excrete whilst on the planet is surgically removed from your
1255 bodyweight when you leave: so every time you go to the lavatory it
1256 is vitally important to get a receipt.
1259 This chapter contains documentation of the Metaproxy source code, and is
1260 of interest only to maintainers and developers. If you need to
1261 change Metaproxy's behaviour or write a new filter, then you will most
1262 likely find this chapter helpful. Otherwise it's a waste of your
1263 good time. Seriously: go and watch a film or something.
1264 <citetitle>This is Spinal Tap</citetitle> is particularly good.
1267 Still here? OK, let's continue.
1270 In general, classes seem to be named big-endianly, so that
1271 <literal>FactoryFilter</literal> is not a filter that filters
1272 factories, but a factory that produces filters; and
1273 <literal>FactoryStatic</literal> is a factory for the statically
1274 registered filters (as opposed to those that are dynamically
1279 <section id="individual.classes">
1280 <title>Individual classes</title>
1282 The classes making up the Metaproxy application are here listed by
1283 class-name, with the names of the source files that define them in
1288 <title><literal>mp::FactoryFilter</literal>
1289 (<filename>factory_filter.cpp</filename>)</title>
1291 A factory class that exists primarily to provide the
1292 <literal>create()</literal> method, which takes the name of a
1293 filter class as its argument and returns a new filter of that
1294 type. To enable this, the factory must first be populated by
1295 calling <literal>add_creator()</literal> for static filters (this
1296 is done by the <literal>FactoryStatic</literal> class, see below)
1297 and <literal>add_creator_dyn()</literal> for filters loaded
1303 <title><literal>mp::FactoryStatic</literal>
1304 (<filename>factory_static.cpp</filename>)</title>
1306 A subclass of <literal>FactoryFilter</literal> which is
1307 responsible for registering all the statically defined filter
1308 types. It does this by knowing about all those filters'
1309 structures, which are listed in its constructor. Merely
1310 instantiating this class registers all the static classes. It is
1311 for the benefit of this class that <literal>struct
1312 metaproxy_1_filter_struct</literal> exists, and that all the filter
1313 classes provide a static object of that type.
1318 <title><literal>mp::filter::Base</literal>
1319 (<filename>filter.cpp</filename>)</title>
1321 The virtual base class of all filters. The filter API is, on the
1322 surface at least, extremely simple: two methods.
1323 <literal>configure()</literal> is passed a DOM tree representing
1324 that part of the configuration file that pertains to this filter
1325 instance, and is expected to walk that tree extracting relevant
1326 information. And <literal>process()</literal> processes a
1327 package (see below). That surface simplicitly is a bit
1328 misleading, as <literal>process()</literal> needs to know a lot
1329 about the <literal>Package</literal> class in order to do
1335 <title><literal>mp::filter::AuthSimple</literal>,
1336 <literal>Backend_test</literal>, etc.
1337 (<filename>filter_auth_simple.cpp</filename>,
1338 <filename>filter_backend_test.cpp</filename>, etc.)</title>
1340 Individual filters. Each of these is implemented by a header and
1341 a source file, named <filename>filter_*.hpp</filename> and
1342 <filename>filter_*.cpp</filename> respectively. All the header
1343 files should be pretty much identical, in that they declare the
1344 class, including a private <literal>Rep</literal> class and a
1345 member pointer to it, and the two public methods. The only extra
1346 information in any filter header is additional private types and
1347 members (which should really all be in the <literal>Rep</literal>
1348 anyway) and private methods (which should also remain known only
1349 to the source file, but C++'s brain-damaged design requires this
1350 dirty laundry to be exhibited in public. Thanks, Bjarne!)
1353 The source file for each filter needs to supply:
1358 A definition of the private <literal>Rep</literal> class.
1363 Some boilerplate constructors and destructors.
1368 A <literal>configure()</literal> method that uses the
1369 appropriate XML fragment.
1374 Most important, the <literal>process()</literal> method that
1375 does all the actual work.
1382 <title><literal>mp::Package</literal>
1383 (<filename>package.cpp</filename>)</title>
1385 Represents a package on its way through the series of filters
1386 that make up a route. This is essentially a Z39.50 or SRU APDU
1387 together with information about where it came from, which is
1388 modified as it passes through the various filters.
1393 <title><literal>mp::Pipe</literal>
1394 (<filename>pipe.cpp</filename>)</title>
1396 This class provides a compatibility layer so that we have an IPC
1397 mechanism that works the same under Unix and Windows. It's not
1398 particularly exciting.
1403 <title><literal>mp::RouterChain</literal>
1404 (<filename>router_chain.cpp</filename>)</title>
1411 <title><literal>mp::RouterFleXML</literal>
1412 (<filename>router_flexml.cpp</filename>)</title>
1419 <title><literal>mp::Session</literal>
1420 (<filename>session.cpp</filename>)</title>
1427 <title><literal>mp::ThreadPoolSocketObserver</literal>
1428 (<filename>thread_pool_observer.cpp</filename>)</title>
1435 <title><literal>mp::util</literal>
1436 (<filename>util.cpp</filename>)</title>
1438 A namespace of various small utility functions and classes,
1439 collected together for convenience. Most importantly, includes
1440 the <literal>mp::util::odr</literal> class, a wrapper for YAZ's
1446 <title><literal>mp::xml</literal>
1447 (<filename>xmlutil.cpp</filename>)</title>
1449 A namespace of various XML utility functions and classes,
1450 collected together for convenience.
1456 <section id="other.source.files">
1457 <title>Other Source Files</title>
1459 In addition to the Metaproxy source files that define the classes
1460 described above, there are a few additional files which are
1461 briefly described here:
1465 <term><literal>metaproxy_prog.cpp</literal></term>
1468 The main function of the <command>metaproxy</command> program.
1473 <term><literal>ex_router_flexml.cpp</literal></term>
1476 Identical to <literal>metaproxy_prog.cpp</literal>: it's not clear why.
1481 <term><literal>test_*.cpp</literal></term>
1484 Unit-tests for various modules.
1490 ### Still to be described:
1491 <literal>ex_filter_frontend_net.cpp</literal>,
1492 <literal>filter_dl.cpp</literal>,
1493 <literal>plainfile.cpp</literal>,
1494 <literal>tstdl.cpp</literal>.
1501 <chapter id="refguide">
1502 <title>Reference guide</title>
1504 The material in this chapter is drawn directly from the individual
1505 manual entries. In particular, the Metaproxy invocation section is
1506 available using <command>man metaproxy</command>, and the section
1507 on each individual filter is available using the name of the filter
1508 as the argument to the <command>man</command> command.
1512 <section id="progref">
1513 <title>Metaproxy invocation</title>
1518 <section id="filterref">
1519 <title>Reference guide to Metaproxy filters</title>
1526 <!-- Keep this comment at the end of the file
1531 sgml-minimize-attributes:nil
1532 sgml-always-quote-attributes:t
1535 sgml-parent-document: "main.xml"
1536 sgml-local-catalogs: nil
1537 sgml-namecase-general:t