X-Git-Url: http://sru.miketaylor.org.uk/?a=blobdiff_plain;f=doc%2Fbook.xml;h=d6e6f8d355072044c5dcdbc43f20e069925f8f38;hb=8fc15e69384e20bb9c305a84683c36311bdec9f3;hp=47569c21d6036f09b967c2cb725a8257c387b11d;hpb=23d7c8c225d1491935460c25581337e265f9d648;p=metaproxy-moved-to-github.git
diff --git a/doc/book.xml b/doc/book.xml
index 47569c2..d6e6f8d 100644
--- a/doc/book.xml
+++ b/doc/book.xml
@@ -1,40 +1,196 @@
-
+
Metaproxy - User's Guide and ReferenceMikeTaylor
-
- AdamDickmeiss
-
-
- 2006
- Index Data
-
+
+ AdamDickmeiss
+
+
+ 2006
+ Index Data
+
- Metaproxy - Generic Z39.50/SRU router/proxy.
+ Metaproxy is a universal router, proxy and encapsulated
+ metasearcher for information retrieval protocols. It accepts,
+ processes, interprets and redirects requests from IR clients using
+ standard protocols such as ANSI/NISO Z39.50 (and in the future SRU
+ and SRW), as well as functioning as a limited
+ HTTP server. Metaproxy is configured by an XML file which
+ specifies how the software should function in terms of routes that
+ the request packets can take through the proxy, each step on a
+ route being an instantiation of a filter. Filters come in many
+ types, one for each operation: accepting Z39.50 packets, logging,
+ query transformation, multiplexing, etc. Further filter-types can
+ be added as loadable modules to extend Metaproxy functionality,
+ using the filter API.
+
+
+ The terms under which Metaproxy will be distributed have yet to be
+ established, but it will not necessarily be open source; so users
+ should not at this stage redistribute the code without explicit
+ written permission from the copyright holders, Index Data ApS.
-
+
Introduction
-
- Overview
- Metaproxy
- is ..
+ Metaproxy
+ is a standalone program that acts as a universal router, proxy and
+ encapsulated metasearcher for information retrieval protocols such
+ as Z39.50, and in the future SRU and SRW. To clients, it acts as a
+ server of these
+ protocols: it can be searched, records can be retrieved from it,
+ etc. To servers, it acts as a client: it searches in them,
+ retrieves records from them, etc. it satisfies its clients'
+ requests by transforming them, multiplexing them, forwarding them
+ on to zero or more servers, merging the results, transforming
+ them, and delivering them back to the client. In addition, it
+ acts as a simple HTTP server; support for further protocols can be
+ added in a modular fashion, through the creation of new filters.
+
+ Anything goes in!
+ Anything goes out!
+ Cold bananas, fish, pyjamas,
+ Mutton, beef and trout!
+ - attributed to Cole Porter.
+
- ### We should probably consider saying a little more by way of
- introduction.
+ Metaproxy is a more capable alternative to
+ YAZ Proxy,
+ being more powerful, flexible, configurable and extensible. Among
+ its many advantages over the older, more pedestrian work are
+ support for multiplexing (encapsulated metasearching), routing by
+ database name, authentication and authorisation and serving local
+ files via HTTP. Equally significant, its modular architecture
+ facilitites the creation of pluggable modules implementing further
+ functionality.
-
-
+
+
+
+
+ The Metaproxy Licence
+
+
+ No decision has yet been made on the terms under which
+ Metaproxy will be distributed.
+
+ It is possible that, unlike
+ other Index Data products, metaproxy may not be released under a
+ free-software licence such as the GNU GPL. Until a decision is
+ made and a public statement made, then, and unless it has been
+ delivered to you other specific terms, please treat Metaproxy as
+ though it were proprietary software.
+ The code should not be redistributed without explicit
+ written permission from the copyright holders, Index Data ApS.
+
+
+
+
+
+
+ The Metaproxy Architecture
+
+ The Metaproxy architecture is based on three concepts:
+ the package,
+ the route
+ and the filter.
+
+
+
+ Packages
+
+
+ A package is request or response, encoded in some protocol,
+ issued by a client, making its way through Metaproxy, send to or
+ received from a server, or sent back to the client.
+
+
+ The core of a package is the protocol unit - for example, a
+ Z39.50 Init Request or Search Response, or an SRU searchRetrieve
+ URL or Explain Response. In addition to this core, a package
+ also carries some extra information added and used by Metaproxy
+ itself.
+
+
+ In general, packages are doctored as they pass through
+ Metaproxy. For example, when the proxy performs authentication
+ and authorisation on a Z39.50 Init request, it removes the
+ authentication credentials from the package so that they are not
+ passed onto the back-end server; and when search-response
+ packages are obtained from multiple servers, they are merged
+ into a single unified package that makes its way back to the
+ client.
+
+
+
+
+ Routes
+
+
+ Packages make their way through routes, which can be thought of
+ as programs that operate on the package data-type. Each
+ incoming package initially makes its way through a default
+ route, but may be switched to a different route based on various
+ considerations. Routes are made up of sequences of filters (see
+ below).
+
+
+
+
+ Filters
+
+
+ Filters provide the individual instructions within a route, and
+ effect the necessary transformations on packages. A particular
+ configuration of Metaproxy is essentially a set of filters,
+ described by configuration details and arranged in order in one
+ or more routes. There are many kinds of filter - about a dozen
+ at the time of writing with more appearing all the time - each
+ performing a specific function and configured by different
+ information.
+
+
+ The word ``filter'' is sometimes used rather loosely, in two
+ different ways: it may be used to mean a particular
+ type of filter, as when we speak of ``the
+ auth_simplefilter'' or ``the multi filter''; or it may be used
+ to be a specific instance of a filter
+ within a Metaproxy configuration. For example, a single
+ configuration will often contain multiple instances of the
+ z3950_client filter. In
+ operational terms, of these is a separate filter. In practice,
+ context always make it clear which sense of the word ``filter''
+ is being used.
+
+
+ Extensibility of Metaproxy is primarily through the creation of
+ plugins that provide new filters. The filter API is small and
+ conceptually simple, but there are many details to master. See
+ the section below on
+ extensions.
+
+
+
+
+
+ Since packages are created and handled by the system itself, and
+ routes are conceptually simple, most of the remainder of this
+ document concentrates on filters. After a brief overview of the
+ filter types follows, along with some thoughts on possible future
+ directions.
+
+
+
@@ -49,7 +205,7 @@
complex data type, namely the ``package''.
- A package represents a Z39.50 or SRW/U request (whether for Init,
+ A package represents a Z39.50 or SRU/W request (whether for Init,
Search, Scan, etc.) together with information about where it came
from. Packages are created by front-end filters such as
frontend_net (see below), which reads them from
@@ -61,7 +217,7 @@
There are many kinds of filter: some that are defined statically
- as part of Metaproxy, and other that may be provided by third parties
+ as part of Metaproxy, and others may be provided by third parties
and dynamically loaded. They all conform to the same simple API
of essentially two methods: configure() is
called at startup time, and is passed a DOM tree representing that
@@ -84,6 +240,7 @@
(auth_simple,
log,
multi,
+ query_rewrite,
session_shared,
template,
virt_db).
@@ -91,14 +248,27 @@
-
- Individual filters
+
+ Overview of filter types
+
+ We now briefly consider each of the types of filter supported by
+ the core Metaproxy binary. This overview is intended to give a
+ flavour of the available functionality; more detailed information
+ about each type of filter is included below in
+ the reference guide to Metaproxy filters.
+
The filters are here named by the string that is used as the
type attribute of a
<filter> element in the configuration
file to request them, with the name of the class that implements
- them in parentheses.
+ them in parentheses. (The classname is not needed for normal
+ configuration and use of Metaproxy; it is useful only to
+ developers.)
+
+
+ The filters are here listed in alphabetical order:
@@ -110,10 +280,13 @@
lists username:password
pairs, one per line, colon separated. When a session begins, it
is rejected unless username and passsword are supplied, and match
- a pair in the register.
-
-
- ### discuss authorisation phase
+ a pair in the register. The configuration file may also specific
+ the name of another file that is the target register: this lists
+ lists username:dbname,dbname...
+ sets, one per line, with multiple database names separated by
+ commas. When a search is processed, it is rejected unless the
+ database to be searched is one of those listed as available to
+ the user.
@@ -123,7 +296,8 @@
A sink that provides dummy responses in the manner of the
yaz-ztest Z39.50 server. This is useful only
- for testing.
+ for testing. Seriously, you don't need this. Pretend you didn't
+ even read this section.
@@ -131,10 +305,10 @@
<literal>frontend_net</literal>
(mp::filter::FrontendNet)
- A source that accepts Z39.50 and SRW connections from a port
+ A source that accepts Z39.50 connections from a port
specified in the configuration, reads protocol units, and
- feeds them into the next filter, eventually returning the
- result to the origin.
+ feeds them into the next filter in the route. When the result is
+ revceived, it is returned to the original origin.
@@ -163,8 +337,23 @@
<literal>multi</literal>
(mp::filter::Multi)
- Performs multicast searching. See the extended discussion of
- multi-database searching below.
+ Performs multicast searching.
+ See
+ the extended discussion
+ of virtual databases and multi-database searching below.
+
+
+
+
+ <literal>query_rewrite</literal>
+ (mp::filter::QueryRewrite)
+
+ Rewrites Z39.50 Type-1 and Type-101 (``RPN'') queries by a
+ three-step process: the query is transliterated from Z39.50
+ packet structures into an XML representation; that XML
+ representation is transformed by an XSLT stylesheet; and the
+ resulting XML is transliterated back into the Z39.50 packet
+ structure.
@@ -174,8 +363,16 @@
When this is finished, it will implement global sharing of
result sets (i.e. between threads and therefore between
- clients), but it's not yet done.
+ clients), yielding performance improvements especially when
+ incoming requests are from a stateless environment such as a
+ web-server, in which the client process representing a session
+ might be any one of many. However:
+
+
+ This filter is not yet completed.
+
+
@@ -186,7 +383,8 @@
should be called nop or
passthrough?) This exists not to be used, but
to be copied - to become the skeleton of new filters as they are
- written.
+ written. As with backend_test, this is not
+ intended for civilians.
@@ -194,8 +392,14 @@
<literal>virt_db</literal>
(mp::filter::Virt_db)
- Performs virtual database selection. See the extended discussion
- of virtual databases below.
+ Performs virtual database selection: based on the name of the
+ database in the search request, a server is selected, and its
+ address added to the request in a VAL_PROXY
+ otherInfo packet. It will subsequently be used by a
+ z3950_client filter.
+ See
+ the extended discussion
+ of virtual databases and multi-database searching below.
@@ -215,12 +419,13 @@
-
+ Future directions
Some other filters that do not yet exist, but which would be
useful, are briefly described. These may be added in future
- releases.
+ releases (or may be created by third parties, as loadable
+ modules).
@@ -233,19 +438,19 @@
- srw2z3950 (filter)
+ frontend_sru (source)
- Translate SRW requests into Z39.50 requests.
+ Receive SRU (and perhaps SRW) requests.
- srw_client (sink)
+ sru2z3950 (filter)
- SRW searching and retrieval.
-
+ Translate SRU requests into Z39.50 requests.
+
@@ -257,6 +462,14 @@
+ srw_client (sink)
+
+
+ SRW searching and retrieval.
+
+
+
+ opensearch_client (sink)
@@ -281,7 +494,9 @@
its configuration file can be thought of as a program for that
interpreter. Configuration is by means of a single file, the name
of which is supplied as the sole command-line argument to the
- yp2 program.
+ metaproxy program. (See
+ the reference guide
+ below for more information on invoking Metaproxy.)
The configuration files are written in XML. (But that's just an
@@ -306,7 +521,7 @@
-
+ Overview of XML structure
All elements and attributes are in the namespace
@@ -327,15 +542,19 @@
The <start> element is empty, but carries a
route attribute, whose value is the name of
- route at which to start running - analogouse to the name of the
+ route at which to start running - analogous to the name of the
start production in a formal grammar.
If present, <filters> contains zero or more <filter>
- elements; filters carry a type attribute and
- contain various elements that provide suitable configuration for
- filters of that type. The filter-specific elements are described
- below. Filters defined in this part of the file must carry an
+ elements. Each filter carries a type attribute
+ which specifies what kind of filter is being defined
+ (frontend_net, log, etc.)
+ and contain various elements that provide suitable configuration
+ for a filter of its type. The filter-specific elements are
+ described in
+ the reference guide below.
+ Filters defined in this part of the file must carry an
id attribute so that they can be referenced
from elsewhere.
@@ -351,131 +570,72 @@
<filters> section. Alternatively, a route within a filter
may omit the refid attribute, but contain
configuration elements similar to those used for filters defined
- in the <filters> section.
+ in the <filters> section. (In other words, each filter in a
+ route may be included either by reference or by physical
+ inclusion.)
-
- Filter configuration
+
+ An example configuration
- All <filter> elements have in common that they must carry a
- type attribute whose value is one of the
- supported ones, listed in the schema file and discussed below. In
- additional, <filters>s occurring the <filters> section
- must have an id attribute, and those occurring
- within a route must have either a refid
- attribute referencing a previously defined filter or contain its
- own configuration information.
+ The following is a small, but complete, Metaproxy configuration
+ file (included in the distribution as
+ metaproxy/etc/config0.xml).
+ This file defines a very simple configuration that simply proxies
+ to whatever backend server the client requests, but logs each
+ request and response. This can be useful for debugging complex
+ client-server dialogues.
+
+
+
+
+
+ @:9000
+
+
+
+
+
+
+
+
+
+
+
+
+]]>
- In general, each filter recognises different configuration
- elements within its element, as each filter has different
- functionality. These are as follows:
+ It works by defining a single route, called
+ start, which consists of a sequence of three
+ filters. The first and last of these are included by reference:
+ their <filter> elements have
+ refid attributes that refer to filters defined
+ within the prior <filters> section. The
+ middle filter is included inline in the route.
+
+
+ The three filters in the route are as follows: first, a
+ frontend_net filter accepts Z39.50 requests
+ from any host on port 9000; then these requests are passed through
+ a log filter that emits a message for each
+ request; they are then fed into a z3950_client
+ filter, which forwards the requests to the client-specified
+ backend Z39.509 server. When the response arrives, it is handed
+ back to the log filter, which emits another
+ message; and then to the front-end filter, which returns the
+ response to the client.
-
-
- <literal>auth_simple</literal>
-
- <filter type="auth_simple">
- <userRegister>../etc/example.simple-auth</userRegister>
- </filter>
-
-
-
-
- <literal>backend_test</literal>
-
- <filter type="backend_test"/>
-
-
-
-
- <literal>frontend_net</literal>
-
- <filter type="frontend_net">
- <threads>10</threads>
- <port>@:9000</port>
- </filter>
-
-
-
-
- <literal>http_file</literal>
-
- <filter type="http_file">
- <mimetypes>/etc/mime.types</mimetypes>
- <area>
- <documentroot>.</documentroot>
- <prefix>/etc</prefix>
- </area>
- </filter>
-
-
-
-
- <literal>log</literal>
-
- <filter type="log">
- <message>B</message>
- </filter>
-
-
-
-
- <literal>multi</literal>
-
- <filter type="multi"/>
-
-
-
-
- <literal>session_shared</literal>
-
- <filter type="session_shared">
- ### Not yet defined
- </filter>
-
-
-
-
- <literal>template</literal>
-
- <filter type="template"/>
-
-
-
-
- <literal>virt_db</literal>
-
- <filter type="virt_db">
- <virtual>
- <database>loc</database>
- <target>z3950.loc.gov:7090/voyager</target>
- </virtual>
- <virtual>
- <database>idgils</database>
- <target>indexdata.dk/gils</target>
- </virtual>
- </filter>
-
-
-
-
- <literal>z3950_client</literal>
-
- <filter type="z3950_client">
- <timeout>30</timeout>
- </filter>
-
-
- Virtual database as multi-database searching
+ Virtual databases and multi-database searching
@@ -498,14 +658,16 @@
-
- Module Reference
-
- The material in this chapter includes the man pages material
-
- &manref;
+
+
+
+ Writing extensions for Metaproxy
+ ### To be written
+
+
+
Classes in the Metaproxy source code
@@ -514,7 +676,18 @@
Introductory notesStop! Do not read this!
- You won't enjoy it at all.
+ You won't enjoy it at all. You should just skip ahead to
+ the reference guide,
+ which tells
+
+ you things you really need to know, like the fact that the
+ fabulously beautiful planet Bethselamin is now so worried about
+ the cumulative erosion by ten billion visiting tourists a year
+ that any net imbalance between the amount you eat and the amount
+ you excrete whilst on the planet is surgically removed from your
+ bodyweight when you leave: so every time you go to the lavatory it
+ is vitally important to get a receipt.
This chapter contains documentation of the Metaproxy source code, and is
@@ -537,7 +710,7 @@
-
+ Individual classes
The classes making up the Metaproxy application are here listed by
@@ -570,7 +743,7 @@
structures, which are listed in its constructor. Merely
instantiating this class registers all the static classes. It is
for the benefit of this class that struct
- yp2_filter_struct exists, and that all the filter
+ metaproxy_1_filter_struct exists, and that all the filter
classes provide a static object of that type.
@@ -664,7 +837,7 @@
<literal>mp::RouterChain</literal>
(<filename>router_chain.cpp</filename>)
- ###
+ ### to be written
@@ -672,7 +845,7 @@
<literal>mp::RouterFleXML</literal>
(<filename>router_flexml.cpp</filename>)
- ###
+ ### to be written
@@ -680,7 +853,7 @@
<literal>mp::Session</literal>
(<filename>session.cpp</filename>)
- ###
+ ### to be written
@@ -688,7 +861,7 @@
<literal>mp::ThreadPoolSocketObserver</literal>
(<filename>thread_pool_observer.cpp</filename>)
- ###
+ ### to be written
@@ -714,7 +887,7 @@
-
+ Other Source Files
In addition to the Metaproxy source files that define the classes
@@ -726,7 +899,7 @@
metaproxy_prog.cpp
- The main function of the yp2 program.
+ The main function of the metaproxy program.
@@ -754,23 +927,36 @@
plainfile.cpp,
tstdl.cpp.
-
-
-
-
- --
-
-
-
-
-
-
-
-
+
+
+
+ Reference guide
+
+ The material in this chapter is drawn directly from the individual
+ manual entries. In particular, the Metaproxy invocation section is
+ available using man metaproxy, and the section
+ on each individual filter is available using the name of the filter
+ as the argument to the man command.
+
+
+
+
+ Metaproxy invocation
+ &progref;
+
+
+
+
+ Reference guide to Metaproxy filters
+ &manref;
+
+
+
+
+