3 z3950_client: fix string handling for init diagnostics in the case where
6 query_rewrite: no charset conversion for error queries. Avoid performing
7 charset conversion for query conversion errors. The resulting Type-1
8 query is invalid in this case.
12 For graceful stop (SIGUSR1), metaproxy will kill children processes.
14 Log when metaproxy is invoked in test mode (-t).
16 Log PID for each log message.
18 multi: combine both NSDs and multiple NSDs into one (diagnostics).
20 z3950_client: fixup multiple NSD's too. Like surrogate diagnostics,
21 a (backend=target) is appended for each diagnostic so that can be
22 determined the origin of error.
24 multi: hideerrors also in use for present response.
26 Solr example in zoom man page.
28 Fix typo in query_rewrite config example.
30 Log at least first non surrogate diagnostic. Instead of
31 "Z_Records_multipleNSD"
33 multi: Fix check for opt Records in present response. A SEGV could occur
34 if a target did not return records and no diagnostics.
38 Yet another fix for graceful stop.
42 sru_z3950: relay surrogate diagnostics.
46 zoom: Honor extraArgs, strip # in zurl.
48 multi: fix SEGV for scan. Reported by Rustam T. Usmanov.
50 zoom: url_recipe allow %{var[reg]} substitutions.
51 It's like ${var[reg]} but does URI component encoding as well.
55 RPM/Deb: require yazpp 1.3.2, because it fixes a missing close of
56 non-keepalive HTTP sessions.
58 http_file: allow zero size files (bug fix)
60 Fix zoom schema: auth_url is optional
64 zoom: fix un-init memory (could result in SEGV at session close)
66 frontend_net: handle X-Forwarded-For. This is stored in package.origin()
67 and may be used in filters as usual.
69 zoom: authentication via Torus lookup
70 Enabled by new configuration setting: auth_url . If this is defined,
71 the zoom module will contact Torus for auth/IP lookup and that will
72 define realm to be used in session.
76 virt_db: Fix database name - first character we removed by mistake.
77 Bug introduced in 1.3.40.
81 virt_db: Don't use "unix path" as database for Unix sockets.
85 z3950_client: Don't tamper with / in zurl - fixes problems
86 using Unix sockets for backends.
88 session_shared: fix integer overflow problem with hit count.
90 session_shared: avoid long usage of cached queries.
92 zoom: New database argument , torus_url to set URL of Torus
97 Fixes for 'metaproxy reload' for RPM/Debian packages.
99 zoom: Allow rpn2cql conversion to be augmented with a properties file.
100 Element rpn2cql can be specified per-target for the zoom filter.
102 log: New category "line" which is a single-line log message with
103 most important parts of a HTTP/Z39.50 request/response.
105 --- 1.3.37 2012/06/01
107 frontend_net: add facility to report live statistics about response
108 times and current thread usage. The report is an XML document
109 and is triggered for a specific URL path. This is set by <stat-req>
110 in the configuration of frontend_net. By default this report is
111 disabled (same as empty value for <stat-req>).
113 Fix reload problem in init.d script for RPM/Debian.
114 The '/etc/init.d/metaproxy reload' in some cases would stop but not
115 start metaproxy - effectively stopping the service.
117 Don't display start log message when config testing (-t).
119 zoom: avoid CCL message limit of 1K in ZOOM filter.
121 --- 1.3.36 2012/05/10
123 metaproxy init script reacts to reload which does graceful stop +
124 start and is used by logrorate script.
126 --- 1.3.35 2012/05/09
128 Fixes for Windows compilation.
130 Metaproxy daemon reacts to SIGUSR1 in which case it will leave
131 existing sessions running but terminate the monitor process that is
132 used in keepalive mode.
134 Change start log message to 'metaproxy start'. Was 'Metaproxy start'
135 before. Consistent with 'metaproxy stop' message.
137 zoom: document zoom's proxy_timeout
139 z3950_client: fixup addinfo for init response
141 virt_db: return other diagnostic if init is rejected. Diagnostic 236:
142 "Access to specified database denied" is returned, but only if init
143 response from backend does not carry diagnostics already.
145 multi: relays user info for first rejected target. The other-information
146 field for init response for first rejected target is in common init
147 response. This ensures that some backend init diagnostics are returned to
150 --- 1.3.34 2012/04/25
152 frontend_log: extend logging (when message is given in config).
154 --- 1.3.33 2012/04/25
156 zoom: Fix check for proxy. Use diagnostic 1074 for proxy failures.
158 --- 1.3.32 2012/04/24
160 sru_z3950 filter passes through incoming Z39.50 requests again. Was
161 not handled by mistake in Metaproxy versions 1.3.29, 1.3.30 and 1.3.31.
163 --- 1.3.31 2012/04/24
165 zoom: fixes for retry/proxy failover logic. More logging.
167 --- 1.3.30 2012/04/23
169 zoom: return addinfo="proxy failure" for proxy failures. This also
170 changes the scheme for dealing with proxies a bit. zoom now always
171 do proxy check, even if there's no failover. Fortunately, that's not
174 zoom: Init rejected wo diagnostics maps to Bib-1:1014.
175 Bib-1 1014, "Init/AC: Authentication System error", seems pretty
176 generic. We don't know why the init would fail; IP, user, block IP, etc.
177 sru_z3950 filter maps this to SRU diagnostic 3: "Authentication error".
179 zoom: guess if invalid SRU response was an auth failure
181 session_shared: disable session retry by default . Can be enabled by
182 restart="true" in resultset section.
184 --- 1.3.29 2012/04/18
186 zoom: may read cf-proxy's config settings.
187 This reduces manual configuration of the content proxy server host.
188 The ZOOM module will use the old style session.proxyhost if "server"
189 is given (deprecated) and use the new style proxyhost/session if
190 "config_file" is in use.
192 frontend_net: remove incoming requests that can not be handled
193 Requests in queue for filter frontend_net are removed if client
194 closes connection for the session and request is not yet handled.
196 sru_z3950: serialize requests. This is to ensure that pipelined
197 HTTP requests are handled properly.
199 --- 1.3.28 2012/04/04
201 New class, wrbuf, that wraps YAZ' WRBUF.
203 --- 1.3.27 2012/03/30
205 Metaproxy daemon is now running as user "metaproxy". This user is
206 managed by package maintainer scripts. The user information is stored
207 in /etc/metaproxy/metaproxy.user .
209 --- 1.3.26 2012/03/20
211 New filter, sort, that allows sorting of XML records via Z39.50
214 Filter record_transform: USEMarcon support. For example:
215 <retrieval syntax="usmarc">
216 <backend syntax="unimarc" name="F">
217 <usemarcon stage1="/etc/usemarcon/uni2us/uni2us.ini"/>
221 Filter multi: empty route pattern uses route as default.
223 <target route="z3950.indexdata.com">z3950.indexdata.com</route>
224 may use the shorter notation
225 <target route="z3950.indexdata.com"/>
227 Filter query_rewrite: use config path to locate XSLT.
229 Filter frontend_net: log statistics.
230 If a custom message element in frontend_net configuration it makes
231 the frontend_net filter produce a log entry whenever an operation
232 is completed. The log is always written using yaz_log with the
233 custom message given, the package ID (session ID) , the time
234 the operation has been active (end-to-end) , the threads currently
235 active/total and the size of the request input/output queues.
237 query_rewrite: charset conversion of query terms.
238 This is an optional feature. Enabled by element <charset> in
239 configuration with attribute 'from' and 'to' that specifies
240 from/to encodings. The default 'from' encoding is UTF-8.
242 Filter record_transform: honor piggyback searches.
244 Filter zoom: re-connect when database is reused.
245 Re-connect is necessary if a connection is lost or a timeout occurs.
246 If not, we'll be seeing zero hits, when we shouldn't.
248 --- 1.3.25 2012/02/29
250 zoom: Change scheme for dealing with content proxy URLs. The module
251 now has its own common XSL transform that is enabled by record_xsl
252 attribute in the configuration. This XSL gets parameter cproxyhost
253 that holds the host for the content proxy session . The genearted-url
254 is created only if urlRecipe is enabled. In previous version default
255 value of urlRecipe was ${md-electronic-url}; it is now empty and
256 disabled by default. Also generated-url no longer includes the content
257 proxy host. Content proxying URLs is supposed to the handle in the record_xsl
260 zoom: new setting timeout that specifies timeout for outgoing Z39.50/SRU
261 connections. Default value is 40 (seconds).
263 zoom: tune CCL to Z39.50 diagnostics mapping. Throw Bib-1: 108
264 "Malformed query" for syntax errors and related ones; throw
265 Bib-1: 3 "Unsupported search" for everything else. Details from
266 the CCL parser is provided as additional-info.
268 multi: new setting hideerrors that makes multi only relay diagnostics
269 to a client if all backends fail.
271 --- 1.3.24 2012/02/23
273 zoom: empty ccl_map field-mappings disables CCL map.
275 zoom: fix Explain search that could return error if session was
278 --- 1.3.23 2012/02/10
280 zoom: returns Explain records for database IR-Explain---1. The
281 target profile records are converted to Explain records via XSLT.
283 session_shared: make error recovery configurable. Whether a search
284 is restarted on errors or not, is configurable by attribute "restart"
285 in element "resultset" in configuration.
287 session_shared: only relay init for first request. session_shared, for
288 each backend class, now only relays first init request (rather than all).
289 The module only needs to save init response info once. This avoids the
290 need to check for session max.
292 --- 1.3.22 2012/02/02
294 zoom: fix memory leaks.
296 session_shared: maximum number of sessons is configurable. Default
297 value is 100. Result-sets for backends without named result sets are
298 only re-used if databases is identical for a search. This is under the
299 assumption that performance may suffer for those backends when
300 changing the database for each search.
302 multi: combines diagnostics from all backends. All diagnostics from
303 backends are combined into multiple non-surrogate diagnostics; rather
304 than just one diagnostic from first backend.
306 --- 1.3.21 2011/12/27
308 zoom: relay realm to cproxy parameter file.
310 Configuration: Allow filter collection element inside route. This
311 makes it possible to include multiple filters at once.
313 zoom: nocproxy also disables urlRecipe.
315 --- 1.3.20 2011/12/21
317 zoom: return diagnostic if CQL sortby conv fails. Diagnostic Bib-1 214
318 "Illegal sort relation" is returned. Gets translated to SRU diagnostic
319 90: "Unsupported direction value".
321 zoom: honor database argument nocproxy. This disables creating a content
322 connector session (for non-CF targets, usually) and also relays nocproxy
323 as database argument to a CF target.
325 --- 1.3.19 2011/12/16
327 zoom: Multiple proxy hosts may be given for proxy rich database
328 parameter. List is comma separated.
330 http_file: decodes URI paths properly. Does not return 404: not found
331 if no prefix are matches. Instead packages are passed on.
333 --- 1.3.18 2011/12/08
335 zoom: content-{user,password,proxy} are passed on to content proxy
336 system and takes precedence over user,password,proxy .
338 sru_z3950: SRU extra argument x-log-enable=1 enables a search response
339 with a log element including log material from children filters. At
340 this point, only the zoom filter uses this log facility. It will
341 eventually be used by other filters as well. In particular it might be
342 useful for this feature to track all filters in used during a search.
344 zoom: for x-log-enable=1, APDUs sent by ZOOM C, records as received
345 and queries as they are manipulated are logged.
347 zoom: extend addinfo with native diagnostics. Diagnostics as received
348 by ZOOM C are put into addinfo - including ZOOM, HTTP, SRU and Bib-1
349 diagnostics. This indicates both a backend diagnostic and the original
352 zoom: default urlRecipe value is ${md-electronic-url}
354 zoom: target profile setting sruVersion is recognized.
356 --- 1.3.17 2011/11/17
358 zoom: log target profile record fetched (AKA Torus).
360 --- 1.3.16 2011/11/17
362 Change target profile parameter CfSubDb to CfSubDB.
364 --- 1.3.15 2011/11/12
366 zoom: bug fix: realm was relayed to CF targets by mistake (realm only
367 makes sense to the zoom module itself).
369 --- 1.3.14 2011/11/11
371 zoom: realm may be given as part of database.
373 --- 1.3.13 2011/10/26
375 Fix Origin class assigment method; was removd by mistake in 1.3.12.
377 --- 1.3.12 2011/10/26
379 zoom: database args x-name=value are passed through to backend database.
381 SRU filter: user may specify session ID that is passed to log entries
382 that follows. It is specified by using x-session-id in the SRU arguments.
384 zoom: fix null pointer reference which would occur for bad url parameter
385 inside "torus" or if url was not given.
387 --- 1.3.11 2011/09/09
388 Fix bug in filter session_shared where sessions would not expire
389 properly (session ttl). This in turn could make the target close the
390 connection before expected and session_shared would return diagnostic
391 2: system temporarily available.
393 --- 1.3.10 2011/09/09
395 zoom: new target profile setting: sortmap_field. If sortmap_field is given
396 the field is mapped to the value for the sortmap. Values not matching
397 sortmap_field are passed verbatim.
399 zoom: new target profile setting: sortStrategy. One of: "z3950", "type7",
400 "cql", "sru11" or "embed". The "embed" chooses type-7 or CQL sortby
401 depending on whether RPN or CQL is actually sent to the target.
405 zoom: urlRecipe result is stored in <pz:metadata type="generated-url">.
407 zoom: For content proxy authentication is stored in param name
408 "auth", instead of "authentication".
412 zoom: relay schema in SRU mode. Previously recordSchema was not affected
417 zoom: new target profile element literalTransform that holds XSLT
418 (encoded as string) for target. Only if present and non-zero length it is
419 considered present. This takes precedence over transform.
421 Don't rely on trang anymore for Git checkout builds. Configuration in etc
422 will only be properly validated if trang is available. From now on only
423 relax NC files are bundled with the source dist.
425 Updates to check for Boost unit test (autoconf stuff). On MacOSX this
426 component was not detected when it should be.
430 z3950_client: mark non-surrogate diagnostics from backend.
432 zoom: a proxy may be specified for HTTP fetch of target profiles.
436 Honor piggyback. Makes some SRU targets behave that insist on
437 maximumRecords > 0, but also increases performance.
441 Fix dlpath in installed metaproxy.xml for RPM/Deb package. The
442 file was not update for libdir/metaproxy4 in version 1.3.2.
446 Change location of Metaproxy modules from libdir/metaproxy to
447 libdir/metaproxy4. Directory libdir/metaproxy is also owned by
448 libmetaproxy3 and, thus, should not be used by libmetaproxy4.
450 New facility to dump APDUs from zoom filter. Enabled by
451 <log apdu="true"/> in the zoom filter configuation.
453 zoom filter bug fix: SRU diagnostics received from backend was
454 not properly converted to Bib-1 diagnostics.
456 Package metaproxy depends on libmetaproxy4, same version.
460 zoom: fix a bug WRT passing authentication parameters to backends.
464 Fixes for windows Makefile.
466 --- 1.2.10 2011/07/25
468 Change major version of metaproxy library from 3 to 4. This in turn
469 makes new package names for Debian and RPM.
473 zoom: enable marc-8 encoding by default for usmarc and opac (in pz2
476 zoom: configurable element sets for record transforms.
478 zoom: the target profile based transform is specified by element_transform
479 in element torus of of the ZOOM filter configuration ; default value
480 is "pz2". Target profile based retrieval is specified by element_raw
481 in configuration. It will perform same steps, but omit the XSL transform.
482 Default value is "raw".
484 zoom: element set is omitted if not set in profile. But only if pz2
485 transform is in effect. Previously the default element set was to "F".
487 zoom: For non-CF targets database arguments are parsed. User may supply
488 db,user=myuser&password=mypass for non-CF targets. These values override
489 what's given the authentication from the target profile.
491 session_shared: fix reference to freed memory. Would occur if a search
492 response had non-surrogate diagnostics.
496 Route may be given for each port in frontend_net.
500 Documentation for zoom filter. Local target profiles may be specified.
502 Fixes for zoom filter.
506 First version with zoom filter.
510 Filter z3950_client may be configured to always send a close APDU
511 to backend target when connection/session is closed. This will only
512 happen if close APDU is not already sent. Enabled by configuration
513 for z3950_client (force_close).
515 Filter virt_db and multi: relay close APDU.
517 Filter virt_db and multi: relay init size parameters
518 preferredMessageSize and maximumRecordSize.
520 Filter sru_z3950: increase init sz parameters from 1MB to 10 MB.
524 RPM package for Metaproxy works for both SLES11 and Centos 5.5.
526 metaproxy-config: exec_prefix, libdir, includedir. Use these variables
527 as determined by configure.
529 RPM: Release includes 'indexdata' name. Use indexdata in release name so
530 this package can be distinguished from other vendors.
534 Fix filter multi: relay idAuthentication parameters.
538 Add metaproxy-config for returning compiler flags, libraries.
542 Fix --with-boost=PREFIX .
546 session_shared: fix use of maintenance thread.
548 Introduce 'start' method for filters and routers. It gets called after
549 configure and after fork, but before processing. This changes API -
550 from libmetaproxy2 to libmetaproxy3.
554 session_shared: ensure backend session is closed if init rejected
557 z3950_client: fix NULL ptr reference that could occur if two clients
558 connected concurrently and if max-sockets was in effect.
562 Fix bug 3497 - Metaproxy unstable while stopping.
564 Fix bug 3495 - SEGV in SRU filter.
568 Filter multi throws diagnostic 'Present Out of Range' when
569 appropriate - rather than returning 0 records.
571 Filter virt_db: fix bad result reuse. Could happen if a result
572 was overriden and a backend did not supported named result sets.
576 Filter log: category access logs diagnostics even if searchStatus
577 or presentStatus is 'success'.
581 The SRU to Z39.50 filter does repeated Z39.50 present requests
582 to fetch all records. Needed when a Z39.50 server returns partial
583 results (due to message size limits).
587 Metaproxy now kills all child processes that are spawned by
588 modules (in case of fork, exec).
592 Debian package now installs /etc/metaproxy/metaproxy.xml which
593 includes all XML files /etc/metaproxy/filters-enabled .
594 The include facility uses YAZ' yaz_xml_include_simple (YAZ 4.0.1).
598 Package based on YAZ version 4 and YAZ++ 1.2.0. It now includes a
599 development package to facilitate 3rd party Metaproxy modules.
601 --- 1.0.23 2010/01/04
603 Filter virt_db: Close backend filter session when Init rejected is received
604 from it (in which case it's not really closed yet). This will ensure we don't
605 run out of resources such as FDs in this case.
607 sru_z3950 filter: Accept any database (including those not listed).
609 --- 1.0.22 2009/12/03
611 Prepare for 64-bit YAZ. yazproxy will still work with YAZ++ 1.1.4 or
612 later which means YAZ 3.0.47 or later.
614 --- 1.0.20 2009/11/19
616 For filter virt_db, the database is a glob pattern and the virtual
617 database sections are consulted in the order given.
619 --- 1.0.19 2009/07/08
621 For filter multi, the target (for routing) may be given as a glob-pattern.
623 More documentation for filters virt_db and multi.
625 Update schema to allow for no database in sru_z3950 filter.
627 --- 1.0.18 2009/06/30
629 Added limit filter to put sleep (penalty) for heavy use.
631 Filter frontend_net may be configured to limit the number of new connections
632 from same IP to some value (connect-max).
634 Fixed the load_balance filter.
636 --- 1.0.17 2009/06/16
638 Fix path for metaproxy executable for Debian init.d script.
640 --- 1.0.16 2009/06/10
642 SRU to Z39.50 filter honors x-username and x-password. If only
643 x-username is given that translates to open authentication (Z39.50v2). If
644 both x-username and x-password is present that translates to idPass
645 authentication (Z39.50v3).
647 New facility to change current working directory (-w).
649 Update to upstream skeleton of init.d script. Should fix bug #2619.
651 Filter session_shared now caches records (used to only cache connections
654 Filter session_shared now invalidates a Z39.50 session if a server
655 returns -1 hits. This facility was also part of yazproxy and gets around
656 a bug in Voyager-based Z39.50 servers.
658 --- 1.0.15 2008/11/18
660 Fixed reference counting bug in Z39.50 client module (could cause SEGV).
662 --- 1.0.14 2008/06/09
664 Fixed the value of SRU record schema in searchRetrieve response. It was set
665 to a fixed value of "dc" before.
667 Serialize identical SRU requests. This is useful if HTTP client software
668 fires identical requests against metaproxy.
670 Allow Z39.50 target to be specified for SRU requests by using SRU extra
671 parameter x-target. Allows SRU to contact arbitrary Z39.50 targets and
672 avoid static configuration. This facility is handled by the sru_z3950 module
673 and the z3950_client module.
675 Allow maximum number of sockets to be used for a Z39.50 client by the
676 use of SRU extra parameter x-max-sockets. This facility is handled by the
677 sru_z3950 module and the z3950_client module.
679 --- 1.0.13.1 2008/02/29
681 Simplified check for Boost libraries.
683 --- 1.0.13 2008/02/29
685 First Debian package.
687 The sru_z9350 filter now recognizes SRU parameter z-target as being
688 a ZURL of the form host/database .
690 Mergetype may be specified for filter 'multi'.
692 --- 1.0.12 2007/09/19
694 Fixed problem in Makefile WRT library specifictions. This could produce
695 linker errors (when they should not happen).
697 Fixed bug #1341: MetaProxy returning different results. Problem was that
698 fix-up of record database did not normalize the database name.
700 Documentation updates for Windows compilation.
702 --- 1.0.11 2007/06/14
704 Fixed problem with logfile being truncated.
706 --- 1.0.10 2007/05/23
708 Fixed bug #1136: Access rights not enforced.
710 Extension for log module. The log category 'access' logs more details.
711 New category 'user-access' added which is simlar to 'access' but adds
712 the Z39.50 user as for all entries in a session.
716 Added support for configurable default/force target/vhost for module Z39.50
721 Implemented filter, session_shared, which optimizes usage of Z39.50
722 sessions and result sets.
724 New filter, record_transform, which converts MARC/XML records.
726 New filter, sru_z3950, which converts SRU to Z39.50 packages.
728 New filter, load_balance, in development.
730 New filter, zeerex_xplain.
732 XML framework for config files started.
734 Master RelaxNG Compact schema's are found in xml/schema/*.rnc,
735 RelaxNG XML schemas and XML Schema's are generated from RNC syntax.
736 XML config files are checked against RNG and XML Schema's.
738 XSLT transformation framework added in xml/xslt/*.xsl, consisting mainly of
739 LOC Marc21 stylesheets found at http://www.loc.gov/marcxml/xslt.
740 Added stylesheet explain2cqlpqftxt.xsl which transforms a standard SRU ZeeReX
741 explain record to YAZ CQL-to-PQF text config file.
745 Added 'timeout' setting for filter frontend_net which specifies the number
746 of seconds a session is idle before it is closed. Default timeout is
751 Database name in records returned set for virt_db/multi filters.
753 Filter session_shared now functional.
755 Log filter can be configured to write to a given file using 'filename'
756 element. Also information category can be specified with 'category'. If
757 filename is omitted, the log filter writes to stdout. See etc/config5.xml
762 Fixed compilation on Windows.
766 Added documentation for the hidenavailable option for the multi filter.
770 Fixed bug #573: Hide errors for individual servers. If the element
771 'hideunavailable' is given in the configuration, individual servers
772 being unavailable are hidden (unless all are unavailable).
774 Fixed bug #574: Database names are recognised case-sensitively.
776 Fixed bug #567: Fix up database name in Name-Plus-Record.
778 Fixed bug #568: Update win/makefile for VS 2005.
780 Fixed bug #569: Some options throws unknown exception.
784 More information about virtual hosts. The 'etc' directory is part of the
789 Build instructions for Windows. NMake makefile part of distribution tar.
795 Filters functional: auth_simple, backend_test, frontend_net, http_file,
796 log, multi, query_rewrite, template, virt_db, z3950_client.
798 Filters in development: session_shared.
800 Pseudo-filters: template, dl