Adam Dickmeiss [Tue, 12 Dec 2006 15:23:45 +0000 (15:23 +0000)]
Rename isb to isb_postings.
Adam Dickmeiss [Tue, 12 Dec 2006 15:22:26 +0000 (15:22 +0000)]
Possible fix of two bugs (1) bug #759: Crash during retrieval of some
records (2) bug #765.
Adam Dickmeiss [Tue, 12 Dec 2006 13:54:33 +0000 (13:54 +0000)]
Added gnuplot scripts for dictisam timings, filesystems comparisons.
Adam Dickmeiss [Tue, 12 Dec 2006 13:51:23 +0000 (13:51 +0000)]
All output to stdout for easy feet into gnuplot. The following is now
logged: user,real,sys for dict+isam, postings, documents, distinct
words, new words, dict splits, isamb leaf splits, isamb interior splits.
Adam Dickmeiss [Tue, 12 Dec 2006 13:46:41 +0000 (13:46 +0000)]
Added functions isamb_set_int_count, isamb-set_cache_size,
isamb_get_int_splits, isamb_get_leaf_splits, iasmb_set_int_count.
Adam Dickmeiss [Tue, 12 Dec 2006 13:42:23 +0000 (13:42 +0000)]
Added statistics functions dict_get_no_{lookup,insert,split}.
Adam Dickmeiss [Mon, 11 Dec 2006 17:08:03 +0000 (17:08 +0000)]
Remove unused variable (dst_buf)
Adam Dickmeiss [Mon, 11 Dec 2006 15:08:55 +0000 (15:08 +0000)]
Allow MARC line format to be read.
Adam Dickmeiss [Mon, 11 Dec 2006 10:02:14 +0000 (10:02 +0000)]
Using timing utility
Adam Dickmeiss [Mon, 11 Dec 2006 09:50:36 +0000 (09:50 +0000)]
Added a utility for doing timings.
Adam Dickmeiss [Sun, 10 Dec 2006 21:02:28 +0000 (21:02 +0000)]
More appropriate usage
Adam Dickmeiss [Sun, 10 Dec 2006 21:00:56 +0000 (21:00 +0000)]
Make MARC indexer with different ISAM strategy.
Adam Dickmeiss [Sun, 10 Dec 2006 20:59:52 +0000 (20:59 +0000)]
Use rand() twice on platforms where RAND_MAX < 2^16.
Adam Dickmeiss [Sun, 10 Dec 2006 11:54:29 +0000 (11:54 +0000)]
Added new test which makes variations to ISAM entry size.
Adam Dickmeiss [Sun, 10 Dec 2006 11:54:11 +0000 (11:54 +0000)]
Produce .dat files with bench1. prefix
Adam Dickmeiss [Sun, 10 Dec 2006 11:49:16 +0000 (11:49 +0000)]
Allow size of ISAM entry to be specified in bench test
Adam Dickmeiss [Sat, 9 Dec 2006 08:03:57 +0000 (08:03 +0000)]
Added some performance test utilities.
Adam Dickmeiss [Thu, 7 Dec 2006 21:13:56 +0000 (21:13 +0000)]
Added some ISAMB benchmark code
Adam Dickmeiss [Thu, 7 Dec 2006 19:23:56 +0000 (19:23 +0000)]
Allow different values for cache parameter for isamb_open: 0=normal,
1=caching of pages, -1 testing (NULL ISAM).
Adam Dickmeiss [Thu, 7 Dec 2006 10:01:32 +0000 (10:01 +0000)]
Remove @XSLT_LIBS@ from idzebra-config --libs / --lalibs output. XSLT
libs is part of @YAZLIB@ already.
Adam Dickmeiss [Wed, 6 Dec 2006 22:52:38 +0000 (22:52 +0000)]
Use zint for numbers in update statistics.
Adam Dickmeiss [Wed, 6 Dec 2006 13:00:12 +0000 (13:00 +0000)]
Added new object check_res.
Adam Dickmeiss [Wed, 6 Dec 2006 10:26:54 +0000 (10:26 +0000)]
Added more to list of options.
Adam Dickmeiss [Wed, 6 Dec 2006 10:26:40 +0000 (10:26 +0000)]
Deal with a picky ambiguity with configs like a.b where both suffix and
prefix are allowed. Both a and b must be checked.
Adam Dickmeiss [Tue, 5 Dec 2006 14:06:29 +0000 (14:06 +0000)]
Fixed bug #445: zebra.cfg does not check for valid directives.
Adam Dickmeiss [Tue, 5 Dec 2006 09:26:37 +0000 (09:26 +0000)]
Provide defaults for profilePath and modulePath for the zebra{srv,idx}.
Adam Dickmeiss [Tue, 5 Dec 2006 09:26:04 +0000 (09:26 +0000)]
Reformat a bit.
Adam Dickmeiss [Tue, 5 Dec 2006 09:24:47 +0000 (09:24 +0000)]
Deal with NULL profile_path.
Adam Dickmeiss [Tue, 5 Dec 2006 09:24:31 +0000 (09:24 +0000)]
For Res(ources), maintain reference count (poor man's smart ptr).
Adam Dickmeiss [Tue, 5 Dec 2006 08:14:59 +0000 (08:14 +0000)]
Experiment with alternative reckeys store.
Adam Dickmeiss [Tue, 5 Dec 2006 08:14:47 +0000 (08:14 +0000)]
Only add first in field marker if there field has term(s) in it
Adam Dickmeiss [Sun, 3 Dec 2006 16:05:13 +0000 (16:05 +0000)]
Fixed bug #739: Extended service update fails with Alvis filter.
This change makes it possible to have register and shadow as same
directory. The size limits is only counted for each file type (so if
shadow is 10M and register is 20M the total size of directory is 30M).
Adam Dickmeiss [Sun, 3 Dec 2006 15:55:02 +0000 (15:55 +0000)]
Removed unused var
Adam Dickmeiss [Thu, 30 Nov 2006 11:03:57 +0000 (11:03 +0000)]
Removed a few heavily used yaz_log calls
Adam Dickmeiss [Thu, 30 Nov 2006 10:33:18 +0000 (10:33 +0000)]
Added support for specification of approximative limits for whole query.
This is specified as attribute type 12. Semantics is the same as
estimatehits in zebra.cfg.
Adam Dickmeiss [Thu, 30 Nov 2006 10:29:23 +0000 (10:29 +0000)]
Change IDs for the special retrieval stuff.
Adam Dickmeiss [Wed, 29 Nov 2006 18:06:57 +0000 (18:06 +0000)]
Disable xelm path ! case code in case of index_name (att in 1.3) being NULL.
It's not documented and not in use for Zebra 2. The .abs reader can
never generate it so it's dead code which just slows indexing down.
Adam Dickmeiss [Wed, 29 Nov 2006 10:31:29 +0000 (10:31 +0000)]
Cut memory for sorting in half when threaded sort is in effect.
Adam Dickmeiss [Wed, 29 Nov 2006 10:24:01 +0000 (10:24 +0000)]
Version 2.0.7.
Marc Cromme [Wed, 29 Nov 2006 09:01:53 +0000 (09:01 +0000)]
shortened zebra::meta:.sysno lement XML format
Marc Cromme [Tue, 28 Nov 2006 14:37:45 +0000 (14:37 +0000)]
added example of config file
Marc Cromme [Tue, 28 Nov 2006 14:18:26 +0000 (14:18 +0000)]
added entry on encoding directive
Marc Cromme [Tue, 28 Nov 2006 13:05:57 +0000 (13:05 +0000)]
added more information on character mapping
added entry on equivalent directive
added short examples
Adam Dickmeiss [Tue, 28 Nov 2006 11:42:13 +0000 (11:42 +0000)]
Towards 2.0.6.
Adam Dickmeiss [Tue, 28 Nov 2006 08:44:10 +0000 (08:44 +0000)]
Issue warning if staticrank is < 0. And set to 0
Adam Dickmeiss [Tue, 28 Nov 2006 08:43:53 +0000 (08:43 +0000)]
Added assert
Adam Dickmeiss [Mon, 27 Nov 2006 21:50:15 +0000 (21:50 +0000)]
Zebra 2 RPM with the following packages: idzebra-2.0, libidzebra-2.0,
libidzebra-2.0-modules, libidzebra-2.0-devel.
Adam Dickmeiss [Mon, 27 Nov 2006 14:27:12 +0000 (14:27 +0000)]
Require YAZ 2.1.40 or later
Adam Dickmeiss [Mon, 27 Nov 2006 10:40:33 +0000 (10:40 +0000)]
Avoid indexing of zebraInfo stuff. Problem is that explain.abs has lines
elm (4,2) targetInfo ExplainCategory
and it will zebraInfo child as ExplainCategory. This is now prevented
by making a zebraInfo node with no children.
Adam Dickmeiss [Mon, 27 Nov 2006 10:10:14 +0000 (10:10 +0000)]
Better debugging for indexing.. Introduction of zebra_log_dict_entry.
Adam Dickmeiss [Mon, 27 Nov 2006 10:09:48 +0000 (10:09 +0000)]
Fixed bug #736: Updates gets slower. The problem was that duplicate
name tags (with database name in it) was generated for attributeDetails.
This caused indexes to grow and grow for each update iteration.
Adam Dickmeiss [Mon, 27 Nov 2006 09:17:01 +0000 (09:17 +0000)]
Disable hit snippets for now. It does not work well and it slows retrieval
down a lot.
Adam Dickmeiss [Sat, 25 Nov 2006 09:15:19 +0000 (09:15 +0000)]
Fixed bug #735: Delete fails to process rest of file after a record that
does not exist,
Adam Dickmeiss [Fri, 24 Nov 2006 19:48:23 +0000 (19:48 +0000)]
Added new object files to list
Adam Dickmeiss [Fri, 24 Nov 2006 13:25:42 +0000 (13:25 +0000)]
Rephase a bit.
Adam Dickmeiss [Fri, 24 Nov 2006 13:25:02 +0000 (13:25 +0000)]
Mention --disable-shared option as a way to prevent modules to be
built.
Adam Dickmeiss [Fri, 24 Nov 2006 13:05:11 +0000 (13:05 +0000)]
Moved section about special retrieval zebra:: to Architecture chapter.
Added table with supported zebra:: elements.
Marc Cromme [Fri, 24 Nov 2006 12:21:31 +0000 (12:21 +0000)]
updated test since non-used attributes are not longer printed out
Adam Dickmeiss [Fri, 24 Nov 2006 11:35:23 +0000 (11:35 +0000)]
Using wrbuf rather than sprintf. Allow all syntaxes for ::data fetch.
Adam Dickmeiss [Thu, 23 Nov 2006 21:38:26 +0000 (21:38 +0000)]
Added check for sys/utsname.h.
Adam Dickmeiss [Thu, 23 Nov 2006 21:35:02 +0000 (21:35 +0000)]
Added Include of string.h
Adam Dickmeiss [Thu, 23 Nov 2006 21:34:22 +0000 (21:34 +0000)]
Omit test with multiple processes and update on FreeBSD. Bug #733.
Adam Dickmeiss [Thu, 23 Nov 2006 18:42:48 +0000 (18:42 +0000)]
Make buildconf.sh work for FreeBSD
Adam Dickmeiss [Thu, 23 Nov 2006 14:01:21 +0000 (14:01 +0000)]
Use size of rec->info stuff in meta record fetch. Failing that will
result in a SEGV or similar.
Marc Cromme [Thu, 23 Nov 2006 09:03:50 +0000 (09:03 +0000)]
implemented full metadata display of fast element set names
implemented more test of fast element set names, including zebra::meta and zebra::data element set names
updated documentation on fast element set names
Adam Dickmeiss [Wed, 22 Nov 2006 14:12:49 +0000 (14:12 +0000)]
Added include of yaz/xmalloc.h.
Adam Dickmeiss [Wed, 22 Nov 2006 14:06:53 +0000 (14:06 +0000)]
Remove function tl_start_log.
Adam Dickmeiss [Wed, 22 Nov 2006 11:13:17 +0000 (11:13 +0000)]
Added a few memset for structures that are later written to disk.
This makes valgind happy on platforms which aligns members in structs.
Adam Dickmeiss [Wed, 22 Nov 2006 10:26:12 +0000 (10:26 +0000)]
Added initializer value for two locals
Adam Dickmeiss [Wed, 22 Nov 2006 09:42:27 +0000 (09:42 +0000)]
For build script: -p enables profiling.
Adam Dickmeiss [Tue, 21 Nov 2006 22:17:49 +0000 (22:17 +0000)]
Get rid of SYSNO which is zint anyway. Removed various prototypes
of unused/undefined functions.
Adam Dickmeiss [Tue, 21 Nov 2006 17:55:47 +0000 (17:55 +0000)]
Added
Adam Dickmeiss [Tue, 21 Nov 2006 17:48:08 +0000 (17:48 +0000)]
Make threading for index sort configurable.
Adam Dickmeiss [Tue, 21 Nov 2006 14:54:12 +0000 (14:54 +0000)]
Remove log msg
Adam Dickmeiss [Tue, 21 Nov 2006 14:32:38 +0000 (14:32 +0000)]
Split it_key stuff into it_key.c. For indexing, sort key chunks in a separate
thread (POSIX threads only).
Adam Dickmeiss [Mon, 20 Nov 2006 13:59:35 +0000 (13:59 +0000)]
Increase reckeys hash size from 1023 to 32767.
Adam Dickmeiss [Mon, 20 Nov 2006 13:59:13 +0000 (13:59 +0000)]
Remove obsolete SORT_EXTRA code (no longer in use).
Adam Dickmeiss [Fri, 17 Nov 2006 17:32:45 +0000 (17:32 +0000)]
Contact address for Zebra is zebra-help@indexdata.dk
Marc Cromme [Fri, 17 Nov 2006 14:54:00 +0000 (14:54 +0000)]
updated docs to reflect change to 'zebra::index::' special elem set name
Marc Cromme [Fri, 17 Nov 2006 14:24:56 +0000 (14:24 +0000)]
added tests fror wrong element set names which are not recognized
Marc Cromme [Fri, 17 Nov 2006 13:47:22 +0000 (13:47 +0000)]
preventing later memory acess violation when one leaves retrieve.c:zebra_special_fetch by error condition before setting *rec_lenp by setting int len = 0; upon initialization.
Marc Cromme [Fri, 17 Nov 2006 08:20:17 +0000 (08:20 +0000)]
removing unnecessary comments
not setting the *rec_lenp = 0; at retrieve.c:127 to provoke test failure with segfault in the last unit test of test/api/t16.c , which is commented out. It seems like *rec_lenp = wrbuf_len(wrbuf); at retrieve.c:247 leaves rec_lenp in an uninitialized state, which causes a segfault later. More investigation needed.
Marc Cromme [Thu, 16 Nov 2006 21:58:54 +0000 (21:58 +0000)]
protected wrong buffer-length write by initializing to 0, but there is more debugging needed: there is still something fishy with the last out-commented test of test/api/t16.c.
Marc Cromme [Thu, 16 Nov 2006 20:44:28 +0000 (20:44 +0000)]
added tests for zebra::data, zebra::meta::sysno and zebra::index::title:p element sets. Last one segfaults and need further debugging, it is commented out.
Adam Dickmeiss [Thu, 16 Nov 2006 13:27:54 +0000 (13:27 +0000)]
For alvis filter make XML pass-through if stylesheet is attribute
is omitted (not specified).
Adam Dickmeiss [Thu, 16 Nov 2006 12:48:28 +0000 (12:48 +0000)]
Added new test for check for zebra:: element set names.
Adam Dickmeiss [Thu, 16 Nov 2006 11:11:36 +0000 (11:11 +0000)]
Fix setting of recordAttr->recordSize in case of storeData method.
Adam Dickmeiss [Thu, 16 Nov 2006 11:10:46 +0000 (11:10 +0000)]
Output format = input for zebra::data fetch (raw fetch).
Adam Dickmeiss [Thu, 16 Nov 2006 10:49:11 +0000 (10:49 +0000)]
idzebra:: stuff functional.
Adam Dickmeiss [Thu, 16 Nov 2006 10:48:35 +0000 (10:48 +0000)]
Annotate zerbaExplain_lookup_attr_str
Marc Cromme [Wed, 15 Nov 2006 14:09:43 +0000 (14:09 +0000)]
added suggestions for better workflow in comments
Marc Cromme [Tue, 14 Nov 2006 14:32:13 +0000 (14:32 +0000)]
re-organized record retrieval such that special element set names
zebra::index display all indexed terms from all indexes
zebra::index::title display all terms from only title index
zebra::index::title:p display all terms from only title index type p
zebra::meta::sysno display only zebra internal record sysno
zebra::data dislpay raw record binary data as SUTRS
all work
More testing with zebra:.data is needed
Sitll missing zebra::meta::filename, zebra::meta::count, etc ..
Better error messages would be an asset...
Adam Dickmeiss [Tue, 14 Nov 2006 12:41:19 +0000 (12:41 +0000)]
Added small test of mfile sub system
Adam Dickmeiss [Tue, 14 Nov 2006 12:11:48 +0000 (12:11 +0000)]
Avoid abort. EXTRA_CHECK defined and annotated
Adam Dickmeiss [Tue, 14 Nov 2006 12:03:48 +0000 (12:03 +0000)]
Use assert rather than abort
Adam Dickmeiss [Tue, 14 Nov 2006 10:45:34 +0000 (10:45 +0000)]
Changed test to pass for Libxml 2.6.16.
Adam Dickmeiss [Tue, 14 Nov 2006 10:03:21 +0000 (10:03 +0000)]
Fix mf_write call in cf_commit_flat.
Adam Dickmeiss [Tue, 14 Nov 2006 08:12:06 +0000 (08:12 +0000)]
Work on bug #550: Avoid exit. In particular the mfile/cfile/bfile has
been refactored quite a bit. Remaining exit uses zebra_exit.
Removed redundant apitest.c.
Marc Cromme [Mon, 13 Nov 2006 14:53:40 +0000 (14:53 +0000)]
added section on special zebra:: elemset names, and updated docs to follow suit with changes alvis oai filter example
Marc Cromme [Mon, 13 Nov 2006 13:53:49 +0000 (13:53 +0000)]
written XML formatting for new zebra-specific elem set 'zebra::' which lets you dump the exact content of indexed fields in records.
Using for example elemset 'zebra::dc_title' dumps only the physical string index 'dc_title' fields in the virtual, from indexes constructed, record. Usisn 'zebra::dc_title:w' dumps only te subset of those which are indexed as type 'w'.
Added test record for oai-alvis example with utf-8 sequences and nasty XML escapes. Testing works on GRS-1 filters, but yet not on 'Alvis' filters, the direct YAZ function zebra_storekeys_fetch in index/retrieve.c seems not be called at all .. more debugging needed.