1 <chapter id="fields-and-charsets">
2 <!-- $Id: field-structure.xml,v 1.6 2006-11-23 09:03:50 marc Exp $ -->
3 <title>Field Structure and Character Sets
7 In order to provide a flexible approach to national character set
8 handling, Zebra allows the administrator to configure the set up the
9 system to handle any 8-bit character set — including sets that
10 require multi-octet diacritics or other multi-octet characters. The
11 definition of a character set includes a specification of the
12 permissible values, their sort order (this affects the display in the
13 SCAN function), and relationships between upper- and lowercase
14 characters. Finally, the definition includes the specification of
15 space characters for the set.
19 The operator can define different character sets for different fields,
20 typical examples being standard text fields, numerical fields, and
21 special-purpose fields such as WWW-style linkages (URx).
24 <section id="default-idx-file">
25 <title>The default.idx file</title>
27 The field types, and hence character sets, are associated with data
28 elements by the .abs files (see above).
29 The file <literal>default.idx</literal>
30 provides the association between field type codes (as used in the .abs
31 files) and the character map files (with the .chr suffix). The format
32 of the .idx file is as follows
39 <term>index <replaceable>field type code</replaceable></term>
42 This directive introduces a new search index code.
43 The argument is a one-character code to be used in the
44 .abs files to select this particular index type. An index, roughly,
45 corresponds to a particular structure attribute during search. Refer
46 to <xref linkend="zebrasrv-search"/>.
48 </listitem></varlistentry>
50 <term>sort <replaceable>field code type</replaceable></term>
53 This directive introduces a
54 sort index. The argument is a one-character code to be used in the
55 .abs fie to select this particular index type. The corresponding
56 use attribute must be used in the sort request to refer to this
57 particular sort index. The corresponding character map (see below)
58 is used in the sort process.
60 </listitem></varlistentry>
62 <term>completeness <replaceable>boolean</replaceable></term>
65 This directive enables or disables complete field indexing.
66 The value of the <replaceable>boolean</replaceable> should be 0
67 (disable) or 1. If completeness is enabled, the index entry will
68 contain the complete contents of the field (up to a limit), with words
69 (non-space characters) separated by single space characters
70 (normalized to " " on display). When completeness is
71 disabled, each word is indexed as a separate entry. Complete subfield
72 indexing is most useful for fields which are typically browsed (eg.
73 titles, authors, or subjects), or instances where a match on a
74 complete subfield is essential (eg. exact title searching). For fields
75 where completeness is disabled, the search engine will interpret a
76 search containing space characters as a word proximity search.
78 </listitem></varlistentry>
80 <varlistentry id="default.idx.firstinfield">
81 <term>firstinfield <replaceable>boolean</replaceable></term>
84 This directive enables or disables first-in-field indexing.
85 The value of the <replaceable>boolean</replaceable> should be 0
88 </listitem></varlistentry>
90 <varlistentry id="default.idx.alwaysmatches">
91 <term>alwaysmatches <replaceable>boolean</replaceable></term>
94 This directive enables or disables alwaysmatches indexing.
95 The value of the <replaceable>boolean</replaceable> should be 0
98 </listitem></varlistentry>
101 <term>charmap <replaceable>filename</replaceable></term>
104 This is the filename of the character
105 map to be used for this index for field type.
107 </listitem></varlistentry>
112 <section id="character-map-files">
113 <title>The character map file format</title>
115 The contents of the character map files are structured as follows:
122 <term>lowercase <replaceable>value-set</replaceable></term>
125 This directive introduces the basic value set of the field type.
126 The format is an ordered list (without spaces) of the
127 characters which may occur in "words" of the given type.
128 The order of the entries in the list determines the
129 sort order of the index. In addition to single characters, the
130 following combinations are legal:
138 Backslashes may be used to introduce three-digit octal, or
139 two-digit hex representations of single characters
140 (preceded by <literal>x</literal>).
141 In addition, the combinations
142 \\, \\r, \\n, \\t, \\s (space — remember that real
143 space-characters may not occur in the value definition), and
144 \\ are recognized, with their usual interpretation.
150 Curly braces {} may be used to enclose ranges of single
151 characters (possibly using the escape convention described in the
152 preceding point), eg. {a-z} to introduce the
153 standard range of ASCII characters.
154 Note that the interpretation of such a range depends on
155 the concrete representation in your local, physical character set.
161 paranthesises () may be used to enclose multi-byte characters -
162 eg. diacritics or special national combinations (eg. Spanish
163 "ll"). When found in the input stream (or a search term),
164 these characters are viewed and sorted as a single character, with a
165 sorting value depending on the position of the group in the value
173 </listitem></varlistentry>
175 <term>uppercase <replaceable>value-set</replaceable></term>
178 This directive introduces the
179 upper-case equivalencis to the value set (if any). The number and
180 order of the entries in the list should be the same as in the
181 <literal>lowercase</literal> directive.
183 </listitem></varlistentry>
185 <term>space <replaceable>value-set</replaceable></term>
188 This directive introduces the character
189 which separate words in the input stream. Depending on the
190 completeness mode of the field in question, these characters either
191 terminate an index entry, or delimit individual "words" in
192 the input stream. The order of the elements is not significant —
193 otherwise the representation is the same as for the
194 <literal>uppercase</literal> and <literal>lowercase</literal>
197 </listitem></varlistentry>
199 <term>map <replaceable>value-set</replaceable>
200 <replaceable>target</replaceable></term>
203 This directive introduces a mapping between each of the
204 members of the value-set on the left to the character on the
205 right. The character on the right must occur in the value
206 set (the <literal>lowercase</literal> directive) of the
207 character set, but it may be a paranthesis-enclosed
208 multi-octet character. This directive may be used to map
209 diacritics to their base characters, or to map HTML-style
210 character-representations to their natural form, etc. The
211 map directive can also be used to ignore leading articles in
212 searching and/or sorting, and to perform other special
213 transformations. See section <xref
214 linkend="leading-articles"/>.
216 </listitem></varlistentry>
220 <section id="leading-articles">
221 <title>Ignoring leading articles</title>
223 In addition to specifying sort orders, space (blank) handling,
224 and upper/lowercase folding, you can also use the character map
225 files to make Zebra ignore leading articles in sorting records,
226 or when doing complete field searching.
229 This is done using the <literal>map</literal> directive in the
230 character map file. In a nutshell, what you do is map certain
231 sequences of characters, when they occur <emphasis> in the
232 beginning of a field</emphasis>, to a space. Assuming that the
233 character "@" is defined as a space character in your file, you
239 The effect of these directives is to map either 'the' or 'The',
240 followed by a space character, to a space. The hat ^ character
241 denotes beginning-of-field only when complete-subfield indexing
242 or sort indexing is taking place; otherwise, it is treated just
243 as any other character.
246 Because the <literal>default.idx</literal> file can be used to
247 associate different character maps with different indexing types
248 -- and you can create additional indexing types, should the need
249 arise -- it is possible to specify that leading articles should
250 be ignored either in sorting, in complete-field searching, or
254 If you ignore certain prefixes in sorting, then these will be
255 eliminated from the index, and sorting will take place as if
256 they weren't there. However, if you set the system up to ignore
257 certain prefixes in <emphasis>searching</emphasis>, then these
258 are deleted both from the indexes and from query terms, when the
259 client specifies complete-field searching. This has the effect
260 that a search for 'the science journal' and 'science journal'
261 would both produce the same results.
265 <section id="default-idx-zebra">
266 <title>Accessing Zebra internal record data using
267 the <literal>zebra::</literal> element sets</title>
269 Starting with <literal>Zebra</literal> version
270 <literal>2.0.4-2</literal> or newer, one has the possibility to
272 <literal>zebra::data</literal>,
273 <literal>zebra::meta</literal> and
274 <literal>zebra::index</literal> element set names.
278 Usage of the <literal>zebra::</literal> element sets accesses
279 record data directly from the internal storage, and will
280 therefore work exactly the same way, irrespectively of indexing
284 These element set names are optimized for retrieval speed, and
285 will perform better than using for example
286 <literal>alvis</literal> filter XSLT based extraction of small
287 parts of the records.
291 For example, to fetch the raw binary record data stored in the
292 zebra internal storage, or on the filesystem, the following
293 commands can be issued:
295 Z> f @attr 1=title my
297 Z> elements zebra::data
308 <literal>zebra::data</literal> element set name is
309 defined for any record syntax, but will always fetch
310 the raw record data in exactly the original form. No record syntax
311 specific transformations will be applied to the raw record data.
315 Also, Zebra internal metadata about the record can be accessed:
317 Z> f @attr 1=title my
319 Z> elements zebra::meta::sysno
322 displays in <literal>XML</literal> record syntax only internal
323 record system number, whereas
325 Z> f @attr 1=title my
327 Z> elements zebra::meta
330 displays all available metadata on the record. These include sytem
331 number, database name, indexed filename, filter used for indexing,
332 score and static ranking information and finally bytesize of record.
337 <literal>zebra::meta</literal> element set names are only
339 <literal>SUTRS</literal> and <literal>XML</literal> record
344 Sometimes, it is very hard to figure out what exactly has been
345 indexed how and in which indexes. Using the indexing stylesheet of
346 the Alvis filter, one can at least see which portion of the record
347 went into which index, but a similar aid does not exist for all
348 other indexing filters.
352 <literal>zebra::index</literal> element set names are provided to
353 access information on per record indexed fields. For example, the
356 Z> f @attr 1=title my
358 Z> elements zebra::index
361 will display all indexed tokens from all indexed fields of the
362 first record, and it will display in <literal>SUTRS</literal>
363 record syntax, whereas
365 Z> f @attr 1=title my
367 Z> elements zebra::index::title
369 Z> elements zebra::index::title:p
372 displays in <literal>XML</literal> record syntax only the content
373 of the zebra string index <literal>title</literal>, or
374 even only the type <literal>p</literal> phrase indexed part of it.
378 The special <literal>zebra::index</literal>
379 element set names are only
381 <literal>SUTRS</literal> and <literal>XML</literal> record
384 <para> Trying to access numeric <literal>Bib-1</literal> use
385 attributes or trying to access non-existent zebra intern string
386 access points will result in a
388 Diagnostic [25]: Specified element set name not valid for specified database
394 <!-- Keep this comment at the end of the file
399 sgml-minimize-attributes:nil
400 sgml-always-quote-attributes:t
403 sgml-parent-document: "zebra.xml"
404 sgml-local-catalogs: nil
405 sgml-namecase-general:t