1 <chapter id="fields-and-charsets">
2 <title>Field Structure and Character Sets
6 In order to provide a flexible approach to national character set
7 handling, &zebra; allows the administrator to configure the set up the
8 system to handle any 8-bit character set — including sets that
9 require multi-octet diacritics or other multi-octet characters. The
10 definition of a character set includes a specification of the
11 permissible values, their sort order (this affects the display in the
12 SCAN function), and relationships between upper- and lowercase
13 characters. Finally, the definition includes the specification of
14 space characters for the set.
18 The operator can define different character sets for different fields,
19 typical examples being standard text fields, numerical fields, and
20 special-purpose fields such as WWW-style linkages (URx).
24 Zebra 1.3 and Zebra versions 2.0.18 and earlier required that the field
25 type is a single character, e.g. <literal>w</literal> (for word), and
26 <literal>p</literal> for phrase. Zebra 2.0.20 and later allow field types
27 to be any string. This allows for greater flexibility - in particular
28 per-locale (language) fields can be defined.
32 Version 2.0.20 of Zebra can also be configured - per field - to use the
33 <ulink url="&url.icu;">ICU</ulink> library to perform tokenization and
34 normalization of strings. This is an alternative to the "charmap"
35 files which has been part of Zebra since its first release.
38 <section id="default-idx-file">
39 <title>The default.idx file</title>
41 The field types, and hence character sets, are associated with data
42 elements by the indexing rules (say <literal>title:w</literal>) in the
43 various filters. Fields are defined in a field definition file which,
44 by default, is called <filename>default.idx</filename>.
45 This file provides the association between field type codes
46 and the character map files (with the .chr suffix). The format
47 of the .idx file is as follows
54 <term>index <replaceable>field type code</replaceable></term>
57 This directive introduces a new search index code.
58 The argument is a one-character code to be used in the
59 .abs files to select this particular index type. An index, roughly,
60 corresponds to a particular structure attribute during search. Refer
61 to <xref linkend="zebrasrv-search"/>.
63 </listitem></varlistentry>
65 <term>sort <replaceable>field code type</replaceable></term>
68 This directive introduces a
69 sort index. The argument is a one-character code to be used in the
70 .abs fie to select this particular index type. The corresponding
71 use attribute must be used in the sort request to refer to this
72 particular sort index. The corresponding character map (see below)
73 is used in the sort process.
75 </listitem></varlistentry>
77 <term>completeness <replaceable>boolean</replaceable></term>
80 This directive enables or disables complete field indexing.
81 The value of the <replaceable>boolean</replaceable> should be 0
82 (disable) or 1. If completeness is enabled, the index entry will
83 contain the complete contents of the field (up to a limit), with words
84 (non-space characters) separated by single space characters
85 (normalized to " " on display). When completeness is
86 disabled, each word is indexed as a separate entry. Complete subfield
87 indexing is most useful for fields which are typically browsed (e.g.,
88 titles, authors, or subjects), or instances where a match on a
89 complete subfield is essential (e.g., exact title searching). For fields
90 where completeness is disabled, the search engine will interpret a
91 search containing space characters as a word proximity search.
93 </listitem></varlistentry>
95 <varlistentry id="default.idx.firstinfield">
96 <term>firstinfield <replaceable>boolean</replaceable></term>
99 This directive enables or disables first-in-field indexing.
100 The value of the <replaceable>boolean</replaceable> should be 0
103 </listitem></varlistentry>
105 <varlistentry id="default.idx.alwaysmatches">
106 <term>alwaysmatches <replaceable>boolean</replaceable></term>
109 This directive enables or disables alwaysmatches indexing.
110 The value of the <replaceable>boolean</replaceable> should be 0
113 </listitem></varlistentry>
116 <term>charmap <replaceable>filename</replaceable></term>
119 This is the filename of the character
120 map to be used for this index for field type.
121 See <xref linkend="character-map-files"/> for details.
123 </listitem></varlistentry>
126 <term>icuchain <replaceable>filename</replaceable></term>
129 Specifies the filename with ICU tokenization and
131 See <xref linkend="icuchain-files"/> for details.
132 Using icuchain for a field type is an alternative to
133 charmap. It does not make sense to define both
134 icuchain and charmap for the same field type.
136 </listitem></varlistentry>
139 <example id="field-types">
140 <title>Field types</title>
142 Following are three excerpts of the standard
143 <filename>tab/default.idx</filename> configuration file. Notice
144 that the <literal>index</literal> and <literal>sort</literal>
145 are grouping directives, which bind all other following directives
148 # Traditional word index
149 # Used if completeness is 'incomplete field' (@attr 6=1) and
150 # structure is word/phrase/word-list/free-form-text/document-text
160 # Null map index (no mapping at all)
161 # Used if structure=key (@attr 4=3)
178 <section id="character-map-files">
179 <title>Charmap Files</title>
181 The character map files are used to define the word tokenization
182 and character normalization performed before inserting text into
183 the inverse indexes. &zebra; ships with the predefined character map
184 files <filename>tab/*.chr</filename>. Users are allowed to add
185 and/or modify maps according to their needs.
188 <table id="character-map-table" frame="top">
189 <title>Character maps predefined in &zebra;</title>
193 <entry>File name</entry>
194 <entry>Intended type</entry>
195 <entry>Description</entry>
200 <entry><literal>numeric.chr</literal></entry>
201 <entry><literal>:n</literal></entry>
202 <entry>Numeric digit tokenization and normalization map. All
203 characters not in the set <literal>-{0-9}.,</literal> will be
204 suppressed. Note that floating point numbers are processed
205 fine, but scientific exponential numbers are trashed.</entry>
208 <entry><literal>scan.chr</literal></entry>
209 <entry><literal>:w or :p</literal></entry>
210 <entry>Word tokenization char map for Scandinavian
211 languages. This one resembles the generic word tokenization
212 character map <literal>tab/string.chr</literal>, the main
213 differences are sorting of the special characters
214 <literal>üzæäøöå</literal> and equivalence maps according to
215 Scandinavian language rules.</entry>
218 <entry><literal>string.chr</literal></entry>
219 <entry><literal>:w or :p</literal></entry>
220 <entry>General word tokenization and normalization character
221 map, mostly useful for English texts. Use this to derive your
222 own language tokenization and normalization derivatives.</entry>
225 <entry><literal>urx.chr</literal></entry>
226 <entry><literal>:u</literal></entry>
227 <entry>URL parsing and tokenization character map.</entry>
230 <entry><literal>@</literal></entry>
231 <entry><literal>:0</literal></entry>
232 <entry>Do-nothing character map used for literal binary
233 indexing. There is no existing file associated to it, and
234 there is no normalization or tokenization performed at all.</entry>
241 The contents of the character map files are structured as follows:
244 <term>encoding <replaceable>encoding-name</replaceable></term>
247 This directive must be at the very beginning of the file, and it
248 specifies the character encoding used in the entire file. If
249 omitted, the encoding <literal>ISO-8859-1</literal> is assumed.
252 For example, one of the test files found at
253 <literal>test/rusmarc/tab/string.chr</literal> contains the following
259 <literal>test/charmap/string.utf8.chr</literal> is encoded
265 </listitem></varlistentry>
268 <term>lowercase <replaceable>value-set</replaceable></term>
271 This directive introduces the basic value set of the field type.
272 The format is an ordered list (without spaces) of the
273 characters which may occur in "words" of the given type.
274 The order of the entries in the list determines the
275 sort order of the index. In addition to single characters, the
276 following combinations are legal:
284 Backslashes may be used to introduce three-digit octal, or
285 two-digit hex representations of single characters
286 (preceded by <literal>x</literal>).
287 In addition, the combinations
288 \\, \\r, \\n, \\t, \\s (space — remember that real
289 space-characters may not occur in the value definition), and
290 \\ are recognized, with their usual interpretation.
296 Curly braces {} may be used to enclose ranges of single
297 characters (possibly using the escape convention described in the
298 preceding point), e.g., {a-z} to introduce the
299 standard range of ASCII characters.
300 Note that the interpretation of such a range depends on
301 the concrete representation in your local, physical character set.
307 parentheses () may be used to enclose multi-byte characters -
308 e.g., diacritics or special national combinations (e.g., Spanish
309 "ll"). When found in the input stream (or a search term),
310 these characters are viewed and sorted as a single character, with a
311 sorting value depending on the position of the group in the value
320 For example, <literal>scan.chr</literal> contains the following
321 lowercase normalization and sorting order:
323 lowercase {0-9}{a-y}üzæäøöå
326 </listitem></varlistentry>
328 <term>uppercase <replaceable>value-set</replaceable></term>
331 This directive introduces the
332 upper-case equivalences to the value set (if any). The number and
333 order of the entries in the list should be the same as in the
334 <literal>lowercase</literal> directive.
337 For example, <literal>scan.chr</literal> contains the following
338 uppercase equivalent:
340 uppercase {0-9}{A-Y}ÜZÆÄØÖÅ
343 </listitem></varlistentry>
345 <term>space <replaceable>value-set</replaceable></term>
348 This directive introduces the character
349 which separate words in the input stream. Depending on the
350 completeness mode of the field in question, these characters either
351 terminate an index entry, or delimit individual "words" in
352 the input stream. The order of the elements is not significant —
353 otherwise the representation is the same as for the
354 <literal>uppercase</literal> and <literal>lowercase</literal>
358 For example, <literal>scan.chr</literal> contains the following
361 space {\001-\040}!"#$%&'\()*+,-./:;<=>?@\[\\]^_`\{|}~
364 </listitem></varlistentry>
366 <term>map <replaceable>value-set</replaceable>
367 <replaceable>target</replaceable></term>
370 This directive introduces a mapping between each of the
371 members of the value-set on the left to the character on the
372 right. The character on the right must occur in the value
373 set (the <literal>lowercase</literal> directive) of the
374 character set, but it may be a parenthesis-enclosed
375 multi-octet character. This directive may be used to map
376 diacritics to their base characters, or to map HTML-style
377 character-representations to their natural form, etc. The
378 map directive can also be used to ignore leading articles in
379 searching and/or sorting, and to perform other special
383 For example, <literal>scan.chr</literal> contains the following
384 map instructions among others, to make sure that HTML entity
385 encoded Danish special characters are mapped to the
386 equivalent Latin-1 characters:
394 In addition to specifying sort orders, space (blank) handling,
395 and upper/lowercase folding, you can also use the character map
396 files to make &zebra; ignore leading articles in sorting records,
397 or when doing complete field searching.
400 This is done using the <literal>map</literal> directive in the
401 character map file. In a nutshell, what you do is map certain
402 sequences of characters, when they occur <emphasis> in the
403 beginning of a field</emphasis>, to a space. Assuming that the
404 character "@" is defined as a space character in your file, you
410 The effect of these directives is to map either 'the' or 'The',
411 followed by a space character, to a space. The hat ^ character
412 denotes beginning-of-field only when complete-subfield indexing
413 or sort indexing is taking place; otherwise, it is treated just
414 as any other character.
417 Because the <literal>default.idx</literal> file can be used to
418 associate different character maps with different indexing types
419 -- and you can create additional indexing types, should the need
420 arise -- it is possible to specify that leading articles should
421 be ignored either in sorting, in complete-field searching, or
425 If you ignore certain prefixes in sorting, then these will be
426 eliminated from the index, and sorting will take place as if
427 they weren't there. However, if you set the system up to ignore
428 certain prefixes in <emphasis>searching</emphasis>, then these
429 are deleted both from the indexes and from query terms, when the
430 client specifies complete-field searching. This has the effect
431 that a search for 'the science journal' and 'science journal'
432 would both produce the same results.
434 </listitem></varlistentry>
436 <term>equivalent <replaceable>value-set</replaceable></term>
439 This directive introduces equivalence classes of strings for
440 searching purposes only. It's a one-to-many
441 conversion that takes place only during search before the map
451 a search for the <literal>äsel</literal> will be be match any of
452 <literal>æsel</literal>, <literal>äsel</literal> and
453 <literal>aesel</literal>.
455 </listitem></varlistentry>
460 <section id="icuchain-files">
461 <title>ICU Chain Files</title>
463 The <ulink url="&url.icu;">ICU</ulink> chain files defines a
464 <emphasis>chain</emphasis> of rules
465 which specify the conversion process to be carried out for each
466 record string for indexing.
469 Both searching and sorting is based on the <emphasis>sort</emphasis>
470 normalization that ICU provides. This means that scan and sort will
471 return terms in the sort order given by ICU.
474 Zebra is using YAZ' ICU wrapper. Refer to the
475 <ulink url="&url.yaz.yaz-icu;">yaz-icu man page</ulink> for
476 documentation about the ICU chain rules.
480 Use the yaz-icu program to test your icuchain rules.
483 <example id="indexing-greek-example"><title>Indexing Greek text</title>
485 Consider a system where all "regular" text is to be indexed
486 using as Greek (locale: EL).
487 We would have to change our index type file - to read
498 The ICU chain file <filename>greek.xml</filename> could look
501 <icu_chain locale="el">
502 <transform rule="[:Control:] Any-Remove"/>
504 <transform rule="[[:WhiteSpace:][:Punctuation:]] Remove"/>
512 Zebra is shipped with a field types file <filename>icu.idx</filename>
513 which is an ICU chain version of <filename>default.idx</filename>.
516 <example id="indexing-marcxml-example"><title>MARCXML indexing using ICU</title>
518 The directory <filename>examples/marcxml</filename> includes
519 a complete sample with MARCXML records that are DOM XML indexed
520 using ICU chain rules. Study the
521 <filename>README</filename> in the <filename>marcxml</filename>
522 directory for details.
528 <!-- Keep this comment at the end of the file
533 sgml-minimize-attributes:nil
534 sgml-always-quote-attributes:t
537 sgml-parent-document: "idzebra.xml"
538 sgml-local-catalogs: nil
539 sgml-namecase-general:t