1 <chapter id="fields-and-charsets">
2 <!-- $Id: field-structure.xml,v 1.10 2006-11-28 14:37:45 marc Exp $ -->
3 <title>Field Structure and Character Sets
7 In order to provide a flexible approach to national character set
8 handling, Zebra allows the administrator to configure the set up the
9 system to handle any 8-bit character set — including sets that
10 require multi-octet diacritics or other multi-octet characters. The
11 definition of a character set includes a specification of the
12 permissible values, their sort order (this affects the display in the
13 SCAN function), and relationships between upper- and lowercase
14 characters. Finally, the definition includes the specification of
15 space characters for the set.
19 The operator can define different character sets for different fields,
20 typical examples being standard text fields, numerical fields, and
21 special-purpose fields such as WWW-style linkages (URx).
24 <section id="default-idx-file">
25 <title>The default.idx file</title>
27 The field types, and hence character sets, are associated with data
28 elements by the .abs files (see above).
29 The file <literal>default.idx</literal>
30 provides the association between field type codes (as used in the .abs
31 files) and the character map files (with the .chr suffix). The format
32 of the .idx file is as follows
39 <term>index <replaceable>field type code</replaceable></term>
42 This directive introduces a new search index code.
43 The argument is a one-character code to be used in the
44 .abs files to select this particular index type. An index, roughly,
45 corresponds to a particular structure attribute during search. Refer
46 to <xref linkend="zebrasrv-search"/>.
48 </listitem></varlistentry>
50 <term>sort <replaceable>field code type</replaceable></term>
53 This directive introduces a
54 sort index. The argument is a one-character code to be used in the
55 .abs fie to select this particular index type. The corresponding
56 use attribute must be used in the sort request to refer to this
57 particular sort index. The corresponding character map (see below)
58 is used in the sort process.
60 </listitem></varlistentry>
62 <term>completeness <replaceable>boolean</replaceable></term>
65 This directive enables or disables complete field indexing.
66 The value of the <replaceable>boolean</replaceable> should be 0
67 (disable) or 1. If completeness is enabled, the index entry will
68 contain the complete contents of the field (up to a limit), with words
69 (non-space characters) separated by single space characters
70 (normalized to " " on display). When completeness is
71 disabled, each word is indexed as a separate entry. Complete subfield
72 indexing is most useful for fields which are typically browsed (eg.
73 titles, authors, or subjects), or instances where a match on a
74 complete subfield is essential (eg. exact title searching). For fields
75 where completeness is disabled, the search engine will interpret a
76 search containing space characters as a word proximity search.
78 </listitem></varlistentry>
80 <varlistentry id="default.idx.firstinfield">
81 <term>firstinfield <replaceable>boolean</replaceable></term>
84 This directive enables or disables first-in-field indexing.
85 The value of the <replaceable>boolean</replaceable> should be 0
88 </listitem></varlistentry>
90 <varlistentry id="default.idx.alwaysmatches">
91 <term>alwaysmatches <replaceable>boolean</replaceable></term>
94 This directive enables or disables alwaysmatches indexing.
95 The value of the <replaceable>boolean</replaceable> should be 0
98 </listitem></varlistentry>
101 <term>charmap <replaceable>filename</replaceable></term>
104 This is the filename of the character
105 map to be used for this index for field type.
106 See <xref linkend="character-map-files"/> for details.
108 </listitem></varlistentry>
112 Following are three excerpts of the standard
113 <filename>tab/default.idx</filename> configuration file. Notice
114 that the <literal>index</literal> and <literal>sort</literal>
115 are grouping directives, which bind all other following directives
118 # Traditional word index
119 # Used if completenss is 'incomplete field' (@attr 6=1) and
120 # structure is word/phrase/word-list/free-form-text/document-text
130 # Null map index (no mapping at all)
131 # Used if structure=key (@attr 4=3)
147 <section id="character-map-files">
148 <title>The character map file format</title>
150 The character map files are used to define the word tokenization
151 and character normalization performed before inserting text into
152 the inverse indexes. Zebra ships with the predefined character map
153 files <filename>tab/*.chr</filename>. Users are allowed to add
154 and/or modify maps according to their needs.
157 <table id="querymodel-attribute-sets-table" frame="top">
158 <title>Character maps predefined in Zebra</title>
162 <entry>File name</entry>
163 <entry>Intended type</entry>
164 <entry>Description</entry>
169 <entry><literal>numeric.chr</literal></entry>
170 <entry><literal>:n</literal></entry>
171 <entry>Numeric digit tokenization and normalization map. All
172 characters not in the set <literal>-{0-9}.,</literal> will be
173 suppressed. Note that floating point numbers are processed
174 fine, but scientific exponential numbers are trashed.</entry>
177 <entry><literal>scan.chr</literal></entry>
178 <entry><literal>:w or :p</literal></entry>
179 <entry>Word tokenization char map for Scandinavian
180 languages. This one resembles the generic word tokenization
181 character map <literal>tab/string.chr</literal>, the main
182 differences are sorting of the special characters
183 <literal>üzæäøöå</literal> and equivalence maps according to
184 Scandinavian language rules.</entry>
187 <entry><literal>string.chr</literal></entry>
188 <entry><literal>:w or :p</literal></entry>
189 <entry>General word tokenization and normalization character
190 map, mostly useful for English texts. Use this to derive your
191 own language tokenization and normalization derivatives.</entry>
194 <entry><literal>urx.chr</literal></entry>
195 <entry><literal>:u</literal></entry>
196 <entry>URL parsing and tokenization character map.</entry>
199 <entry><literal>@</literal></entry>
200 <entry><literal>:0</literal></entry>
201 <entry>Do-nothing character map used for literal binary
202 indexing. There is no existing file associated to it, and
203 there is no normalization or tokenization performed at all.</entry>
210 The contents of the character map files are structured as follows:
213 <term>encoding <replaceable>encoding-name</replaceable></term>
216 This directive must be at the very beginning of the file, and it
217 specifies the character encoding used in the entire file. If
218 omitted, the encoding <literal>ISO-8859-1</literal> is assumed.
221 For example, one of the test files found at
222 <literal>test/rusmarc/tab/string.chr</literal> contains the following
228 <literal>test/charmap/string.utf8.chr</literal> is encoded
234 </listitem></varlistentry>
237 <term>lowercase <replaceable>value-set</replaceable></term>
240 This directive introduces the basic value set of the field type.
241 The format is an ordered list (without spaces) of the
242 characters which may occur in "words" of the given type.
243 The order of the entries in the list determines the
244 sort order of the index. In addition to single characters, the
245 following combinations are legal:
253 Backslashes may be used to introduce three-digit octal, or
254 two-digit hex representations of single characters
255 (preceded by <literal>x</literal>).
256 In addition, the combinations
257 \\, \\r, \\n, \\t, \\s (space — remember that real
258 space-characters may not occur in the value definition), and
259 \\ are recognized, with their usual interpretation.
265 Curly braces {} may be used to enclose ranges of single
266 characters (possibly using the escape convention described in the
267 preceding point), eg. {a-z} to introduce the
268 standard range of ASCII characters.
269 Note that the interpretation of such a range depends on
270 the concrete representation in your local, physical character set.
276 paranthesises () may be used to enclose multi-byte characters -
277 eg. diacritics or special national combinations (eg. Spanish
278 "ll"). When found in the input stream (or a search term),
279 these characters are viewed and sorted as a single character, with a
280 sorting value depending on the position of the group in the value
289 For example, <literal>scan.chr</literal> contains the following
290 lowercase normalization and sorting order:
292 lowercase {0-9}{a-y}üzæäøöå
295 </listitem></varlistentry>
297 <term>uppercase <replaceable>value-set</replaceable></term>
300 This directive introduces the
301 upper-case equivalences to the value set (if any). The number and
302 order of the entries in the list should be the same as in the
303 <literal>lowercase</literal> directive.
306 For example, <literal>scan.chr</literal> contains the following
307 uppercase equivalent:
309 uppercase {0-9}{A-Y}ÜZÆÄØÖÅ
312 </listitem></varlistentry>
314 <term>space <replaceable>value-set</replaceable></term>
317 This directive introduces the character
318 which separate words in the input stream. Depending on the
319 completeness mode of the field in question, these characters either
320 terminate an index entry, or delimit individual "words" in
321 the input stream. The order of the elements is not significant —
322 otherwise the representation is the same as for the
323 <literal>uppercase</literal> and <literal>lowercase</literal>
327 For example, <literal>scan.chr</literal> contains the following
330 space {\001-\040}!"#$%&'\()*+,-./:;<=>?@\[\\]^_`\{|}~
333 </listitem></varlistentry>
335 <term>map <replaceable>value-set</replaceable>
336 <replaceable>target</replaceable></term>
339 This directive introduces a mapping between each of the
340 members of the value-set on the left to the character on the
341 right. The character on the right must occur in the value
342 set (the <literal>lowercase</literal> directive) of the
343 character set, but it may be a parenthesis-enclosed
344 multi-octet character. This directive may be used to map
345 diacritics to their base characters, or to map HTML-style
346 character-representations to their natural form, etc. The
347 map directive can also be used to ignore leading articles in
348 searching and/or sorting, and to perform other special
349 transformations. See section <xref
350 linkend="leading-articles"/>.
353 For example, <literal>scan.chr</literal> contains the following
354 map instructions among others, to make sure that HTML entity
355 encoded Danish special characters are mapped to the
356 equivalent Latin-1 characters:
363 </listitem></varlistentry>
365 <term>equivalent <replaceable>value-set</replaceable></term>
368 This directive introduces equivalence classes of characters
369 and/or strings for sorting purposes only. It resembles the map
370 directive, but does not affect search and retrieval indexing,
371 but only sorting order under present requests.
374 For example, <literal>scan.chr</literal> contains the following
375 equivalent sorting instructions, which can be uncommented:
383 </listitem></varlistentry>
387 <section id="leading-articles">
388 <title>Ignoring leading articles</title>
390 In addition to specifying sort orders, space (blank) handling,
391 and upper/lowercase folding, you can also use the character map
392 files to make Zebra ignore leading articles in sorting records,
393 or when doing complete field searching.
396 This is done using the <literal>map</literal> directive in the
397 character map file. In a nutshell, what you do is map certain
398 sequences of characters, when they occur <emphasis> in the
399 beginning of a field</emphasis>, to a space. Assuming that the
400 character "@" is defined as a space character in your file, you
406 The effect of these directives is to map either 'the' or 'The',
407 followed by a space character, to a space. The hat ^ character
408 denotes beginning-of-field only when complete-subfield indexing
409 or sort indexing is taking place; otherwise, it is treated just
410 as any other character.
413 Because the <literal>default.idx</literal> file can be used to
414 associate different character maps with different indexing types
415 -- and you can create additional indexing types, should the need
416 arise -- it is possible to specify that leading articles should
417 be ignored either in sorting, in complete-field searching, or
421 If you ignore certain prefixes in sorting, then these will be
422 eliminated from the index, and sorting will take place as if
423 they weren't there. However, if you set the system up to ignore
424 certain prefixes in <emphasis>searching</emphasis>, then these
425 are deleted both from the indexes and from query terms, when the
426 client specifies complete-field searching. This has the effect
427 that a search for 'the science journal' and 'science journal'
428 would both produce the same results.
433 <!-- Keep this comment at the end of the file
438 sgml-minimize-attributes:nil
439 sgml-always-quote-attributes:t
442 sgml-parent-document: "zebra.xml"
443 sgml-local-catalogs: nil
444 sgml-namecase-general:t