1 <chapter id="fields-and-charsets">
2 <!-- $Id: field-structure.xml,v 1.2 2006-09-05 12:01:31 adam Exp $ -->
3 <title>Field Structure and Character Sets
7 In order to provide a flexible approach to national character set
8 handling, Zebra allows the administrator to configure the set up the
9 system to handle any 8-bit character set — including sets that
10 require multi-octet diacritics or other multi-octet characters. The
11 definition of a character set includes a specification of the
12 permissible values, their sort order (this affects the display in the
13 SCAN function), and relationships between upper- and lowercase
14 characters. Finally, the definition includes the specification of
15 space characters for the set.
19 The operator can define different character sets for different fields,
20 typical examples being standard text fields, numerical fields, and
21 special-purpose fields such as WWW-style linkages (URx).
24 <section id="default-idx-file">
25 <title>The default.idx file</title>
27 The field types, and hence character sets, are associated with data
28 elements by the .abs files (see above).
29 The file <literal>default.idx</literal>
30 provides the association between field type codes (as used in the .abs
31 files) and the character map files (with the .chr suffix). The format
32 of the .idx file is as follows
39 <term>index <replaceable>field type code</replaceable></term>
42 This directive introduces a new search index code.
43 The argument is a one-character code to be used in the
44 .abs files to select this particular index type. An index, roughly,
45 corresponds to a particular structure attribute during search. Refer
46 to <xref linkend="zebrasrv-search"/>.
48 </listitem></varlistentry>
50 <term>sort <replaceable>field code type</replaceable></term>
53 This directive introduces a
54 sort index. The argument is a one-character code to be used in the
55 .abs fie to select this particular index type. The corresponding
56 use attribute must be used in the sort request to refer to this
57 particular sort index. The corresponding character map (see below)
58 is used in the sort process.
60 </listitem></varlistentry>
62 <term>completeness <replaceable>boolean</replaceable></term>
65 This directive enables or disables complete field indexing.
66 The value of the <replaceable>boolean</replaceable> should be 0
67 (disable) or 1. If completeness is enabled, the index entry will
68 contain the complete contents of the field (up to a limit), with words
69 (non-space characters) separated by single space characters
70 (normalized to " " on display). When completeness is
71 disabled, each word is indexed as a separate entry. Complete subfield
72 indexing is most useful for fields which are typically browsed (eg.
73 titles, authors, or subjects), or instances where a match on a
74 complete subfield is essential (eg. exact title searching). For fields
75 where completeness is disabled, the search engine will interpret a
76 search containing space characters as a word proximity search.
78 </listitem></varlistentry>
80 <term>charmap <replaceable>filename</replaceable></term>
83 This is the filename of the character
84 map to be used for this index for field type.
86 </listitem></varlistentry>
91 <section id="character-map-files">
92 <title>The character map file format</title>
94 The contents of the character map files are structured as follows:
101 <term>lowercase <replaceable>value-set</replaceable></term>
104 This directive introduces the basic value set of the field type.
105 The format is an ordered list (without spaces) of the
106 characters which may occur in "words" of the given type.
107 The order of the entries in the list determines the
108 sort order of the index. In addition to single characters, the
109 following combinations are legal:
117 Backslashes may be used to introduce three-digit octal, or
118 two-digit hex representations of single characters
119 (preceded by <literal>x</literal>).
120 In addition, the combinations
121 \\, \\r, \\n, \\t, \\s (space — remember that real
122 space-characters may not occur in the value definition), and
123 \\ are recognized, with their usual interpretation.
129 Curly braces {} may be used to enclose ranges of single
130 characters (possibly using the escape convention described in the
131 preceding point), eg. {a-z} to introduce the
132 standard range of ASCII characters.
133 Note that the interpretation of such a range depends on
134 the concrete representation in your local, physical character set.
140 paranthesises () may be used to enclose multi-byte characters -
141 eg. diacritics or special national combinations (eg. Spanish
142 "ll"). When found in the input stream (or a search term),
143 these characters are viewed and sorted as a single character, with a
144 sorting value depending on the position of the group in the value
152 </listitem></varlistentry>
154 <term>uppercase <replaceable>value-set</replaceable></term>
157 This directive introduces the
158 upper-case equivalencis to the value set (if any). The number and
159 order of the entries in the list should be the same as in the
160 <literal>lowercase</literal> directive.
162 </listitem></varlistentry>
164 <term>space <replaceable>value-set</replaceable></term>
167 This directive introduces the character
168 which separate words in the input stream. Depending on the
169 completeness mode of the field in question, these characters either
170 terminate an index entry, or delimit individual "words" in
171 the input stream. The order of the elements is not significant —
172 otherwise the representation is the same as for the
173 <literal>uppercase</literal> and <literal>lowercase</literal>
176 </listitem></varlistentry>
178 <term>map <replaceable>value-set</replaceable>
179 <replaceable>target</replaceable></term>
182 This directive introduces a mapping between each of the
183 members of the value-set on the left to the character on the
184 right. The character on the right must occur in the value
185 set (the <literal>lowercase</literal> directive) of the
186 character set, but it may be a paranthesis-enclosed
187 multi-octet character. This directive may be used to map
188 diacritics to their base characters, or to map HTML-style
189 character-representations to their natural form, etc. The
190 map directive can also be used to ignore leading articles in
191 searching and/or sorting, and to perform other special
192 transformations. See section <xref
193 linkend="leading-articles"/>.
195 </listitem></varlistentry>
199 <section id="leading-articles">
200 <title>Ignoring leading articles</title>
202 In addition to specifying sort orders, space (blank) handling,
203 and upper/lowercase folding, you can also use the character map
204 files to make Zebra ignore leading articles in sorting records,
205 or when doing complete field searching.
208 This is done using the <literal>map</literal> directive in the
209 character map file. In a nutshell, what you do is map certain
210 sequences of characters, when they occur <emphasis> in the
211 beginning of a field</emphasis>, to a space. Assuming that the
212 character "@" is defined as a space character in your file, you
218 The effect of these directives is to map either 'the' or 'The',
219 followed by a space character, to a space. The hat ^ character
220 denotes beginning-of-field only when complete-subfield indexing
221 or sort indexing is taking place; otherwise, it is treated just
222 as any other character.
225 Because the <literal>default.idx</literal> file can be used to
226 associate different character maps with different indexing types
227 -- and you can create additional indexing types, should the need
228 arise -- it is possible to specify that leading articles should
229 be ignored either in sorting, in complete-field searching, or
233 If you ignore certain prefixes in sorting, then these will be
234 eliminated from the index, and sorting will take place as if
235 they weren't there. However, if you set the system up to ignore
236 certain prefixes in <emphasis>searching</emphasis>, then these
237 are deleted both from the indexes and from query terms, when the
238 client specifies complete-field searching. This has the effect
239 that a search for 'the science journal' and 'science journal'
240 would both produce the same results.
244 <!-- Keep this comment at the end of the file
249 sgml-minimize-attributes:nil
250 sgml-always-quote-attributes:t
253 sgml-parent-document: "zebra.xml"
254 sgml-local-catalogs: nil
255 sgml-namecase-general:t