(require-extension utf8)
iset and syntax-case
USAGE
To make your code Unicode aware, just do the following:
(require-extension utf8)
(module mymodule ()
(import utf8)
... ; unicode-aware code
)
then all core, extra and regex string operations will be
Unicode aware. string-length will return the number of codepoints,
not the number of bytes, string-ref will index by codepoints and
return a char with an integer value up to 2^21, regular expressions
will match single codepoints rather than bytes and understand Unicode
character classes, etc.
Strings are still native strings and may be passed to external libraries (either Scheme or foreign) perfectly safely. Libraries that do parsing invariably do so on ASCII character boundaries and are thus guaranteed to be compatible. Libraries that reference strings by index would need to be modified with a UTF-8 version. Currently all existing eggs are UTF-8 safe to my knowledge.
Alternately, you may import utf8 at the top-level:
; require modules using byte-semantics ... (require-extension utf8) (import utf8) ... ; require modules using utf8-semantics ... ; unicode-aware code
By importing directly into the top-level, any subsequently loaded code will also use Unicode-aware semantics, even if it was not written with Unicode in mind. This is more powerful but slightly less safe, since third party units may make assumptions about character ranges or string size.
To use Unicode-aware SRFI-13 and SRFI-14 using UTF-8 semantics:
(require-extension utf8-srfi-13)
(module ()
(import utf8-srfi-13)
... ; unicode-aware SRFI-13
)
(require-extension utf8-srfi-14)
(module ()
(import utf8-srfi-14)
... ; unicode-capable SRFI-14
)
The SRFI-14 module provides an alternative to the standard Chicken SRFI-14. As a pure superset which handles arbitrary-sized characters it should be usable as a drop-in replacement. The only aspect related to UTF-8 is STRING->CHAR-SET assumes the string is UTF-8 encoded.
UNICODE CHAR-SETS
The default SRFI-14 char-sets are defined using ASCII-only characters, since this is both useful and lighter-weight. To obtain full Unicode char-set definitions, use the char-set unit:
(require-extension char-set)
The following char-sets are provided based on the Unicode properties:
char-set:alphabetic char-set:arabic char-set:armenian char-set:ascii-hex-digit char-set:bengali char-set:bidi-control char-set:bopomofo char-set:braille char-set:buhid char-set:canadian-aboriginal char-set:cherokee char-set:common char-set:cypriot char-set:cyrillic char-set:dash char-set:default-ignorable-code-point char-set:deprecated char-set:deseret char-set:devanagari char-set:diacritic char-set:ethiopic char-set:extender char-set:georgian char-set:gothic char-set:grapheme-base char-set:grapheme-extend char-set:grapheme-link char-set:greek char-set:gujarati char-set:gurmukhi char-set:han char-set:hangul char-set:hanunoo char-set:hebrew char-set:hex-digit char-set:hiragana char-set:hyphen char-set:id-continue char-set:id-start char-set:ideographic char-set:ids-binary-operator char-set:ids-trinary-operator char-set:inherited char-set:join-control char-set:kannada char-set:katakana char-set:katakana-or-hiragana char-set:khmer char-set:lao char-set:latin char-set:limbu char-set:linear-b char-set:logical-order-exception char-set:lowercase char-set:malayalam char-set:math char-set:mongolian char-set:myanmar char-set:noncharacter-code-point char-set:ogham char-set:old-italic char-set:oriya char-set:osmanya char-set:quotation-mark char-set:radical char-set:runic char-set:shavian char-set:sinhala char-set:soft-dotted char-set:sterm char-set:syriac char-set:tagalog char-set:tagbanwa char-set:tai-le char-set:tamil char-set:telugu char-set:terminal-punctuation char-set:thaana char-set:thai char-set:tibetan char-set:ugaritic char-set:unified-ideograph char-set:uppercase char-set:variation-selector char-set:white-space char-set:xid-continue char-set:xid-start char-set:yi
UNICODE CASE-MAPPINGS
The SRFI-13 case-mapping procedures (string-upcase, etc.) are defined
using only ASCII case-mappings, since this is both useful and
lighter-weight. To get full Unicode aware case-mappings, do
(require-extension case-map)
which provides the upcase, downcase, and titlecase procedures. These take a first argument of either a string or port, and an optional second argument of locale (as a string), returning the appropriate locale-aware case-mapped string.
BYTE-STRINGS
Sometimes you may need access to the original string primitives so
you can directly access bytes, such as if you were implementing your
own regex library or text buffer and wanted optimal performance. For
these cases we have renamed the original primitives by replacing
string with byte-string. Thus byte-string-length is the length
in bytes, not characters, of the strings (the equivalent of Gauche's
string-size). byte-string-set! can corrupt the UTF-8 encoding and
should be used sparingly if at all.
LOW LEVEL API
Direct manipulation of the utf8 encoding is factored away in the utf8-lolevel unit. This includes an abstract string-pointer API, and an analogous string-pointer implementation for ASCII strings in the string-pointer unit, however as the API is not fixed you use these at your own risk.
LIMITATIONS
peek-char currently does not have Unicode semantics (i.e. it peeks
only a single byte) to avoid problems with port buffering.
char-sets are not interchangeable between the existing srfi-14 code and Unicode code (i.e. do not pass a Unicode char-set to an external library that directly uses the old srfi-14).
PERFORMANCE
string-length, string-ref and string-set! are all O(n) operations as
opposed to the usual O(1) since UTF-8 is a variable width encoding.
Use of these should be discouraged - it is much cleaner to use the
high-level SRFI-13 procedures and string ports. For examples of how
to do common idioms without these procedures look at any string-based
code in Gauche.
Furthermore, string-set! and other procedures that modify strings in
place may invoke gc if the mutated result does not fit within the
same UTF-8 encoding size as the original string. If only mutating
7-bit ASCII strings (or only mutating within fixed encoding sizes
such as Cyrillic->Cyrillic) then no gc will occur.
string?, string=?, string-append, all R5RS string comparisons, and
read-line are unmodified.
Regular expression matching will be just as fast except in the case of Unicode character classes (which were not possible before anyway).
All other procedures incur zero to minor overhead, but keep the same asymptotic performance.
DISCUSSION
There are two ways to add Unicode string support to an existing language: redefine the strings themselves (i.e. add a new string type), or redefine the operations on the strings. The former causes a schism in your string libraries, dividing them between Unicode-aware and not, either doubling your library implementations or limiting them to one type or the other. You can't freely pass strings to other libraries without keeping track of their types and converting when needed. It becomes slow and unwieldy. C and Perl are the only language I know of who seriously tried this. In Perl the modules which worked with Unicode strings were minimal, frequent type conversions were needed, a general mess ensued, and Perl very quickly switched to the latter approach. In C as well, the libraries supporting wchar are still minimal, while most libraries still only support char.
UTF-8 is ideal for the in-place sort of extension because it is backwards compatible with ASCII. Any ASCII (7-bit) byte found within a UTF-8 string is guaranteed to be that character, not part of a multibyte character, so parsing libraries that work on ASCII characters work unmodified. This includes most existing text formats and network protocols. The EUC (Extended Unix Code) encodings also have this feature so a similar module could be implemented allowing users to (require 'euc-jp) for example and work in Japanese EUC rather than Unicode. Other encodings such as Shift_JIS satisfy the requirement that an ASCII string has the same meaning in the encoding, but multibyte characters in the encoding may include ASCII bytes, breaking the rule we need for safe ASCII parsing. A few encodings like UTF-16 and UTF-32 are completely incompatible. UTF-16 is primarily only used these days by Java, a victim of the unfortunate fact that at first UTF-16 was fixed with but is no longer with the advent of surrogate pairs. Note that even without this module you can write source code in Chicken in any ASCII compatible encoding like ISO-8859-* or UTF-8 and define symbols with that encoding (letting you replace lambda with syntax for a real greek lambda, for example).
Other languages that use UTF-8 include Perl, Python, TCL. XML and increasingly more and more network standards are using UTF-8 by default, and major databases all support UTF-8. Libraries with UTF-8 support include Gtk, SDL, and freetype.
Copyright (c) 2004-2005, Alex Shinn
All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following
conditions are met:
Redistributions of source code must retain the above copyright notice, this list of conditions and the following
disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following
disclaimer in the documentation and/or other materials provided with the distribution.
Neither the name of the author nor the names of its contributors may be used to endorse or promote
products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS
OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR
CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.