tur/unicode
Unicode classes, case folding and UTF-8 decoding for String.
What a Markdown parser needs past ASCII: CommonMark's Unicode whitespace and punctuation, full Unicode case folding, and the code point on either side of a byte index. A Char here holds a whole code point, not a byte.
The tables are the ones behind #lang r7rs's (scheme char): tools/gen-r7rs-unicode.py generates them into stdlib/r7rs/unicode.tur from the Unicode Character Database that tools/fetch-ucd.sh pins (Unicode 16.0.0). The interpreter registers natives over the same generated C, so compiled code, the JIT and the interpreter agree.
Since: markdown-library-plan U
unicode/space?
(unicode/space? [c :Char] :bool)
is c Unicode whitespace, as CommonMark defines it?
CommonMark 0.31: a character in the Zs general category, or tab, line feed, form feed or carriage return. That is narrower than char.tur's space? on ASCII (no vertical tab) and wider past it (no-break space, the em space, the ideographic space...).
| c | the code point to test |
true when c is Unicode whitespace.
(unicode/space? (int->char 0x3000)) ; => true (ideographic space)
Since: markdown-library-plan U
unicode/punct?
(unicode/punct? [c :Char] :bool)
is c Unicode punctuation, as CommonMark defines it?
CommonMark 0.31: a character in a P (punctuation) or S (symbol) general category. Every ASCII punctuation character is one, so this agrees with char.tur's punct? on ASCII; past it, it takes in quotation marks, dashes, currency signs, arrows and the replacement character U+FFFD.
| c | the code point to test |
true when c is Unicode punctuation.
(unicode/punct? (int->char 0x201C)) ; => true (left double quotation mark)
Since: markdown-library-plan U
unicode/fold-case
(unicode/fold-case [s :String] :String)
s with full Unicode case folding applied.
CaseFolding.txt's C and F mappings, so the result can be longer than s: "Stra\xc3\x9fe" and "STRASSE" both fold to "strasse". Two strings that fold to the same bytes match without regard to case, which is how CommonMark matches link reference labels. Invalid UTF-8 is copied through unchanged. The fold stops at a NUL byte.
| s | the String to fold (borrowed) |
A fresh owned String.
(string/to-cstr (unicode/fold-case (string/from-cstr "HeLLo"))) ; => "hello"
Since: markdown-library-plan U
utf8/-in?
(utf8/-in? [b :int lo :int hi :int] :bool)
internal: is b in [lo, hi]?
utf8/-len
(utf8/-len [b :int] :int)
internal: the sequence length a lead byte announces, or 0 for a byte that cannot lead.
utf8/-second-lo
(utf8/-second-lo [b :int] :int)
internal: the smallest valid second byte after lead b.
utf8/-second-hi
(utf8/-second-hi [b :int] :int)
internal: the largest valid second byte after lead b.
utf8/-bad
(utf8/-bad [end :int] :(Pair Char int))
internal: U+FFFD, paired with the index end.
utf8/-tail
(utf8/-tail [s :String j :int end :int cp :int] :(Pair Char int))
internal: fold the continuation bytes [j, end) of s into cp.
utf8/-lead
(utf8/-lead [s :String i :int b0 :int n :int] :(Pair Char int))
internal: decode the n-byte sequence whose lead byte b0 is at i.
utf8/-step
(utf8/-step [s :String i :int] :(Pair Char int))
internal: the code point at in-range index i, and the index past it.
utf8/decode-at
(utf8/decode-at [s :String i :int] :(Option (Pair Char int)))
the code point that starts at byte index i of s.
Decodes the UTF-8 sequence at i. A byte that cannot start a sequence, an overlong form, a surrogate or a sequence cut short decodes as U+FFFD, covering the bytes up to where it went wrong, so a scan that follows the returned index always moves forward.
| s | the String to read (borrowed) | |
| i | a byte index into s |
(some (pair c next)): the code point and the index just past it. (none) when i is out of range.
(utf8/decode-at (string/from-cstr "\xc3\xa9t\xc3\xa9") 0) ; => (some (pair #\x00E9 2))
Since: markdown-library-plan U
utf8/-start-before
(utf8/-start-before [s :String i :int j :int] :int)
internal: the lead byte's index for the code point ending at i, at most 4 bytes back.
utf8/decode-before
(utf8/decode-before [s :String i :int] :(Option (Pair Char int)))
the code point that ends just before byte index i of s.
The mirror of utf8/decode-at, for looking left: the character before a delimiter run, say. When the bytes before i are not the end of a valid sequence, the byte at i - 1 decodes as U+FFFD on its own.
| s | the String to read (borrowed) | |
| i | a byte index into s; the code point ends at i - 1 |
(some (pair c start)): the code point and the index of its first byte. (none) when i is 0 or less, or past the end of s.
(utf8/decode-before (string/from-cstr "a\xc3\xa9") 3) ; => (some (pair #\x00E9 1))
Since: markdown-library-plan U
utf8/-cont
(utf8/-cont [cp :int k :int] :int)
internal: the continuation byte holding bits [6k, 6k+6) of cp.
utf8/-valid?
(utf8/-valid? [cp :int] :bool)
internal: is cp a Unicode scalar value (in range, not a surrogate)?
utf8/-encoded-len
(utf8/-encoded-len [cp :int] :int)
internal: how many bytes the UTF-8 encoding of valid cp takes.
utf8/-lead-byte
(utf8/-lead-byte [cp :int n :int] :int)
internal: the first byte of the n-byte encoding of cp, n of 2 to 4.
utf8/-push-conts!
(utf8/-push-conts! [b :StringBuilder cp :int k :int] :void)
internal: append cp's continuation bytes k down to 0.
utf8/push-char!
(utf8/push-char! [b :StringBuilder c :Char] :void)
append the UTF-8 encoding of the code point c to b.
One to four bytes, the inverse of utf8/decode-at. A surrogate, a negative value or one past U+10FFFF is not a character, so it appends U+FFFD instead.
| b | the builder to append to | |
| c | the code point to encode |
(let [b (builder/new)]
(do (utf8/push-char! b (int->char 233))
(string/len (builder/finish b)))) ; => 2
Since: markdown-library-plan U2