text #
written against zig 0.16.
zig has no string type. text is []const u8, and the language makes no promise
about what encoding those bytes are in. indexing and slicing are byte
operations, which is correct for a byte slice and wrong for most of the things
people write with it.
the two vocabularies #
almost every text operation is described in one of two units, and they are easy to mix up because ASCII makes them identical:
- bytes — buffer sizes, length caps, wire formats. slicing is right here.
- characters — "first character", "last three characters", "one character per class". slicing is wrong here the moment input leaves ASCII.
"’s" is four bytes: E2 80 99 73. taking token[0] yields 0xE2, which is
not a character. it is the lead byte of one, and alone it identifies nothing —
the same first byte begins thousands of different characters.
what this looks like in practice #
feature extraction for spaCy's NER model defines three token features over characters: the first character, the last three, and a per-character shape string. written over bytes, they look right:
prefix_buf[0] = token[0]; // first byte
var suffix_buf: [3]u8 = undefined; // last three bytes
for (token) |c| { ... } // shape, per byte
for ASCII this is exactly right, and ASCII tests pass. for a curly apostrophe
the prefix hashes a lone 0xE2, the suffix starts in the middle of a
character, and the shape emits three symbols where spaCy emits one. the model
looks up embeddings for tokens that do not exist and answers confidently.
nothing crashes, because bytes are always valid bytes.
decoding #
std.unicode has what you need to work in characters:
const n = try std.unicode.utf8ByteSequenceLength(bytes[0]); // 1..4
const cp = try std.unicode.utf8Decode(bytes[0..n]);
var it = (try std.unicode.Utf8View.init(s)).iterator(); // codepoint walk
going backwards has no helper. continuation bytes are 10xxxxxx, so walking
back to a character boundary is a loop over b & 0xC0 == 0x80.
classification stops at ASCII #
std.ascii answers isAlphabetic, isUpper, toLower for bytes below 128.
above that, std has no case mapping and no character categories — those are
large tables and zig does not ship them.
this matters when a spec is written in terms of "letter" or "uppercase".
spaCy's shape feature calls Проверка five letters and lowercases Ω to ω;
reproducing that needs unicode tables from somewhere. the honest options are to
carry the tables, take a dependency, or document the gap. picking one of ASCII's
answers and hoping is the option that looks like it works.
truncation #
a byte cap that lands mid-character produces invalid UTF-8, which json encoders and terminals both object to. walk characters and stop before the one that would cross the limit:
/// truncate to at most `max` bytes without splitting a codepoint
pub fn utf8TruncateLen(s: []const u8, max: usize) usize {
const limit = @min(s.len, max);
var i: usize = 0;
while (i < limit) {
const seq_len = std.unicode.utf8ByteSequenceLength(s[i]) catch {
i += 1; // invalid lead byte
continue;
};
if (i + seq_len > limit) break;
i += seq_len;
}
return i;
}
any fixed-size field holding user text wants this — entity names, log lines,
anything copied into a [64]u8.
rules of thumb #
- if a spec says "character", decode. if it says "byte", slice.
- a fixed-size buffer is a byte count; the loop that fills it is a character loop. keeping those straight in the same function is most of the work.
- test with a curly apostrophe, an emoji, and a Cyrillic word. ASCII tests pass either way, which is what makes this class of bug survive a test suite.
related #
- structs — slices are views; the same discipline applies to what a slice points at
- json — invalid UTF-8 surfaces first at the encoder