Converting between UTF-8 and legacy character encodings, as the WHATWG Encoding Standard and glibc's iconv define them.
zig-charset GLOSSARY.md
3.0 kB · 88 lines
Markdown
at main

zig-charset #

zig-charset converts between UTF-8 and legacy character encodings. These are the words its code and documentation use. Which definition of an encoding is meant always matters, because the same name can mean different bytes in different places.

Definitions of an encoding #

Encoding: One of the forty encodings as the WHATWG Encoding Standard defines them, which is the set browsers implement and is closed. Avoid: charset (for this type), codec

Glibc charset: A charset as glibc 2.44's iconv defines it, matched byte for byte, including which inputs fail. Only a charset this library reproduces exactly counts as one. Avoid: glibc mode, iconv mode, flavor, profile

EBCDIC code page: One of the IBM EBCDIC code pages this library carries outside the Encoding Standard, reached only by naming it in code.

Korean encoding: Wansung or Johab, the Korean encodings the Encoding Standard leaves out, reached only by naming them in code.

Naming an encoding #

Label: A string the Encoding Standard resolves to an Encoding, such as one taken from a Content-Type header. Avoid: name, alias (for WHATWG strings)

Name: A string glibc's iconv accepts for a charset, whether its canonical name or an alias, in any of the spellings glibc treats as the same. Avoid: label (for glibc strings)

Supported (name): A name that glibc accepts and that resolves to a glibc charset this library reproduces.

Unsupported (name): A name that glibc accepts but that this library does not reproduce. This includes any name whose suffix asks glibc to transliterate or to skip input.

Unknown (name): A name glibc's iconv would refuse to open.

Handling bad input #

Error mode: What a decoder does with bytes that mean nothing, or what an encoder does with a scalar value it cannot represent. It applies to every definition of an encoding alike, and it is not a choice between definitions. Avoid: mode (unqualified)

Unmappable code point: A Unicode scalar value that the target encoding has no bytes for.

Outside Unicode: A byte sequence that glibc's UTF-8 or UCS-4 decodes to something that is not a Unicode scalar value: a value above U+10FFFF, or a surrogate. This library refuses it, and it is the one place this library knowingly differs from glibc. glibc itself writes a surrogate only as UCS-4. Avoid: invalid UTF-8 (which glibc rejects too)

Byte order #

Byte order mark: U+FEFF at the very start of a text in a charset whose byte order it chooses. Anywhere else, or in a charset with a fixed byte order, it is only the character U+FEFF. Avoid: BOM (in prose), signature

Machine byte order: The byte order of the computer doing the conversion, which some glibc charsets use. The same name can therefore mean different bytes on different machines. Avoid: native order, host order, platform endianness