releasecandidate.bsky.social / token-string
Short (up to 65,535 bytes) immutable strings to e.g. parse tokens, implemented in Rust. These are sometimes called "German Strings", because Germans have written the paper mentioning them
0
fork

Configure Feed

Select the types of activity you want to include in your feed.

Rust 100.0%
43 1 5

Clone this repository

https://tangled.org/releasecandidate.bsky.social/token-string https://tangled.org/did:plc:bsvlnvc2ypnlpvcttd7b5jlz/token-string
git@tangled.org:releasecandidate.bsky.social/token-string git@tangled.org:did:plc:bsvlnvc2ypnlpvcttd7b5jlz/token-string

For self-hosted knots, clone URLs may differ based on your setup.

Download tar.gz
README.md

token-string#

MPL v2.0 license Built with cargo-make

Short immutable strings to e.g. parse tokens, implemented in Rust. These are sometimes called "German Strings", because Germans have written the paper mentioning them, see The Paper for details.

Memory Layout of a TokenString#

packet-beta
title Small TokenString
0-1: "String length"
2-7: "Prefix"
8-15: "Short string data"

packet-beta
title TokenString
0-1: "String length"
2-7: "Prefix"
8-15: "Pointer to data array"

Usage of the Library#

Building and Testing the Source#

The Paper#

Neumann, Freitag: Umbra: A Disk-Based System with In-Memory Performance mentions comparable small strings in chapter 3.1 String Handling at page 5.

The main difference to the variant I've implemented is that I use only 2 bytes (16 bits) for the string length instead of 4 bytes (32 bits) in the paper. So the maximum string length of TokenStrings is 65KB bytes compared to 4GB in the paper. But we gain 2 bytes of length for the short strings contained in the struct itself, 14 bytes vs. 12 bytes in the paper's version.

License#

The source in this repository is licensed under the Mozilla Public License version 2.0, see file ./LICENSE.