Files
README.md

spacez #

spaCy's en_core_web_sm NER pipeline, reimplemented in zig.

hash embeddings → residual CNN → BILUO transition parser, all from the original weights. no python, no runtime dependencies, ~6MB binary.

install #

zig fetch --save https://tangled.sh/@zzstoatzz.io/spacez/archive/main

then in build.zig:

const spacez = b.dependency("spacez", .{}).module("spacez");
exe.root_module.addImport("spacez", spacez);

usage #

const spacez = @import("spacez");

var model = try spacez.Model.load(spacez.en_core_web_sm);

var entities: [64]spacez.SpanEntity = undefined;
const n = spacez.recognize(&model, "Barack Obama visited Paris.", &entities);

for (entities[0..n]) |e| {
    // e.label = .PERSON, .GPE, etc.
    // e.start / e.end = byte offsets into source text
    // e.margin = how far the winning action led its best alternative
}

how sure was it? #

each span carries the parser's own decision data:

  • margin — the gap between the winning action's score and the best valid alternative at the moment it was chosen. a span built from several actions reports its weakest link, since a span is only as certain as the least certain decision behind it. inf when the winner had no valid competitor.
  • score — the winning action's raw output, likewise the minimum across the span.

these are not probabilities. the parser decodes greedily with no softmax and no beam, so the numbers are unnormalized and carry no calibrated meaning on their own. margin is the more useful of the two: a gap is scale-relative, so it survives the shifts in raw score that come with different contexts.

what the runner-up usually is. the competing action is most often a different label for the same span, not "no entity here". so a small margin tends to mean the parser was torn over the label, and says little about whether a span is an entity at all. measured on en_core_web_sm:

span label margin
Paris GPE 8.75
Kira PERSON 6.57
Barack Obama PERSON 4.03
Lmao PERSON 1.21
Wat da heck PERSON 0.74
Trump ORG 0.19
BLAKE3 ORG 0.03

BLAKE3 and Trump — both real entities — sit below the junk spans, because each is genuinely ambiguous between two labels. use these numbers to reason about label confidence, not to filter noise.

if you do gate on them, measure the distribution on your own text first — the useful cut differs by domain, and a threshold copied from someone else's corpus is a guess wearing a number. parser.argmaxValidScored exposes the same data one decision at a time for anyone driving the parser directly.


weights are embedded at compile time — no files to download or manage.

## what's here

- **hash embeddings** — murmurhash + bloom → 4 embedding tables, reduced via maxout
- **CNN encoder** — 4 residual blocks (seq2col → dense → maxout → layernorm → residual add)
- **transition parser** — greedy BILUO with 73 actions (9 entity types × 4 transitions + 1)
- **tokenizer** — full port of spaCy's English tokenizer (~1,000 special cases, prefix/suffix/infix rules)
- **SIMD ops** — vectorized matvec, vadd using zig's `@Vector` intrinsics

## validation

73/73 test sentences produce identical entities and byte offsets to spaCy `en_core_web_sm` 3.8.0, including multi-byte UTF-8, contractions, hyphens, possessives, informal text, and zero-entity inputs.

```bash
# head-to-head comparison (requires spaCy + en_core_web_sm)
uv run --python 3.12 --with spacy \
  --with 'en_core_web_sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl' \
  python scripts/compare.py

entity types #

PERSON, ORG, GPE, PRODUCT, EVENT, WORK_OF_ART, FAC, NORP, LOC

(plus CARDINAL, DATE, MONEY, PERCENT, ORDINAL, QUANTITY, TIME, LAW — inherited from the model)