coral#

real-time semantic percolation from the Bluesky firehose.

live demo | source

what it does#

extracts named entities (people, organizations, places, events) from Bluesky posts and tracks how they cluster together. entities that get discussed together form edges. when a real-world event spans multiple topics, clusters merge - discourse percolates into a unified conversation.

how it works#

1. NER bridge consumes the turbostream firehose, runs spaCy NER, extracts entities
2. labeler integration drops spam before it hits the graph (via Hailey's labeler)
3. entity graph tracks co-occurrences (entities in same post = edge), computes clusters via union-find
4. pheromone edges - edge weights decay exponentially, reinforced on repeated co-occurrence (ant colony optimization inspired)
5. surprise trending - entities ranked by statistical surprise vs baseline (z‑like), not raw counts
6. frontend visualizes entity activity, cluster structure, and firehose health

theoretical background#

the system draws from several sources:

percolation theory - we use the Newman-Ziff algorithm for efficient cluster detection. on lattices, percolation has a sharp phase transition at p_c ≈ 0.593. our graph isn't a lattice, so we calibrate empirically.

heterogeneous activity - Xie et al. 2021 showed that real social networks percolate at ~1/10th the uniform-theory threshold due to heterogeneous user activity. we weight mentions by user activity rate following this insight.

NER for topic detection - inspired by Hailey's trending topics. rather than embeddings on raw text (too noisy), extract structured entities to reduce surface area.

ATProto labeler system - spam filtering via com.atproto.label. we subscribe to Hailey's labeler stream and drop posts from accounts labeled as spam before NER processing.

stack#

• ner (python): turbostream consumer + spaCy NER + labeler gate → POST to backend
• backend (zig): entity graph + websocket server + SQLite persistence
• site: static html/css/js on cloudflare pages

run locally#

cd backend && zig build run                 # backend (entity graph + websocket)
cd ner && uv run coral-bridge               # NER bridge (turbostream → spaCy → backend)
cd site && npx wrangler pages dev .         # frontend

deploy#

cd backend && fly deploy
cd ner && fly deploy
cd site && npx wrangler pages deploy . --project-name coral

future work#

ideas being explored (not commitments):

• semantic positioning - currently entities hash to arbitrary grid positions. could use embeddings to place semantically similar entities near each other, making the 2D layout a meaningful projection of topic space. unclear whether to embed entity names, representative posts, or cluster centroids.

• temporal co-activity edges - entities that spike together might be related even without same-post co-occurrence. "earthquake" and "LA" could both trend during an event without always appearing together.

• percolation calibration - the 50% threshold is arbitrary. need to correlate cluster merges with real-world events to understand what "discourse unification" actually looks like in the data.

references#

• Newman & Ziff, Efficient Monte Carlo algorithm and high-precision results for percolation, Phys. Rev. Lett. 85 (2000)
• Xie et al., Detecting and Modelling Real Percolation and Phase Transitions of Information on Social Media, Nature Human Behaviour (2021)
• Hailey, Bluesky Trending Topics - NER approach for topic detection
• Stauffer & Aharony, Introduction to Percolation Theory - theoretical foundations
• ATProto Labels - moderation architecture