site.standard.* Migration Review #

Overview #

This migration adds support for the site.standard.* lexicon schema alongside the existing pub.leaflet.* schema. The goal is to unify how documents and publications are stored and accessed through a normalization layer.

Architecture Analysis #

Core Strategy: Normalization Layer #

The migration uses a read-time normalization approach:

• Both pub.leaflet.* and site.standard.* records are stored in the same database tables (documents, publications)
• When reading, a normalization layer converts both formats to the canonical site.standard.* format
• The canonical format (NormalizedDocument, NormalizedPublication) is what components consume

Key Files:

• lexicons/src/normalize.ts - Core normalization functions and types
• src/utils/normalizeRecords.ts - Database-aware wrappers for normalization

Database Schema #

The schema adds new tables but continues to use existing tables for both formats:

• site_standard_publications - New table (but appears unused in migration?)
• site_standard_documents - New table (but appears unused in migration?)
• site_standard_documents_in_publications - Junction table
• site_standard_subscriptions - New subscription table

Question: Why are new tables added if both formats use the existing documents and publications tables?

Appview (Firehose Consumer) #

The appview (appview/index.ts) now:

• Listens for site.standard.* collections on the firehose
• Stores both pub.leaflet.* and site.standard.* records in the same tables
• Uses the same storage structure (documents/publications tables)

Context Refactoring #

Major change: PostPageContext.tsx is deleted and replaced with:

• DocumentContext.tsx - Provides normalized document/publication data
• LeafletContentContext.tsx - Provides leaflet-specific content (pages)

This separation allows:

• Components to access normalized data without knowing the source format
• Content-specific handling to be isolated

Key Questions/Concerns #

1. Database Table Strategy (CLARIFIED) #

The schema has two parallel storage strategies:

Strategy 1 - Live Appview (firehose):

• Both pub.leaflet.* and site.standard.* records → existing documents and publications tables
• Uses normalization layer to read both formats uniformly

Strategy 2 - Backfill Script (scripts/backfill-site-standard.ts):

• Only site.standard.* records → new site_standard_* tables
• For historical data backfill from the network

CONCERN: This creates data duplication/inconsistency:

• Live site.standard.* records go to documents table
• Backfilled site.standard.* records go to site_standard_documents table
• No code appears to READ from the site_standard_* tables

Questions to resolve:

• Is the backfill script meant to populate the main tables instead?
• Are the site_standard_* tables temporary or for a future migration phase?
• Will there be code to merge/unify these tables?

2. Legacy Data Handling #

The normalization functions (isLeafletDocument, etc.) include fallback detection:

// Legacy records without $type but with pages array
(Array.isArray(r.pages) && typeof r.author === "string")

This suggests there may be legacy data without proper $type discrimination.

3. Null Safety in Normalization #

Multiple places call normalizeDocumentRecord which can return null. Need to verify all call sites handle null properly.

4. Theme Handling #

The migration preserves theme in two places:

• normalizedPublication?.theme - publication-level theme
• normalizedDocument?.theme - document-level theme (for backwards compat)

Components use: let theme = normalizedPublication?.theme || normalizedDocument?.theme

5. Field Mapping Changes #

Key field mappings in normalization:

• publication (leaflet) → site (standard)
• postRef (leaflet) → bskyPostRef (standard)
• pages (leaflet) → content: { $type: "pub.leaflet.content", pages } (standard)
• base_path (leaflet) → url (standard) with https:// prefix

---

Files Changed Review Status #

Core Infrastructure (Reviewed) #

• src/utils/normalizeRecords.ts - Clean wrapper implementation
• lexicons/src/normalize.ts - Core normalization logic
• drizzle/schema.ts - New tables added
• drizzle/relations.ts - Relations for new tables
• appview/index.ts - Firehose handling for both formats
• contexts/DocumentContext.tsx - New context
• contexts/LeafletContentContext.tsx - New context

Data Fetching (Needs Review) #

• app/lish/[did]/[publication]/[rkey]/getPostPageData.ts - Key data fetcher
• app/(home-pages)/reader/getReaderFeed.ts
• app/(home-pages)/reader/getSubscriptions.ts
• app/(home-pages)/discover/getPublications.ts
• app/(home-pages)/p/[didOrHandle]/getProfilePosts.ts
• app/(home-pages)/tag/[tag]/getDocumentsByTag.ts

Component Updates (Needs Review) #

• Notification components (6 files)
• Post page components (multiple files)
• Dashboard components
• Theme components

API Routes (Needs Review) #

• app/api/rpc/[command]/get_profile_data.ts
• app/api/rpc/[command]/get_publication_data.ts
• app/lish/feeds/[...path]/route.ts
• Other routes

---

Delegated Reviews #

See separate sections for detailed file-by-file reviews.