Content System Documentation#

This document explains how the ATProto content system works in this project, including real-time streaming, repository browsing, and type generation.

Overview#

The content system is built around ATProto (Authenticated Transfer Protocol) and provides three main capabilities:

Real-time Streaming: Live updates from your ATProto repository
Repository Browsing: Explore collections and records from any ATProto account
Type Generation: Automatically generate TypeScript types for discovered lexicons

Architecture#

Core Components#

src/lib/atproto/
├── client.ts              # Basic ATProto API client
├── jetstream-client.ts    # Real-time streaming via WebSocket
└── atproto-browser.ts     # Repository browsing and analysis

Test Pages#

src/pages/
├── jetstream-test.astro           # Real-time streaming test
├── atproto-browser-test.astro     # Repository browsing test
├── lexicon-generator-test.astro   # Type generation test
└── galleries.astro               # Image galleries display

1. Real-time Streaming (Jetstream)#

How It Works#

The jetstream system connects to ATProto's real-time streaming service to receive live updates from your repository.

Key Features:

WebSocket connection to wss://jetstream1.us-east.bsky.network/subscribe
DID filtering (only shows your configured account)
Real-time commit events (create/update/delete)
Low latency updates

Configuration:

// From src/lib/config/site.ts
export const defaultConfig: SiteConfig = {
  atproto: {
    handle: 'tynanpurdy.com',
    did: 'did:plc:6ayddqghxhciedbaofoxkcbs',
    pdsUrl: 'https://bsky.social',
  },
  // ...
};

Usage:

// In jetstream-test.astro
const client = new JetstreamClient();
client.onRecord((record) => {
  console.log('New record:', record);
  // Display record in UI
});
await client.startStreaming();

Message Format:

interface JetstreamRecord {
  uri: string;
  cid: string;
  value: any;
  indexedAt: string;
  collection: string;
  $type: string;
  service: string;
  did: string;
  time_us: number;
  operation: 'create' | 'update' | 'delete';
}

2. Repository Browsing (ATProto Browser)#

How It Works#

The ATProto browser allows you to explore any ATProto repository's collections and records, similar to atptools.

Key Features:

Resolve handles to DIDs automatically
Discover all collections in a repository
Browse records from specific collections
Get repository metadata and profiles

API Endpoints Used:

// Resolve handle to DID
GET /xrpc/com.atproto.identity.resolveHandle?handle={handle}

// Get repository info
GET /xrpc/com.atproto.repo.describeRepo?repo={did}

// Get records from collection
GET /xrpc/com.atproto.repo.listRecords?repo={did}&collection={collection}&limit={limit}

// Get specific record
GET /xrpc/com.atproto.repo.getRecord?uri={uri}

Usage:

// In atproto-browser-test.astro
const browser = new AtprotoBrowser();

// Get repository info
const repoInfo = await browser.getRepoInfo('tynanpurdy.com');

// Get collections
const collections = await browser.getAllCollections('tynanpurdy.com');

// Get records from collection
const records = await browser.getCollectionRecords('tynanpurdy.com', 'app.bsky.feed.post', 50);

3. Type Generation (Lexicon Generator)#

How It Works#

The lexicon generator analyzes your repository to discover all lexicon types and generates TypeScript interfaces for them.

Process:

Get all collections from your repository
Sample records from each collection
Group records by $type
Analyze record structure and properties
Generate TypeScript interfaces

Generated Output:

// Auto-generated TypeScript types
export interface AppBskyFeedPost {
  $type: 'app.bsky.feed.post';
  text: string;
  createdAt: string;
  // ... other properties
}

export interface CollectionTypes {
  'app.bsky.feed.post': 'app.bsky.feed.post';
  'app.bsky.actor.profile': 'app.bsky.actor.profile';
  // ... other collections
}

export type AllLexicons = AppBskyFeedPost | AppBskyActorProfile | /* ... */;

// Helper functions
export function isLexiconType(record: any, type: string): boolean;
export function getCollectionTypes(collection: string): string[];

Usage:

// In lexicon-generator-test.astro
const generator = new SimpleLexiconGenerator();
const result = await generator.generateTypesForRepository();
// result.typeDefinitions contains the generated TypeScript code

Content Flow#

1. Real-time Updates#

ATProto Repository → Jetstream → WebSocket → UI Updates

2. Repository Browsing#

User Input → Handle Resolution → Collection Discovery → Record Fetching → UI Display

3. Type Generation#

Repository Analysis → Lexicon Discovery → Property Analysis → TypeScript Generation → File Export

Configuration#

Environment Variables#

# Your ATProto account
ATPROTO_HANDLE=tynanpurdy.com
ATPROTO_DID=did:plc:6ayddqghxhciedbaofoxkcbs
ATPROTO_PDS_URL=https://bsky.social

# Site configuration
SITE_TITLE=Your Site Title
SITE_AUTHOR=Your Name

Site Configuration#

// src/lib/config/site.ts
export interface SiteConfig {
  site: {
    title: string;
    description: string;
    author: string;
    url: string;
  };
  atproto: {
    handle: string;
    did: string;
    pdsUrl: string;
  };
}

Error Handling#

Common Issues#

Jetstream Connection Failures:
- Check WebSocket endpoint availability
- Verify DID is correct
- Check network connectivity
Repository Access Errors:
- Verify handle/DID exists
- Check PDS server availability
- Ensure proper API permissions
Type Generation Issues:
- Repository must have records to analyze
- Check for valid lexicon types
- Verify record structure

Debugging#

// Enable detailed logging
console.log('🔍 Analyzing collection:', collection);
console.log('📝 New record from jetstream:', record);
console.log('✅ Generated lexicon for:', $type);

Performance Considerations#

Caching#

Repository metadata is cached to reduce API calls
Collection lists are cached during browsing sessions
Type generation results can be saved locally

Rate Limiting#

ATProto APIs have rate limits
Implement delays between requests
Use pagination for large collections

Memory Management#

Limit record samples during type generation
Clear old records from streaming buffers
Implement proper cleanup for WebSocket connections

Future Enhancements#

Potential Improvements#

Enhanced Type Generation:
- Support for nested object types
- Union type detection
- Custom type annotations
Advanced Streaming:
- Multiple DID filtering
- Collection-specific streams
- Event replay capabilities
Repository Analysis:
- Statistical analysis of repository
- Content type distribution
- Usage patterns
UI Enhancements:
- Real-time type updates
- Interactive collection browsing
- Advanced filtering options

Integration with Other Systems#

ATProto Ecosystem#

Compatible with any ATProto PDS
Works with custom lexicons
Supports Bluesky and other ATProto apps

Development Workflow#

Generated types can be used in TypeScript projects
Real-time updates can trigger build processes
Repository changes can trigger deployments

Security Considerations#

Data Privacy#

Only access public repository data
No authentication required for browsing
Streaming only shows public commits

API Usage#

Respect rate limits
Implement proper error handling
Use HTTPS for all API calls

Troubleshooting#

Common Problems#

Jetstream not connecting:
- Check if endpoint is accessible
- Verify DID format
- Check browser WebSocket support
No records found:
- Verify repository has content
- Check collection names
- Ensure proper API permissions
Type generation fails:
- Repository must have records
- Check for valid lexicon structure
- Verify record format

Debug Commands#

# Test ATProto API access
curl "https://bsky.social/xrpc/com.atproto.repo.describeRepo?repo=did:plc:6ayddqghxhciedbaofoxkcbs"

# Test jetstream connection
wscat -c "wss://jetstream1.us-east.bsky.network/subscribe?dids=did:plc:6ayddqghxhciedbaofoxkcbs"

This content system provides a comprehensive foundation for working with ATProto data, from real-time streaming to type-safe development.