Following No Reposts Feed Generator#

A production-ready Bluesky feed generator written in Rust that shows posts from people you follow, excluding all reposts. Built using Jetstream for efficient real-time data consumption with full JWT signature verification.

Table of Contents#

• Features
• How It Works
• Prerequisites
• Installation
• Configuration
• Running Locally
• Deployment
• Publishing Your Feed
• Architecture
• API Endpoints
• Performance
• Development
• Troubleshooting
• Contributing
• License

Features#

• 🚫 No Reposts: Automatically filters out all reposts, showing only original content
• 👥 Personalized: Shows only posts from accounts you follow
• ⚡ Real-time: Updates in real-time as new posts are created
• 🔒 Secure: Full ES256K JWT signature verification with DID resolution
• 📡 Efficient: Uses Jetstream for lightweight event consumption (~850 MB/day vs 200+ GB/day)
• 🗄️ Smart Caching: Automatic cleanup of posts older than 48 hours
• 🔄 Auto-recovery: Automatic reconnection on Jetstream disconnects
• 📊 Observable: Structured logging with configurable verbosity
• 🏗️ Production Ready: Battle-tested error handling and recovery mechanisms

How It Works#

This feed generator:

1. Consumes Events: Connects to Bluesky's Jetstream to receive real-time events for posts and follows
2. Filters Content: Only subscribes to app.bsky.feed.post and app.bsky.graph.follow collections
3. Stores Data: Maintains a local SQLite database of recent posts and follow relationships
4. Serves Feeds: Provides personalized feeds via AT Protocol's app.bsky.feed.getFeedSkeleton endpoint
5. Authenticates Users: Validates JWT tokens by resolving user DIDs and verifying signatures

Prerequisites#

• Rust 1.70+ (install via rustup)
• SQLite 3.35+ (usually pre-installed on modern systems)
• Domain with HTTPS (required for production deployment)
• Bluesky Account (for publishing the feed)

Installation#

1. Clone the Repository#

git clone https://github.com/vitorpy/noreposts-atproto-feed.git
cd noreposts-atproto-feed

2. Build the Project#

cargo build --release

The compiled binary will be at target/release/following-no-reposts-feed.

Configuration#

Environment Variables#

Create a .env file in the project root (see .env.example for reference):

# Required: Database location
DATABASE_URL=sqlite:./feed.db

# Required: Server port
PORT=3000

# Required: Your domain name
FEEDGEN_HOSTNAME=your-domain.com

# Required: Your service DID
FEEDGEN_SERVICE_DID=did:web:your-domain.com

# Optional: Jetstream server (defaults to jetstream1.us-east.bsky.network)
JETSTREAM_HOSTNAME=jetstream1.us-east.bsky.network

Service DID Setup#

Your FEEDGEN_SERVICE_DID should match your domain. For did:web, it's typically:

• Domain: feed.example.com → DID: did:web:feed.example.com
• Domain: example.com → DID: did:web:example.com

Running Locally#

Quick Start#

# Set up environment
cp .env.example .env
# Edit .env with your configuration
nano .env

# Run the server
cargo run --release

The server will:

1. Automatically run database migrations
2. Connect to Jetstream and start consuming events
3. Start the HTTP server on the configured port
4. Serve the DID document at /.well-known/did.json

Testing Locally#

# Test DID document endpoint
curl http://localhost:3000/.well-known/did.json

# Test feed endpoint (requires authentication in production)
curl "http://localhost:3000/xrpc/app.bsky.feed.getFeedSkeleton?feed=at://did:web:your-domain.com/app.bsky.feed.generator/following-no-reposts&limit=10"

Command-Line Options#

# Override environment variables
./following-no-reposts-feed --port 8080 --hostname feed.example.com

# Run database migrations only
./following-no-reposts-feed migrate

# Publish feed to your Bluesky account
./following-no-reposts-feed publish \
  --handle your-handle.bsky.social \
  --password your-app-password \
  --record-name following-no-reposts \
  --display-name "Following (No Reposts)" \
  --description "See posts from people you follow, without any reposts"

# Backfill posts from firehose (optional)
./following-no-reposts-feed backfill --cursor <cursor-value>

Deployment#

1. Build for Production#

cargo build --release --locked

2. Set Up Your Server#

Transfer the binary to your server:

scp target/release/following-no-reposts-feed user@your-server:/opt/feed-generator/

3. Create a Systemd Service#

Create /etc/systemd/system/feed-generator.service:

[Unit]
Description=Bluesky Feed Generator - Following No Reposts
After=network.target

[Service]
Type=simple
User=feedgen
WorkingDirectory=/opt/feed-generator
Environment="DATABASE_URL=sqlite:/opt/feed-generator/feed.db"
Environment="PORT=3000"
Environment="FEEDGEN_HOSTNAME=your-domain.com"
Environment="FEEDGEN_SERVICE_DID=did:web:your-domain.com"
ExecStart=/opt/feed-generator/following-no-reposts-feed
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Enable and start the service:

sudo systemctl daemon-reload
sudo systemctl enable feed-generator
sudo systemctl start feed-generator
sudo systemctl status feed-generator

4. Configure Reverse Proxy#

The feed generator must be accessible via HTTPS. Configure your reverse proxy:

Nginx#

server {
    listen 443 ssl http2;
    server_name your-domain.com;

    ssl_certificate /etc/letsencrypt/live/your-domain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/your-domain.com/privkey.pem;

    location / {
        proxy_pass http://127.0.0.1:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # WebSocket support for Jetstream
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

Caddy#

your-domain.com {
    reverse_proxy localhost:3000
}

5. Verify Deployment#

# Test DID document
curl https://your-domain.com/.well-known/did.json

# Check if feed endpoint is accessible
curl https://your-domain.com/xrpc/app.bsky.feed.getFeedSkeleton

Publishing Your Feed#

Once your feed generator is deployed and accessible via HTTPS:

Method 1: Using the Built-in Publish Command#

./following-no-reposts-feed publish \
  --handle your-handle.bsky.social \
  --password your-app-password \
  --record-name following-no-reposts \
  --display-name "Following (No Reposts)" \
  --description "See posts from people you follow, without any reposts" \
  --avatar ./avatar.png

Note: Use an App Password, not your main account password!

Method 2: Manual Publishing#

1. Get your DID:

   curl "https://bsky.social/xrpc/com.atproto.identity.resolveHandle?handle=yourhandle.bsky.social"
   

2. Create a session:

   curl -X POST https://bsky.social/xrpc/com.atproto.server.createSession \
     -H "Content-Type: application/json" \
     -d '{"identifier": "yourhandle.bsky.social", "password": "your-app-password"}'
   

3. Publish the feed generator record:

   curl -X POST https://bsky.social/xrpc/com.atproto.repo.putRecord \
     -H "Authorization: Bearer YOUR_ACCESS_JWT" \
     -H "Content-Type: application/json" \
     -d '{
       "repo": "your.did",
       "collection": "app.bsky.feed.generator",
       "rkey": "following-no-reposts",
       "record": {
         "$type": "app.bsky.feed.generator",
         "did": "did:web:your-domain.com",
         "displayName": "Following (No Reposts)",
         "description": "See posts from people you follow, without any reposts",
         "createdAt": "2025-01-01T00:00:00.000Z"
       }
     }'
   

Finding Your Feed#

After publishing, your feed will be available at:

https://bsky.app/profile/yourhandle.bsky.social/feed/following-no-reposts

Architecture#

System Components#

┌──────────────┐         ┌──────────────┐         ┌──────────────┐
│   Jetstream  │────────▶│   Consumer   │────────▶│   Database   │
│  (WebSocket) │  Events │   (Async)    │  Store  │   (SQLite)   │
└──────────────┘         └──────────────┘         └──────────────┘
                                │
                                │ Read
                                ▼
┌──────────────┐         ┌──────────────┐         ┌──────────────┐
│   Bluesky    │────────▶│  HTTP Server │────────▶│     Feed     │
│     App      │   JWT   │    (Axum)    │  Query  │  Algorithm   │
└──────────────┘         └──────────────┘         └──────────────┘

Code Structure#

• main.rs: Application entry point, HTTP server setup, routing
• jetstream_consumer.rs: WebSocket client for Jetstream events
• database.rs: SQLite abstraction layer, queries, and migrations
• feed_algorithm.rs: Feed generation logic (filtering by follows, excluding reposts)
• auth.rs: JWT validation with ES256K signature verification
• backfill.rs: Optional historical data backfilling from firehose
• publish.rs: Feed generator publishing utilities
• admin_socket.rs: Unix socket for admin commands
• types.rs: Shared data structures

Data Flow#

1. Event Ingestion: Jetstream sends commit events when posts are created or follows happen
2. Event Processing: Consumer parses events and extracts relevant data
3. Database Storage: Posts and follows are stored in SQLite with TTL
4. Feed Requests: Bluesky app requests feed via getFeedSkeleton
5. Authentication: JWT is validated by resolving DID and verifying signature
6. Feed Generation: Algorithm queries posts from followed users, excludes reposts
7. Response: Ordered list of post URIs returned with pagination cursor

API Endpoints#

GET /.well-known/did.json#

Returns the DID document for the feed generator service.

Response:

{
  "@context": ["https://www.w3.org/ns/did/v1"],
  "id": "did:web:your-domain.com",
  "service": [{
    "id": "#bsky_fg",
    "type": "BskyFeedGenerator",
    "serviceEndpoint": "https://your-domain.com"
  }]
}

GET /xrpc/app.bsky.feed.getFeedSkeleton#

Returns a personalized feed skeleton for the authenticated user.

Query Parameters:

• feed (required): Feed AT-URI (e.g., at://did:web:your-domain.com/app.bsky.feed.generator/following-no-reposts)
• limit (optional): Number of posts (1-100, default: 50)
• cursor (optional): Pagination cursor

Headers:

• Authorization: Bearer JWT token from Bluesky app

Response:

{
  "feed": [
    {"post": "at://did:plc:xxx/app.bsky.feed.post/abc123"},
    {"post": "at://did:plc:yyy/app.bsky.feed.post/def456"}
  ],
  "cursor": "1234567890"
}

Performance#

Resource Usage#

• Memory: ~50-100 MB (depends on database size)
• CPU: Minimal (<1% on modern hardware)
• Bandwidth: ~850 MB/day (Jetstream with compression)
• Storage: ~100-500 MB (48-hour post retention)

Scalability#

The feed generator can handle:

• 10,000+ active users
• 1M+ posts/day ingestion
• 100+ requests/second

Database Optimization#

-- Efficient indexes
CREATE INDEX idx_posts_author_did ON posts(author_did);
CREATE INDEX idx_posts_indexed_at ON posts(indexed_at);
CREATE INDEX idx_follows_follower ON follows(follower_did);

-- Automatic cleanup (posts > 48 hours old)
DELETE FROM posts WHERE indexed_at < datetime('now', '-2 days');

Development#

Running Tests#

cargo test

Database Migrations#

Create a new migration:

sqlx migrate add your_migration_name

Run migrations manually:

cargo install sqlx-cli --no-default-features --features sqlite
sqlx migrate run

Logging#

Set the RUST_LOG environment variable:

# Debug level (verbose)
RUST_LOG=debug cargo run

# Info level (default)
RUST_LOG=info cargo run

# Specific module logging
RUST_LOG=following_no_reposts_feed::jetstream_consumer=debug cargo run

Code Quality#

# Format code
cargo fmt

# Run linter
cargo clippy --all-targets

# Check for security vulnerabilities
cargo audit

Troubleshooting#

Jetstream Connection Issues#

Problem: Cannot connect to Jetstream

Solutions:

• Verify network connectivity: ping jetstream1.us-east.bsky.network
• Check firewall rules (port 443 outbound)
• Try alternative Jetstream servers
• Monitor logs for specific error messages

Database Locked Errors#

Problem: database is locked errors

Solutions:

• Ensure only one instance is running
• Check for long-running transactions
• Increase busy_timeout in database configuration
• Consider using WAL mode: PRAGMA journal_mode=WAL;

Authentication Failures#

Problem: JWT validation errors

Solutions:

• Verify FEEDGEN_SERVICE_DID matches your domain
• Check network access to plc.directory for DID resolution
• Enable debug logging: RUST_LOG=following_no_reposts_feed::auth=debug
• Verify your domain's HTTPS certificate is valid

Feed Not Updating#

Problem: Feed shows stale content

Solutions:

• Check Jetstream connection: look for "Connected to Jetstream" in logs
• Verify database is being updated: sqlite3 feed.db "SELECT COUNT(*) FROM posts;"
• Check for errors in logs: journalctl -u feed-generator -n 100
• Restart the service: systemctl restart feed-generator

High Memory Usage#

Problem: Memory usage growing over time

Solutions:

• Verify automatic cleanup is working: check posts table size
• Manually trigger cleanup: DELETE FROM posts WHERE indexed_at < datetime('now', '-2 days');
• Reduce retention period in code if needed
• Monitor with: ps aux | grep following-no-reposts-feed

Debug Mode#

Enable comprehensive debugging:

RUST_LOG=debug,hyper=info,tokio=info cargo run

Test endpoints directly:

# Test DID endpoint
curl -v https://your-domain.com/.well-known/did.json

# Test feed endpoint with authentication
curl -v -H "Authorization: Bearer YOUR_JWT" \
  "https://your-domain.com/xrpc/app.bsky.feed.getFeedSkeleton?feed=at://did:web:your-domain.com/app.bsky.feed.generator/following-no-reposts&limit=5"

Contributing#

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Make your changes
4. Add tests if applicable
5. Run cargo fmt and cargo clippy
6. Commit your changes (git commit -m 'Add amazing feature')
7. Push to the branch (git push origin feature/amazing-feature)
8. Open a Pull Request

Development Guidelines#

• Follow Rust best practices and idioms
• Add tests for new functionality
• Update documentation for user-facing changes
• Keep commits atomic and well-described
• Ensure CI passes before submitting PR

License#

This project is licensed under the GNU General Public License v3.0 - see the LICENSE file for details.

This ensures the code remains free and open source. If you modify and distribute this software, you must:

• Disclose your source code
• License your modifications under GPLv3
• State significant changes made
• Include the original copyright notice

Resources#

• AT Protocol Documentation
• Bluesky API Reference
• Jetstream Documentation
• ATrium Rust Library
• Feed Generator Guide

Acknowledgments#

• Built with ATrium - Rust libraries for AT Protocol
• Uses Jetstream for efficient event streaming
• Inspired by the Bluesky community's work on custom feeds

Made with ❤️ for the Bluesky community