================================================================================ *roadmap.txt* PDSharp: F#/Giraffe PDS Implementation Roadmap ================================================================================ PHASE 1: IMPLEMENTATION (Build) ================================================================================ Milestone A: Giraffe XRPC Shell -------------------------------------------------------------------------------- - [x] Implement /xrpc/com.atproto.server.describeServer (GET) - [x] Implement NSID-based routing with structured error responses DoD: describeServer responds with stable JSON -------------------------------------------------------------------------------- Milestone B: Identity + Crypto Primitives -------------------------------------------------------------------------------- - [x] DID document fetch/parse for signing key and PDS endpoint - [x] SHA-256 hashing, ECDSA sign/verify (p256 + k256), low-S enforcement DoD: Sign and verify atproto commit hash with low-S -------------------------------------------------------------------------------- Milestone C: DAG-CBOR + CID -------------------------------------------------------------------------------- - [x] Canonical DAG-CBOR encode/decode with IPLD link tagging - [x] CID creation/parsing (multicodec dag-cbor, sha2-256) DoD: Record JSON → stable DAG-CBOR bytes → deterministic CID -------------------------------------------------------------------------------- Milestone D: MST Implementation -------------------------------------------------------------------------------- - [x] Merkle Search Tree per repository spec - [x] Key depth = leading zero bits in SHA-256(key) counted in 2-bit chunks - [x] Node encoding: (l, e[p,k,v,t]) with key prefix compression DoD: Insert/update/delete yields reproducible root CID -------------------------------------------------------------------------------- Milestone E: Commit + BlockStore + putRecord -------------------------------------------------------------------------------- - [x] BlockStore: cid → bytes, indexed by DID/rev/head - [x] Commit signing: UnsignedCommit → DAG-CBOR → sha256 → ECDSA sign - [x] Implement com.atproto.repo.putRecord/createRecord DoD: Write and read records by path/AT-URI -------------------------------------------------------------------------------- Milestone F: CAR Export + Sync Endpoints -------------------------------------------------------------------------------- - [x] CARv1 writer (roots = commit CID, blocks stream) - [x] Implement: sync.getRepo, sync.getBlocks, sync.getBlob DoD: External services can fetch repo snapshot + blocks -------------------------------------------------------------------------------- Milestone G: subscribeRepos Firehose -------------------------------------------------------------------------------- - [x] Monotonic sequence number + commit event generation - [x] WebSocket streaming for subscribeRepos DoD: Relay/client receives commit events after writes -------------------------------------------------------------------------------- Milestone H: Account + Sessions -------------------------------------------------------------------------------- - [x] Implement: server.createAccount, server.createSession, refreshSession - [x] Password/app-password hashing + JWT issuance DoD: Authenticate and write records with accessJwt -------------------------------------------------------------------------------- Milestone I: Lexicon Validation + Conformance -------------------------------------------------------------------------------- - [x] Lexicon validation for writes (app.bsky.* records) - [x] Conformance testing: diff CIDs/CARs/signatures vs reference PDS DoD: Same inputs → same outputs for repo/sync surfaces -------------------------------------------------------------------------------- Milestone J: Storage Backend Configuration -------------------------------------------------------------------------------- - [x] Configure SQLite WAL mode (PDS_SQLITE_DISABLE_WAL_AUTO_CHECKPOINT=true) - [x] Implement S3-compatible blobstore adapter (optional via config) - [x] Configure disk-based vs S3-based blob storage selection DoD: PDS runs with S3 blobs (if configured) and SQLite passes Litestream checks -------------------------------------------------------------------------------- Milestone K: Backup Automation + Guardrails -------------------------------------------------------------------------------- - [ ] Implement BackupOps module (scheduler/cron logic) - [ ] Automated backup jobs: - [ ] Databases (Litestream or raw copy) + /pds/actors backup - [ ] Local disk blobs (if applicable) - [ ] Guardrails & Monitoring: - [x] Uptime check endpoint: /xrpc/_health with JSON status - [x] Alerts: "Latest backup" too old, Disk pressure > 90% - [ ] Log retention policies DoD: - Backups run automatically and report status - Health checks indicate system state - Restore drill: Restore backups onto a fresh host passes verification - Backup set is explicitly documented ================================================================================ PHASE 2: DEPLOYMENT (Self-Host) ================================================================================ Milestone L: Topology + Domain Planning -------------------------------------------------------------------------------- - Choose PDS hostname (pds.example.com) vs handle domain (example.com) - Obtain domain, DNS access, VPS with static IP, reverse proxy DoD: Clear plan for PDS location, handle, and DID resolution -------------------------------------------------------------------------------- Milestone M: DNS + TLS + Reverse Proxy -------------------------------------------------------------------------------- - DNS A/AAAA records for PDS hostname - TLS certs (ACME) via Caddy DoD: https:// responds with valid cert -------------------------------------------------------------------------------- Milestone N: Deploy PDSharp -------------------------------------------------------------------------------- - Deploy built PDS with persistence (SQLite/Postgres + blob storage) - Verify /xrpc/com.atproto.server.describeServer DoD: describeServer returns capabilities payload -------------------------------------------------------------------------------- Milestone O: Account Creation -------------------------------------------------------------------------------- - Create account using admin tooling - Verify authentication: createSession DoD: Obtain session and perform authenticated write -------------------------------------------------------------------------------- Milestone P: Smoke Test Repo + Blobs -------------------------------------------------------------------------------- - Write record via putRecord - Upload blob, verify retrieval via sync.getBlob DoD: Posts appear in clients, media loads reliably -------------------------------------------------------------------------------- Milestone Q: Account Migration -------------------------------------------------------------------------------- - Export/import from bsky.social - Update DID service endpoint - Verify handle/DID resolution DoD: Handle unchanged, DID points to your PDS -------------------------------------------------------------------------------- Milestone R: Updates + Security -------------------------------------------------------------------------------- - Update cadence with rollback plan - Rate limits and access controls at proxy DoD: Update smoothly, maintain stable federation ================================================================================ QUICK CHECKLIST ================================================================================ [x] describeServer endpoint working [x] Crypto primitives (sha256, ECDSA p256/k256, low-S) [x] DAG-CBOR + CID generation correct [x] MST producing deterministic root CIDs [x] putRecord + blockstore operational [x] CAR export + sync endpoints [x] subscribeRepos firehose [x] Authentication (createAccount, createSession) [x] Lexicon validation [ ] Domain + TLS configured [ ] PDS deployed and reachable [ ] Account created, session works [ ] Writes + blobs verified [/] Backups + monitoring in place (health endpoint done, backup automation pending) ================================================================================ REFERENCES ================================================================================ https://atproto.com/guides/self-hosting https://github.com/bluesky-social/pds https://atproto.com/specs/repository https://atproto.com/specs/sync https://atproto.com/specs/blob https://docs.bsky.app/docs/api/com-atproto-server-describe-server https://docs.bsky.app/docs/api/com-atproto-server-create-session https://docs.bsky.app/docs/api/com-atproto-repo-put-record https://docs.bsky.app/docs/api/com-atproto-sync-get-blob