desertthunder.dev / PDSharp
an atproto pds written in F# (.NET 9) 🦒
pds fsharp giraffe dotnet atproto
fork atom
PDSharp / roadmap.txt
at main 8.9 kB view raw
1================================================================================ 2*roadmap.txt* PDSharp: F#/Giraffe PDS Implementation Roadmap 3================================================================================ 4PHASE 1: IMPLEMENTATION (Build) 5================================================================================ 6Milestone A: Giraffe XRPC Shell 7-------------------------------------------------------------------------------- 8- [x] Implement /xrpc/com.atproto.server.describeServer (GET) 9- [x] Implement NSID-based routing with structured error responses 10DoD: describeServer responds with stable JSON 11-------------------------------------------------------------------------------- 12Milestone B: Identity + Crypto Primitives 13-------------------------------------------------------------------------------- 14- [x] DID document fetch/parse for signing key and PDS endpoint 15- [x] SHA-256 hashing, ECDSA sign/verify (p256 + k256), low-S enforcement 16DoD: Sign and verify atproto commit hash with low-S 17-------------------------------------------------------------------------------- 18Milestone C: DAG-CBOR + CID 19-------------------------------------------------------------------------------- 20- [x] Canonical DAG-CBOR encode/decode with IPLD link tagging 21- [x] CID creation/parsing (multicodec dag-cbor, sha2-256) 22DoD: Record JSON → stable DAG-CBOR bytes → deterministic CID 23-------------------------------------------------------------------------------- 24Milestone D: MST Implementation 25-------------------------------------------------------------------------------- 26- [x] Merkle Search Tree per repository spec 27- [x] Key depth = leading zero bits in SHA-256(key) counted in 2-bit chunks 28- [x] Node encoding: (l, e[p,k,v,t]) with key prefix compression 29DoD: Insert/update/delete yields reproducible root CID 30-------------------------------------------------------------------------------- 31Milestone E: Commit + BlockStore + putRecord 32-------------------------------------------------------------------------------- 33- [x] BlockStore: cid → bytes, indexed by DID/rev/head 34- [x] Commit signing: UnsignedCommit → DAG-CBOR → sha256 → ECDSA sign 35- [x] Implement com.atproto.repo.putRecord/createRecord 36DoD: Write and read records by path/AT-URI 37-------------------------------------------------------------------------------- 38Milestone F: CAR Export + Sync Endpoints 39-------------------------------------------------------------------------------- 40- [x] CARv1 writer (roots = commit CID, blocks stream) 41- [x] Implement: sync.getRepo, sync.getBlocks, sync.getBlob 42DoD: External services can fetch repo snapshot + blocks 43-------------------------------------------------------------------------------- 44Milestone G: subscribeRepos Firehose 45-------------------------------------------------------------------------------- 46- [x] Monotonic sequence number + commit event generation 47- [x] WebSocket streaming for subscribeRepos 48DoD: Relay/client receives commit events after writes 49-------------------------------------------------------------------------------- 50Milestone H: Account + Sessions 51-------------------------------------------------------------------------------- 52- [x] Implement: server.createAccount, server.createSession, refreshSession 53- [x] Password/app-password hashing + JWT issuance 54DoD: Authenticate and write records with accessJwt 55-------------------------------------------------------------------------------- 56Milestone I: Lexicon Validation + Conformance 57-------------------------------------------------------------------------------- 58- [x] Lexicon validation for writes (app.bsky.* records) 59- [x] Conformance testing: diff CIDs/CARs/signatures vs reference PDS 60DoD: Same inputs → same outputs for repo/sync surfaces 61-------------------------------------------------------------------------------- 62Milestone J: Storage Backend Configuration 63-------------------------------------------------------------------------------- 64- [x] Configure SQLite WAL mode (PDS_SQLITE_DISABLE_WAL_AUTO_CHECKPOINT=true) 65- [x] Implement S3-compatible blobstore adapter (optional via config) 66- [x] Configure disk-based vs S3-based blob storage selection 67DoD: PDS runs with S3 blobs (if configured) and SQLite passes Litestream checks 68-------------------------------------------------------------------------------- 69Milestone K: Backup Automation + Guardrails 70-------------------------------------------------------------------------------- 71- [ ] Implement BackupOps module (scheduler/cron logic) 72- [ ] Automated backup jobs: 73 - [ ] Databases (Litestream or raw copy) + /pds/actors backup 74 - [ ] Local disk blobs (if applicable) 75- [ ] Guardrails & Monitoring: 76 - [x] Uptime check endpoint: /xrpc/_health with JSON status 77 - [x] Alerts: "Latest backup" too old, Disk pressure > 90% 78 - [ ] Log retention policies 79DoD: 80 - Backups run automatically and report status 81 - Health checks indicate system state 82 - Restore drill: Restore backups onto a fresh host passes verification 83 - Backup set is explicitly documented 84================================================================================ 85PHASE 2: DEPLOYMENT (Self-Host) 86================================================================================ 87Milestone L: Topology + Domain Planning 88-------------------------------------------------------------------------------- 89- Choose PDS hostname (pds.example.com) vs handle domain (example.com) 90- Obtain domain, DNS access, VPS with static IP, reverse proxy 91DoD: Clear plan for PDS location, handle, and DID resolution 92-------------------------------------------------------------------------------- 93Milestone M: DNS + TLS + Reverse Proxy 94-------------------------------------------------------------------------------- 95- DNS A/AAAA records for PDS hostname 96- TLS certs (ACME) via Caddy 97DoD: https://<pds-hostname> responds with valid cert 98-------------------------------------------------------------------------------- 99Milestone N: Deploy PDSharp 100-------------------------------------------------------------------------------- 101- Deploy built PDS with persistence (SQLite/Postgres + blob storage) 102- Verify /xrpc/com.atproto.server.describeServer 103DoD: describeServer returns capabilities payload 104-------------------------------------------------------------------------------- 105Milestone O: Account Creation 106-------------------------------------------------------------------------------- 107- Create account using admin tooling 108- Verify authentication: createSession 109DoD: Obtain session and perform authenticated write 110-------------------------------------------------------------------------------- 111Milestone P: Smoke Test Repo + Blobs 112-------------------------------------------------------------------------------- 113- Write record via putRecord 114- Upload blob, verify retrieval via sync.getBlob 115DoD: Posts appear in clients, media loads reliably 116-------------------------------------------------------------------------------- 117Milestone Q: Account Migration 118-------------------------------------------------------------------------------- 119- Export/import from bsky.social 120- Update DID service endpoint 121- Verify handle/DID resolution 122DoD: Handle unchanged, DID points to your PDS 123-------------------------------------------------------------------------------- 124Milestone R: Updates + Security 125-------------------------------------------------------------------------------- 126- Update cadence with rollback plan 127- Rate limits and access controls at proxy 128DoD: Update smoothly, maintain stable federation 129================================================================================ 130QUICK CHECKLIST 131================================================================================ 132[x] describeServer endpoint working 133[x] Crypto primitives (sha256, ECDSA p256/k256, low-S) 134[x] DAG-CBOR + CID generation correct 135[x] MST producing deterministic root CIDs 136[x] putRecord + blockstore operational 137[x] CAR export + sync endpoints 138[x] subscribeRepos firehose 139[x] Authentication (createAccount, createSession) 140[x] Lexicon validation 141[ ] Domain + TLS configured 142[ ] PDS deployed and reachable 143[ ] Account created, session works 144[ ] Writes + blobs verified 145[/] Backups + monitoring in place (health endpoint done, backup automation pending) 146================================================================================ 147REFERENCES 148================================================================================ 149https://atproto.com/guides/self-hosting 150https://github.com/bluesky-social/pds 151https://atproto.com/specs/repository 152https://atproto.com/specs/sync 153https://atproto.com/specs/blob 154https://docs.bsky.app/docs/api/com-atproto-server-describe-server 155https://docs.bsky.app/docs/api/com-atproto-server-create-session 156https://docs.bsky.app/docs/api/com-atproto-repo-put-record 157https://docs.bsky.app/docs/api/com-atproto-sync-get-blob