···33 push:
44 branches:
55 - main
66- - pdsv2
66+ - next
77 tags:
88 - v*
99env:
+7-6
ACCOUNT_MIGRATION.md
···5566Therefore, we do not recommend migrating your primary account yet. And we specifically recommend _against_ migrating your main account if you do not understand how PLC operations work.
7788-As well, the Bluesky PDS is not currently accepting incoming migrations (it will in the future). Therefore this is currently a one-way street. If you migrate off of `bsky.social`, _you will not be able to return_. However, you will be able to migrate between other PDSs.
88+Also, the Bluesky PDS is not currently accepting incoming migrations (it will in the future). Therefore this is currently a one-way street. If you migrate off of `bsky.social`, _you will not be able to return_. However, you will be able to migrate between other PDSs.
991010
1111···20202121In order to create an account, you first need to prove to the new PDS that you're in control of the DID that you're attempting to register as.
22222323-To do so, you need a JWT signed with the signing key associated with your DID. You can obtain this through calling `com.atproto.server.getServiceAuth` from your old PDS. If you're old PDS is not willing to provide the authentication token, you will need to update your DID document to point to a signing key that you possess in order to mint an authentication token yourself.
2323+To do so, you need a JWT signed with the signing key associated with your DID. You can obtain this through calling `com.atproto.server.getServiceAuth` from your old PDS. If your old PDS is not willing to provide the authentication token, you will need to update your DID document to point to a signing key that you possess in order to mint an authentication token yourself.
24242525With this JWT set as a Bearer token, you can then create an account on the new PDS by calling `com.atproto.server.createAccount`. You'll need to fulfill any challenges that the new PDS requires - such as an invite code.
26262727-After creating an account, you'll have a signing key on the new PDS and an empty repository. You're account will be in a "deactivated" state such that it is not usable yet.
2727+After creating an account, you'll have a signing key on the new PDS and an empty repository. Your account will be in a "deactivated" state such that it is not usable yet.
28282929### Migrating data
30303131Now that you have an account on the new PDS, you can start migrating data into it. After creating your account, you will have received an access token for the new PDS and it will be required for all incoming data.
32323333-First, you can grab your entire repository in the from of a [CAR file](https://ipld.io/specs/transport/car/carv1/) by calling `com.atproto.sync.getRepo`. You can then upload those exact bytes to your new PDS through `com.atproto.repo.importRepo`. The new PDS will parse the CAR file, index all blocks and records, and sign a new commit for the repository.
3333+First, you can grab your entire repository in the form of a [CAR file](https://ipld.io/specs/transport/car/carv1/) by calling `com.atproto.sync.getRepo`. You can then upload those exact bytes to your new PDS through `com.atproto.repo.importRepo`. The new PDS will parse the CAR file, index all blocks and records, and sign a new commit for the repository.
34343535Next, you'll need to upload all relevant blobs. These can be discovered by calling `com.atproto.sync.listBlobs` on your old PDS. For each blob, you'll need to download the contents through `com.atproto.sync.getBlob` and upload them to your new PDS through `com.atproto.repo.uploadBlob`.
3636···69697070The below Typescript code gives an example of how this account migration flow may function. Please note that it is for documentation purposes only and can not be run exactly as is as there is an out-of-band step where you need to get a confirmation token from your email.
71717272-It does also not handle some of the more advanced steps such as verifying a full import, looking for missing blobs, adding your own recovery key, or validating the PLC operation itself.
7272+It also does not handle some of the more advanced steps such as verifying a full import, looking for missing blobs, adding your own recovery key, or validating the PLC operation itself.
73737474```ts
7575import AtpAgent from '@atproto/api'
···107107108108 const serviceJwtRes = await oldAgent.com.atproto.server.getServiceAuth({
109109 aud: newServerDid,
110110+ lxm: 'com.atproto.server.createAccount',
110111 })
111112 const serviceJwt = serviceJwtRes.data.token
112113···200201 await oldAgent.com.atproto.server.deactivateAccount({})
201202}
202203203203-```204204+```
+2-1
Dockerfile
···11FROM node:20.11-alpine3.18 as build
2233-RUN npm install -g pnpm
33+RUN corepack enable
4455# Move files into the image and install
66WORKDIR /app
77COPY ./service ./
88+RUN corepack prepare --activate
89RUN pnpm install --production --frozen-lockfile > /dev/null
9101011# Uses assets from build stage to reduce build size
+67-22
README.md
···11# PDS
2233-Welcome to the repository for the official Bluesky PDS (Personal Data Server). This repository includes container images and documentation designed to assist technical people with self-hosting a Bluesky PDS.
33+Welcome to the repository for the official Bluesky PDS (Personal Data Server). This repository includes container images and documentation designed to assist technical people with hosting a Bluesky PDS.
4455-Head over to the [AT Protocol PDS Admins Discord](https://discord.gg/UWS6FFdhMe) to get started!
55+Head over to the [AT Protocol PDS Admins Discord](https://discord.gg/e7hpHxRfBP) to chat with other folks hosting instances and get important updates about the PDS distribution!
6677## Table of Contents
88···1515 * [What is AT Protocol?](#what-is-at-protocol)
1616 * [Where is the code?](#where-is-the-code)
1717 * [What is the current status of federation?](#what-is-the-current-status-of-federation)
1818- * [What should I know about running a PDS in the developer sandbox?](#what-should-i-know-about-running-a-pds-in-the-developer-sandbox)
1918- [Self-hosting PDS](#self-hosting-pds)
2019 * [Preparation for self-hosting PDS](#preparation-for-self-hosting-pds)
2120 * [Open your cloud firewall for HTTP and HTTPS](#open-your-cloud-firewall-for-http-and-https)
···2625 * [Creating an account using pdsadmin](#creating-an-account-using-pdsadmin)
2726 * [Creating an account using an invite code](#creating-an-account-using-an-invite-code)
2827 * [Using the Bluesky app with your PDS](#using-the-bluesky-app-with-your-pds)
2828+ * [Setting up SMTP](#setting-up-smtp)
2929 * [Updating your PDS](#updating-your-pds)
30303131<!-- tocstop -->
···46464747### Where is the code?
48484949-* [TypeScript code](https://github.com/bluesky-social/atproto)
4949+* [TypeScript code](https://github.com/bluesky-social/atproto/tree/main/packages/pds)
5050* [Go code](https://github.com/bluesky-social/indigo)
51515252### What is the current status of federation?
53535454-As of Feb, 2024, the AT Protocol PDS (Personal Data Server) is now open to federation for self-hosters!
5454+As of Spring 2024, the AT Protocol network is open to federation!
55555656โ Federated domain handles (e.g. `@nytimes.com`)
5757···61616262โ Federated app views (API service)
63636464-โ Federated data for self-hosters (PDS hosting)
6464+โ Federated data (PDS hosting)
65656666โ Federated moderation (labeling)
67676868-๐ง Federated data for large service providers (coming soon)
6969-7070-### What should I know about running a PDS in the developer sandbox?
7171-7272-Developers may now run self-hosted PDS hosts on the production network!
7373-7474-Though it is still recommended to run experiments in the developer sandbox network.
7575-7676-Read the [SANDBOX.md](https://github.com/bluesky-social/pds/blob/main/SANDBOX.md) for an overview of the sandbox network.
7777-7868## Self-hosting PDS
79698070Self-hosting a Bluesky PDS means running your own Personal Data Server that is capable of federating with the wider Bluesky social network.
81718282-> [!IMPORTANT]
8383-> Initially to join the network you'll need to join the [AT Protocol PDS Admins Discord](https://discord.gg/UWS6FFdhMe) and register the hostname of your PDS. We recommend doing so before bringing your PDS online. In the future, this registration check will not be required.
8484-8572### Preparation for self-hosting PDS
86738774Launch a server on any cloud provider, [Digital Ocean](https://digitalocean.com/) and [Vultr](https://vultr.com/) are two popular choices.
···165152166153### Verifying that your PDS is online and accessible
167154168168-You can check if your server is online and healthy by requesting the healthcheck endpoint.
155155+> [!TIP]
156156+> The most common problems with getting PDS content consumed in the live network are when folks substitute the provided Caddy configuration for nginx, apache, or similar reverse proxies. Getting TLS certificates, WebSockets, and virtual server names all correct can be tricky. We are not currently providing tech support for other configurations.
169157170170-You can visit `https://example.com/xrpc/_health` in your browser. You should see a JSON response with a version.
158158+You can check if your server is online and healthy by requesting the healthcheck endpoint.
171159172172-For example:
160160+You can visit `https://example.com/xrpc/_health` in your browser. You should see a JSON response with a version, like:
173161174162```
175163{"version":"0.2.2-beta.2"}
176164```
165165+166166+You'll also need to check that WebSockets are working, for the rest of the network to pick up content from your PDS. You can test by installing a tool like `wsdump` and running a command like:
167167+168168+```bash
169169+wsdump "wss://example.com/xrpc/com.atproto.sync.subscribeRepos?cursor=0"
170170+```
171171+172172+Note that there will be no events output on the WebSocket until they are created in the PDS, so the above command may continue to run with no output if things are configured successfully.
177173178174### Creating an account using pdsadmin
179175···2042001. Enter the URL of your PDS (e.g. `https://example.com/`)
205201206202_Note: because the subdomain TLS certificate is created on-demand, it may take 10-30s for your handle to be accessible. If you aren't seeing your first post/profile, wait 30s and try to make another post._
203203+204204+### Setting up SMTP
205205+206206+To be able to verify users' email addresses and send other emails, you need to set up an SMTP server.
207207+208208+One way to do this is to use an email service. [Resend](https://resend.com/) and [SendGrid](https://sendgrid.com/) are two popular choices.
209209+210210+Create an account and API key on an email service, ensure your server allows access on the required ports, and set these variables in `/pds/pds.env` (example with Resend):
211211+212212+```
213213+PDS_EMAIL_SMTP_URL=smtps://resend:<your api key here>@smtp.resend.com:465/
214214+PDS_EMAIL_FROM_ADDRESS=admin@your.domain
215215+```
216216+217217+If you prefer to use a standard SMTP server (a local one or from your email provider), put your account's username and password in the URL:
218218+219219+```
220220+PDS_EMAIL_SMTP_URL=smtps://username:password@smtp.example.com/
221221+```
222222+223223+Alternatively, if you're running a local sendmail-compatible mail service like Postfix or Exim on the same host, you can configure the PDS to use the sendmail transport by using such URL:
224224+225225+```
226226+PDS_EMAIL_SMTP_URL=smtp:///?sendmail=true
227227+```
228228+229229+_Note: Your PDS will need to be restarted with those variables. This varies depending on your setup. If you followed this installation guide, run `systemctl restart pds`. You might need to restart the server or recreate the container, depending on what you are using._
230230+231231+### Logging
232232+233233+By default, logs from the PDS are printed to `stdout` and end up in Docker's log. You can browse them by running:
234234+235235+```
236236+[sudo] docker logs pds
237237+```
238238+239239+Note: these logs are not persisted, so they will be lost after server reboot.
240240+241241+Alternatively, you can configure the logs to be printed to a file by setting `LOG_DESTINATION`:
242242+243243+```
244244+LOG_DESTINATION=/pds/pds.log
245245+```
246246+247247+You can also change the minimum level of logs to be printed (default: `info`):
248248+249249+```
250250+LOG_LEVEL=debug
251251+```
207252208253### Updating your PDS
209254
-168
SANDBOX.md
···11-# Bluesky Developer Sandbox Guide
22-33-Welcome to the atproto federation developer sandbox!
44-55-This is a completely separate network from our production services that allows us to test out the federation architecture and wire protocol.
66-77-The federation sandbox environment is an area set up for exploration and testing of the technical components of the AT Protocol distributed social network. It is intended for developers and self-hosters to test out data availability in a federated environment.
88-99-To maintain a positive and productive developer experience, we've established this Code of Conduct that outlines our expectations and guidelines. This sandbox environment is initially meant to test the technical components of federation.
1010-1111-Given that this is a testing environment, we will be defederating from any instances that do not abide by these guidelines, or that cause unnecessary trouble, and will not be providing specific justifications for these decisions.
1212-1313-<!-- markdown-toc -i SANDBOX.md -->
1414-1515-<!-- toc -->
1616-1717-- [Guidelines that must be followed](#guidelines-that-must-be-followed)
1818- * [Post responsibly](#post-responsibly)
1919- * [Keep the emphasis on testing](#keep-the-emphasis-on-testing)
2020- * [Do limit account creation](#do-limit-account-creation)
2121- * [Donโt expect persistence or uptime](#dont-expect-persistence-or-uptime)
2222- * [Don't advertise your service as being "Bluesky"](#dont-advertise-your-service-as-being-bluesky)
2323- * [Do not mirror sandbox did:plcs to production](#do-not-mirror-sandbox-didplcs-to-production)
2424- * [Status and Wipes](#status-and-wipes)
2525- + [๐ Beware of dragons!](#%F0%9F%90%89-beware-of-dragons)
2626- + [Routine wipes](#routine-wipes)
2727-- [Getting started](#getting-started)
2828- * [Auto-updates](#auto-updates)
2929- * [Odds & Ends & Warnings & Reminders](#odds--ends--warnings--reminders)
3030-- [Learn more about atproto federation](#learn-more-about-atproto-federation)
3131- * [Network Services](#network-services)
3232- + [PLC](#plc)
3333- + [BGS](#bgs)
3434- + [Bluesky App View](#bluesky-app-view)
3535-- [The PDS](#the-pds)
3636-3737-<!-- tocstop -->
3838-3939-# Guidelines that must be followed
4040-4141-Using the sandbox environment means you agree to adhere to our Guidelines. Please read the following carefully:
4242-4343-## Post responsibly
4444-4545-The sandbox environment is intended to test infrastructure, but user content may be created as part of this testing process. Content generation can be automated or manual.
4646-4747-Do not post content that requires active moderation or violates the [Bluesky Community Guidelines](https://bsky.social/about/support/community-guidelines).
4848-4949-## Keep the emphasis on testing
5050-5151-Weโre striving to maintain a sandbox environment that fosters learning and technical growth. We will defederate with instances that recruit users without making it clear that this is a test environment.
5252-5353-## Do limit account creation
5454-5555-We don't want any one server using a majority of the resources in the sandbox. To keep things balanced, to start, weโre only federating with Personal Data Servers (PDS) with up to 1000 accounts. However, we may change this if needed.
5656-5757-## Donโt expect persistence or uptime
5858-5959-We will routinely be wiping the data on our infrastructure. This is intended to reset the network state and to test sync protocols. Accounts and content should not be mirrored or migrated between the sandbox and real-world environments.
6060-6161-## Don't advertise your service as being "Bluesky"
6262-6363-This is a developer sandbox and is meant for technical users. Do not promote your service as being a way for non-technical users to use Bluesky.
6464-6565-## Do not mirror sandbox did:plcs to production
6666-6767-6868-## Status and Wipes
6969-7070-### ๐ย Beware of dragons!
7171-7272-This hasnโt been production tested yet. It seems to work pretty well, but who knows whatโs lurking under the surface โ that's what this sandbox is for! Have patience with us as we prep for federation.
7373-7474-On that note, please give us feedback either in [Issues](https://github.com/bluesky-social/atproto/issues) (actual bugs) or [Discussions](https://github.com/bluesky-social/atproto/discussions) (higher-level questions/discussions) on the [atproto repo](https://github.com/bluesky-social/atproto).
7575-7676-### Routine wipes
7777-7878-As part of the sandbox, we will be doing routine wipes of all network data.
7979-8080-We expect to perform wipes on a weekly or bi-weekly basis, though we reserve the right to do a wipe at any point.
8181-8282-When we wipe data, we will be wiping it on all services (BGS, App View, PLC). We will also mark any existing DIDs as โinvalidโ & will refuse to index those accounts in the next epoch of the network to discourage users from attempting to โrolloverโ their accounts across wipes.
8383-8484-# Getting started
8585-8686-For complete instructions on getting your PDS set up, check out the [README](./README.md).
8787-8888-To access your account, youโll log in with the client of your choice in the exact same way that you log into production Bluesky, for instance the [Bluesky web client](https://app.bsky-sandbox.dev/). When you do so, please provide the url of *your PDS* as the service that you wish to log in to.
8989-9090-## Auto-updates
9191-9292-Weโve included Watchtower in the PDS distribution. Every day at midnight PST, this will check our GitHub container registry to see if there is a new version of the PDS container & update it on your service.
9393-9494-This will allow us to rapidly iterate on protocol changes, as weโll be able to push them out to the network on a daily basis.
9595-9696-When we do routine network wipes, we will be pushing out a database migration to participating PDS that wipes content and accounts.
9797-9898-You are within your rights to disable Watchtower auto-updates, but we strongly encourage their use and will not be providing support if you decide not to run the most up-to-date PDS distribution.
9999-100100-## Odds & Ends & Warnings & Reminders
101101-102102-๐งช Experiment & have fun!
103103-104104-๐คย Run [feed generators](https://github.com/bluesky-social/feed-generator). They should work the exact same way as production - be sure to adjust your env to listen to Sandbox BGS!
105105-106106-๐ Feel free to run your own AppView or BGS - although itโs a bit more involved & weโll be providing limited support for this.
107107-108108-โ๏ธ Because the atproto network is a distributed system, your PDS can no longer definitively read-after-write. Updates are generally processed pretty quickly, however this discrepancy may show in certain circumstances, such as updating a profile or replying to a thread. We're working on utilities to make this easier to handle.
109109-110110-โฑ๏ธ As a specific case of the above, because we use on-demand TLS with Caddy, your profile may not load at first - please be patient & it should load within 5-10s after account creation. Again, we'll be working to smooth over this.
111111-112112-๐คย Your PDS will provide your handle by default. Custom domain handles should work exactly the same in sandbox as they do on production Bluesky. Although you will not be able to re-use your handle from production Bluesky as you can only have one DID set per handle.
113113-114114-โ ๏ธ If you follow the self-hosted PDS setup instructions, youโll have private key material in your env file - be careful about sharing that!
115115-116116-๐ฃย This is a sandbox version of a **public broadcast protocol** - please do not share sensitive information.
117117-118118-๐คย Help each other out! Respond to issues & discussions, chat in [Matrix](https://matrix.to/#/%23bluesky-dev:matrix.org) or the community-run [Discord](https://discord.gg/3srmDsHSZJ), etc.
119119-120120-# Learn more about atproto federation
121121-122122-Check out the [high-level view of federation](https://bsky.social/about/blog/5-5-2023-federation-architecture).
123123-124124-Dive deeper with the [atproto docs](https://atproto.com/docs).
125125-126126-## Network Services
127127-128128-We are running three services: PLC, BGS, Bluesky "App View"
129129-130130-### PLC
131131-132132-**Hostname:** `plc.bsky-sandbox.dev`
133133-134134-**Code:** https://github.com/bluesky-social/did-method-plc
135135-136136-PLC is the default DID provider for the network. DIDs are the root of your identity in the network. Sandbox PLC functions exactly the same as production PLC, but it is run as a separate service with a separate dataset. The DID resolution client in the self-hosted PDS package is set up to talk the correct PLC service.
137137-138138-### BGS
139139-140140-**Hostname:** `bgs.bsky-sandbox.dev`
141141-142142-**Code:** https://github.com/bluesky-social/indigo/tree/main/bgs
143143-144144-BGS (Big Graph Service) is the firehose for the entire network. It collates data from PDSs & rebroadcasts them out on one giant websocket.
145145-146146-BGS has to find out about your server somehow, so when we do any sort of write, we ping BGS with `com.atproto.sync.requestCrawl` to notify it of new data. This is done automatically in the self-hosted PDS package.
147147-148148-If youโre familiar with the Bluesky production firehose, you can subscribe to the BGS firehose in the exact same manner, the interface & data should be identical
149149-150150-### Bluesky App View
151151-152152-**Hostname:** `api.bsky-sandbox.dev`
153153-154154-**Code:** https://github.com/bluesky-social/atproto/tree/main/packages/bsky
155155-156156-The Bluesky App View aggregates data from across the network to service the Bluesky microblogging application. It consumes the firehose from the BGS, processing it into serviceable views of the network such as feeds, post threads, and user profiles. It functions as a fairly traditional web service.
157157-158158-When you request a Bluesky-related view from your PDS (`getProfile` for instance), your PDS will actually proxy the request up to App View.
159159-160160-Feel free to experiment with running your own App View if you like!
161161-162162-# The PDS
163163-164164-The PDS (Personal Data Server) is where users host their social data such as posts, profiles, likes, and follows. The goal of the sandbox is to federate many PDS together, so we hope youโll run your own.
165165-166166-Weโre not actually running a Bluesky PDS in sandbox. You might see Bluesky team members' accounts in the sandbox environment, but those are self-hosted too.
167167-168168-The PDS that youโll be running is much of the same code that is running on the Bluesky production PDS. Notably, all of the in-pds-appview code has been torn out. You can see the actual PDS code that youโre running on the [atproto/simplify-pds](https://github.com/bluesky-social/atproto/pull/1198) branch.
···2525 Takedown an account specified by DID.
2626 e.g. pdsadmin account takedown did:plc:xyz123abc456
2727 untakedown <DID>
2828- Remove a takedown an account specified by DID.
2828+ Remove a takedown from an account specified by DID.
2929 e.g. pdsadmin account untakedown did:plc:xyz123abc456
3030- password-reset <DID>
3030+ reset-password <DID>
3131 Reset a password for an account specified by DID.
3232 e.g. pdsadmin account reset-password did:plc:xyz123abc456
3333