A community based topic aggregation platform built on atproto
bretton.dev
feat(kagi): add automated registration script and docs
Add Kagi-specific automated registration script and update README.
Changes:
- Move setup-kagi-aggregator.sh to kagi-news/scripts/setup.sh
- Add comprehensive Registration section to README
- Document automated vs manual setup options
- Explain registration workflow and requirements
- Update project structure to reflect new scripts
The setup script automates all 4 registration steps:
1. PDS account creation
2. .well-known file generation
3. Coves registration via XRPC
4. Service declaration creation
This makes the Kagi aggregator self-contained and ready to be
split into its own repository.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
+188
-1
aggregators/kagi-news/README.md
+188
-1
aggregators/kagi-news/README.md
···
31
31
│ ├── sample_rss_item.xml
32
32
│ └── world.xml
33
33
├── scripts/
34
-
│ └── generate_did.py # Helper to generate aggregator DID (TODO)
34
+
│ └── setup.sh # Automated Coves registration script
35
+
├── Dockerfile # Docker image definition
36
+
├── docker-compose.yml # Docker Compose configuration
37
+
├── docker-entrypoint.sh # Container entrypoint script
38
+
├── .dockerignore # Docker build exclusions
35
39
├── requirements.txt # Python dependencies
36
40
├── config.example.yaml # Example configuration
37
41
├── .env.example # Environment variables template
···
39
43
└── README.md
40
44
```
41
45
46
+
## Registration with Coves
47
+
48
+
Before running the aggregator, you must register it with a Coves instance. This creates a DID for your aggregator and registers it with Coves.
49
+
50
+
### Quick Setup (Automated)
51
+
52
+
The automated setup script handles the entire registration process:
53
+
54
+
```bash
55
+
cd scripts
56
+
chmod +x setup.sh
57
+
./setup.sh
58
+
```
59
+
60
+
This will:
61
+
1. **Create a PDS account** for your aggregator (generates a DID)
62
+
2. **Generate `.well-known/atproto-did`** file for domain verification
63
+
3. **Pause for manual upload** - you'll upload the file to your web server
64
+
4. **Register with Coves** instance via XRPC
65
+
5. **Create service declaration** record (indexed by Jetstream)
66
+
67
+
**Manual step required:** During the process, you'll need to upload the `.well-known/atproto-did` file to your domain so it's accessible at `https://yourdomain.com/.well-known/atproto-did`.
68
+
69
+
After completion, you'll have a `kagi-aggregator-config.env` file with:
70
+
- Aggregator DID and credentials
71
+
- Access/refresh JWTs
72
+
- Service declaration URI
73
+
74
+
**Keep this file secure!** It contains your aggregator's credentials.
75
+
76
+
### Manual Setup (Step-by-step)
77
+
78
+
Alternatively, use the generic setup scripts from the main Coves repo for more control:
79
+
80
+
```bash
81
+
# From the Coves project root
82
+
cd scripts/aggregator-setup
83
+
84
+
# Follow the 4-step process
85
+
./1-create-pds-account.sh
86
+
./2-setup-wellknown.sh
87
+
./3-register-with-coves.sh
88
+
./4-create-service-declaration.sh
89
+
```
90
+
91
+
See [scripts/aggregator-setup/README.md](../../scripts/aggregator-setup/README.md) for detailed documentation on each step.
92
+
93
+
### What Happens During Registration?
94
+
95
+
1. **PDS Account Creation**: Your aggregator gets a `did:plc:...` identifier
96
+
2. **Domain Verification**: Proves you control your aggregator's domain
97
+
3. **Coves Registration**: Inserts your DID into the Coves instance's `users` table
98
+
4. **Service Declaration**: Creates a record that gets indexed into the `aggregators` table
99
+
5. **Ready for Authorization**: Community moderators can now authorize your aggregator
100
+
101
+
Once registered and authorized by a community, your aggregator can post content.
102
+
42
103
## Setup
43
104
44
105
### Prerequisites
45
106
46
107
- Python 3.11+
47
108
- python3-venv package (`apt install python3.12-venv`)
109
+
- **Completed registration** (see above)
48
110
49
111
### Installation
50
112
···
83
145
# Run with coverage
84
146
pytest --cov=src --cov-report=html
85
147
```
148
+
149
+
## Deployment
150
+
151
+
### Docker Deployment (Recommended for Production)
152
+
153
+
The easiest way to deploy the Kagi aggregator is using Docker. The cron job runs inside the container automatically.
154
+
155
+
#### Prerequisites
156
+
157
+
- Docker and Docker Compose installed
158
+
- Completed registration (you have `.env` with credentials)
159
+
- `config.yaml` configured with your feed mappings
160
+
161
+
#### Quick Start
162
+
163
+
1. **Configure your environment:**
164
+
```bash
165
+
# Copy and edit configuration
166
+
cp config.example.yaml config.yaml
167
+
cp .env.example .env
168
+
169
+
# Edit .env with your aggregator credentials
170
+
nano .env
171
+
```
172
+
173
+
2. **Start the aggregator:**
174
+
```bash
175
+
docker compose up -d
176
+
```
177
+
178
+
3. **View logs:**
179
+
```bash
180
+
docker compose logs -f
181
+
```
182
+
183
+
4. **Stop the aggregator:**
184
+
```bash
185
+
docker compose down
186
+
```
187
+
188
+
#### Configuration
189
+
190
+
The `docker-compose.yml` file supports these environment variables:
191
+
192
+
- **`AGGREGATOR_HANDLE`** (required): Your aggregator's handle
193
+
- **`AGGREGATOR_PASSWORD`** (required): Your aggregator's password
194
+
- **`COVES_API_URL`** (optional): Override Coves API endpoint (defaults to `https://api.coves.social`)
195
+
- **`RUN_ON_STARTUP`** (optional): Set to `true` to run immediately on container start (useful for testing)
196
+
197
+
#### Testing the Setup
198
+
199
+
Run the aggregator immediately without waiting for cron:
200
+
201
+
```bash
202
+
# Run once and exit
203
+
docker compose run --rm kagi-aggregator python -m src.main
204
+
205
+
# Or set RUN_ON_STARTUP=true in .env and restart
206
+
docker compose restart
207
+
```
208
+
209
+
#### Production Deployment
210
+
211
+
For production, consider:
212
+
213
+
1. **Using Docker Secrets** for credentials:
214
+
```yaml
215
+
secrets:
216
+
aggregator_credentials:
217
+
file: ./secrets/aggregator.env
218
+
```
219
+
220
+
2. **Setting up log rotation** (already configured in docker-compose.yml):
221
+
- Max size: 10MB per file
222
+
- Max files: 3
223
+
224
+
3. **Monitoring health checks:**
225
+
```bash
226
+
docker inspect --format='{{.State.Health.Status}}' kagi-news-aggregator
227
+
```
228
+
229
+
4. **Auto-restart on failure** (already enabled with `restart: unless-stopped`)
230
+
231
+
#### Viewing Cron Logs
232
+
233
+
```bash
234
+
# Follow cron execution logs
235
+
docker compose logs -f kagi-aggregator
236
+
237
+
# View last 100 lines
238
+
docker compose logs --tail=100 kagi-aggregator
239
+
```
240
+
241
+
#### Updating the Aggregator
242
+
243
+
```bash
244
+
# Pull latest code
245
+
git pull
246
+
247
+
# Rebuild and restart
248
+
docker compose up -d --build
249
+
```
250
+
251
+
### Manual Deployment (Alternative)
252
+
253
+
If you prefer running without Docker, use the traditional approach:
254
+
255
+
1. **Install dependencies:**
256
+
```bash
257
+
python3 -m venv venv
258
+
source venv/bin/activate
259
+
pip install -r requirements.txt
260
+
```
261
+
262
+
2. **Configure crontab:**
263
+
```bash
264
+
# Edit the crontab file with your paths
265
+
# Then install it:
266
+
crontab crontab
267
+
```
268
+
269
+
3. **Verify cron is running:**
270
+
```bash
271
+
crontab -l
272
+
```
86
273
87
274
## Development Status
88
275
+195
aggregators/kagi-news/scripts/setup.sh
+195
aggregators/kagi-news/scripts/setup.sh
···
1
+
#!/bin/bash
2
+
3
+
# Script: setup-kagi-aggregator.sh
4
+
# Purpose: Complete setup script for Kagi News RSS aggregator
5
+
#
6
+
# This is a reference implementation showing automated setup for a specific aggregator.
7
+
# Other aggregator developers can use this as a template.
8
+
9
+
set -e
10
+
11
+
echo "================================================"
12
+
echo "Kagi News RSS Aggregator - Automated Setup"
13
+
echo "================================================"
14
+
echo ""
15
+
16
+
# Configuration for Kagi aggregator
17
+
AGGREGATOR_NAME="kagi-news-bot"
18
+
DISPLAY_NAME="Kagi News RSS"
19
+
DESCRIPTION="Aggregates tech news from Kagi RSS feeds and posts to relevant communities"
20
+
SOURCE_URL="https://github.com/coves-social/kagi-aggregator"
21
+
22
+
# Check if config already exists
23
+
if [ -f "kagi-aggregator-config.env" ]; then
24
+
echo "Configuration file already exists. Loading existing configuration..."
25
+
source kagi-aggregator-config.env
26
+
SKIP_ACCOUNT_CREATION=true
27
+
else
28
+
SKIP_ACCOUNT_CREATION=false
29
+
fi
30
+
31
+
# Get runtime configuration
32
+
if [ "$SKIP_ACCOUNT_CREATION" = false ]; then
33
+
read -p "Enter PDS URL (default: https://bsky.social): " PDS_URL
34
+
PDS_URL=${PDS_URL:-https://bsky.social}
35
+
36
+
read -p "Enter email for bot account: " EMAIL
37
+
read -sp "Enter password for bot account: " PASSWORD
38
+
echo ""
39
+
40
+
# Generate handle
41
+
TIMESTAMP=$(date +%s)
42
+
HANDLE="$AGGREGATOR_NAME-$TIMESTAMP.bsky.social"
43
+
44
+
echo ""
45
+
echo "Creating PDS account..."
46
+
echo "Handle: $HANDLE"
47
+
48
+
# Create account
49
+
RESPONSE=$(curl -s -X POST "$PDS_URL/xrpc/com.atproto.server.createAccount" \
50
+
-H "Content-Type: application/json" \
51
+
-d "{
52
+
\"handle\": \"$HANDLE\",
53
+
\"email\": \"$EMAIL\",
54
+
\"password\": \"$PASSWORD\"
55
+
}")
56
+
57
+
if echo "$RESPONSE" | jq -e '.error' > /dev/null 2>&1; then
58
+
echo "✗ Error creating account:"
59
+
echo "$RESPONSE" | jq '.'
60
+
exit 1
61
+
fi
62
+
63
+
DID=$(echo "$RESPONSE" | jq -r '.did')
64
+
ACCESS_JWT=$(echo "$RESPONSE" | jq -r '.accessJwt')
65
+
REFRESH_JWT=$(echo "$RESPONSE" | jq -r '.refreshJwt')
66
+
67
+
echo "✓ Account created: $DID"
68
+
69
+
# Save configuration
70
+
cat > kagi-aggregator-config.env <<EOF
71
+
# Kagi Aggregator Configuration
72
+
AGGREGATOR_DID="$DID"
73
+
AGGREGATOR_HANDLE="$HANDLE"
74
+
AGGREGATOR_PDS_URL="$PDS_URL"
75
+
AGGREGATOR_EMAIL="$EMAIL"
76
+
AGGREGATOR_PASSWORD="$PASSWORD"
77
+
AGGREGATOR_ACCESS_JWT="$ACCESS_JWT"
78
+
AGGREGATOR_REFRESH_JWT="$REFRESH_JWT"
79
+
EOF
80
+
81
+
echo "✓ Configuration saved to kagi-aggregator-config.env"
82
+
fi
83
+
84
+
# Get domain and Coves instance
85
+
read -p "Enter aggregator domain (e.g., kagi-news.example.com): " DOMAIN
86
+
read -p "Enter Coves instance URL (default: https://api.coves.social): " COVES_URL
87
+
COVES_URL=${COVES_URL:-https://api.coves.social}
88
+
89
+
# Setup .well-known
90
+
echo ""
91
+
echo "Setting up .well-known/atproto-did..."
92
+
mkdir -p .well-known
93
+
echo "$DID" > .well-known/atproto-did
94
+
echo "✓ Created .well-known/atproto-did"
95
+
96
+
echo ""
97
+
echo "================================================"
98
+
echo "IMPORTANT: Manual Step Required"
99
+
echo "================================================"
100
+
echo ""
101
+
echo "Upload the .well-known directory to your web server at:"
102
+
echo " https://$DOMAIN/.well-known/atproto-did"
103
+
echo ""
104
+
read -p "Press Enter when the file is uploaded and accessible..."
105
+
106
+
# Verify .well-known
107
+
echo ""
108
+
echo "Verifying .well-known/atproto-did..."
109
+
WELLKNOWN_CONTENT=$(curl -s "https://$DOMAIN/.well-known/atproto-did" || echo "ERROR")
110
+
111
+
if [ "$WELLKNOWN_CONTENT" != "$DID" ]; then
112
+
echo "✗ Error: .well-known/atproto-did not accessible or contains wrong DID"
113
+
echo " Expected: $DID"
114
+
echo " Got: $WELLKNOWN_CONTENT"
115
+
exit 1
116
+
fi
117
+
118
+
echo "✓ .well-known/atproto-did verified"
119
+
120
+
# Register with Coves
121
+
echo ""
122
+
echo "Registering with Coves instance..."
123
+
RESPONSE=$(curl -s -X POST "$COVES_URL/xrpc/social.coves.aggregator.register" \
124
+
-H "Content-Type: application/json" \
125
+
-d "{
126
+
\"did\": \"$DID\",
127
+
\"domain\": \"$DOMAIN\"
128
+
}")
129
+
130
+
if echo "$RESPONSE" | jq -e '.error' > /dev/null 2>&1; then
131
+
echo "✗ Registration failed:"
132
+
echo "$RESPONSE" | jq '.'
133
+
exit 1
134
+
fi
135
+
136
+
echo "✓ Registered with Coves"
137
+
138
+
# Create service declaration
139
+
echo ""
140
+
echo "Creating service declaration..."
141
+
SERVICE_RECORD=$(cat <<EOF
142
+
{
143
+
"\$type": "social.coves.aggregator.service",
144
+
"did": "$DID",
145
+
"displayName": "$DISPLAY_NAME",
146
+
"description": "$DESCRIPTION",
147
+
"sourceUrl": "$SOURCE_URL",
148
+
"createdAt": "$(date -u +"%Y-%m-%dT%H:%M:%SZ")"
149
+
}
150
+
EOF
151
+
)
152
+
153
+
RESPONSE=$(curl -s -X POST "$PDS_URL/xrpc/com.atproto.repo.createRecord" \
154
+
-H "Authorization: Bearer $ACCESS_JWT" \
155
+
-H "Content-Type: application/json" \
156
+
-d "{
157
+
\"repo\": \"$DID\",
158
+
\"collection\": \"social.coves.aggregator.service\",
159
+
\"rkey\": \"self\",
160
+
\"record\": $SERVICE_RECORD
161
+
}")
162
+
163
+
if echo "$RESPONSE" | jq -e '.error' > /dev/null 2>&1; then
164
+
echo "✗ Failed to create service declaration:"
165
+
echo "$RESPONSE" | jq '.'
166
+
exit 1
167
+
fi
168
+
169
+
RECORD_URI=$(echo "$RESPONSE" | jq -r '.uri')
170
+
echo "✓ Service declaration created: $RECORD_URI"
171
+
172
+
# Save final configuration
173
+
cat >> kagi-aggregator-config.env <<EOF
174
+
175
+
# Setup completed on $(date)
176
+
AGGREGATOR_DOMAIN="$DOMAIN"
177
+
COVES_INSTANCE_URL="$COVES_URL"
178
+
SERVICE_DECLARATION_URI="$RECORD_URI"
179
+
EOF
180
+
181
+
echo ""
182
+
echo "================================================"
183
+
echo "✓ Kagi Aggregator Setup Complete!"
184
+
echo "================================================"
185
+
echo ""
186
+
echo "Configuration saved to: kagi-aggregator-config.env"
187
+
echo ""
188
+
echo "Your aggregator is now registered and ready to use."
189
+
echo ""
190
+
echo "Next steps:"
191
+
echo "1. Start your aggregator bot: npm start (or appropriate command)"
192
+
echo "2. Community moderators can authorize your aggregator"
193
+
echo "3. Once authorized, your bot can start posting"
194
+
echo ""
195
+
echo "See docs/aggregators/SETUP_GUIDE.md for more information"