A social knowledge tool for researchers built on ATProto
cosmik.network
1# Fly.io Deployment Guide
2
3This guide covers deploying your application with worker processes to Fly.io using Redis for distributed event processing.
4
5## Current Setup Status ✅
6
7Your application is already configured for multi-process deployment:
8
9- **fly.toml**: Configured with `web` and `feed-worker` processes
10- **package.json**: Has the `worker:feeds` script
11- **tsup.config.ts**: Builds both web and worker entry points
12- **Dockerfile**: Builds all artifacts needed for both processes
13
14## Prerequisites
15
16### 1. Redis Setup
17
18Create and attach a Redis database to your app:
19
20```bash
21# Create Redis database
22fly redis create --name annos-redis
23
24# Attach it to your app (automatically sets REDIS_URL environment variable)
25fly redis attach annos-redis
26
27# Verify Redis status
28fly redis status annos-redis
29```
30
31### 2. Verify Dependencies
32
33Ensure worker dependencies are in production dependencies (not devDependencies):
34
35```json
36{
37 "dependencies": {
38 "bullmq": "^5.56.8",
39 "ioredis": "^5.6.1"
40 }
41}
42```
43
44## Deployment Process
45
46### 1. Deploy the Application
47
48```bash
49# Deploy both web and worker processes
50fly deploy
51
52# This creates:
53# - At least one Machine for the 'web' process
54# - At least one Machine for the 'feed-worker' process
55```
56
57### 2. Verify Deployment
58
59```bash
60# Check that both processes are running
61fly status
62
63# Should show something like:
64# Machines
65# PROCESS ID VERSION REGION STATE ROLE CHECKS LAST UPDATED
66# web e2865641be97 v1 sea started ✓ 2m ago
67# feed-worker 1781973f03 v1 sea started ✓ 2m ago
68```
69
70### 3. Monitor Worker Startup
71
72```bash
73# Check worker logs for successful startup
74fly logs --process feed-worker
75
76# Look for these success messages:
77# "Connected to Redis successfully"
78# "Feed worker started and listening for events..."
79```
80
81## Scaling Workers
82
83### Horizontal Scaling (More Machines)
84
85```bash
86# Scale worker processes independently
87fly scale count feed-worker=2
88
89# Scale by region
90fly scale count feed-worker=2 --region sea,ord
91
92# Scale both web and workers
93fly scale count web=3 feed-worker=2
94```
95
96### Vertical Scaling (More Resources)
97
98```bash
99# Scale worker memory
100fly scale memory 1gb --process-group feed-worker
101
102# Scale worker CPU/RAM preset
103fly scale vm performance-2x --process-group feed-worker
104```
105
106## Troubleshooting
107
108### 1. Worker Not Starting
109
110```bash
111# Check worker logs for errors
112fly logs --process feed-worker
113
114# Common issues:
115# - Missing REDIS_URL environment variable
116# - Redis connection failed
117# - Missing dependencies
118```
119
120### 2. Redis Connection Issues
121
122```bash
123# Test Redis connection from worker
124fly ssh console --process feed-worker
125node -e "console.log('Redis URL:', process.env.REDIS_URL)"
126node -e "const Redis = require('ioredis'); const r = new Redis(process.env.REDIS_URL); r.ping().then(console.log)"
127```
128
129### 3. Worker Not Processing Jobs
130
131```bash
132# Check if jobs are being queued
133fly redis connect annos-redis
134> KEYS bull:*
135> LLEN bull:feeds:waiting
136
137# Monitor worker processing
138fly logs --process feed-worker --follow
139```
140
141## Environment Variables
142
143Both web and worker processes share the same environment variables automatically. Key variables:
144
145- `REDIS_URL` - Set automatically by `fly redis attach`
146- `DATABASE_URL` - Your database connection
147- `NODE_ENV` - Set to "dev" in your fly.toml
148
149## Process Communication
150
151- **Web Process**: Publishes events to Redis queues
152- **Worker Process**: Consumes events from Redis queues
153- **Redis**: Acts as the message broker between processes
154
155## Monitoring
156
157### View Process Status
158
159```bash
160# List all machines by process
161fly ps
162
163# Get detailed app status
164fly status
165```
166
167### Monitor Logs
168
169```bash
170# Web process logs
171fly logs --process web
172
173# Worker process logs
174fly logs --process feed-worker
175
176# All processes
177fly logs
178```
179
180### Resource Usage
181
182```bash
183# Check resource metrics
184fly metrics --process feed-worker
185fly metrics --process web
186```
187
188## Development vs Production
189
190### Local Development
191
192For local development with Redis tunnel:
193
194```bash
195# Create tunnel to Redis
196fly redis connect annos-redis
197
198# Use tunnel address in your local environment
199# Example: redis://localhost:10000
200```
201
202### Production
203
204In production, processes automatically use the `REDIS_URL` environment variable set by Fly.io.
205
206## Key Points
207
2081. **No Dockerfile Changes Needed** - Your current Dockerfile works for both processes
2092. **Shared Environment** - All processes share the same environment variables
2103. **Independent Scaling** - Scale web and worker processes separately
2114. **Automatic Deployment** - `fly deploy` handles both process types
2125. **Redis Integration** - Use Fly's native Redis integration for best performance
213
214## Quick Reference Commands
215
216```bash
217# Deploy
218fly deploy
219
220# Check status
221fly status
222fly ps
223
224# Scale workers
225fly scale count feed-worker=2
226
227# Monitor
228fly logs --process feed-worker --follow
229
230# Debug
231fly ssh console --process feed-worker
232fly redis connect annos-redis
233```