dollspace.gay
A third party ATProto appview
sql fixes
dollspacegay.tngl.sh
+633
+196
DEPLOY.md
+196
DEPLOY.md
···
···
1
+
# Deployment Guide - Applying Code Changes to VPS
2
+
3
+
When you make code changes to Python files, you need to rebuild the Docker images on your VPS for the changes to take effect.
4
+
5
+
## Quick Deploy Commands
6
+
7
+
### For Python Firehose/Worker Changes
8
+
9
+
If you modified files in `python-firehose/` directory:
10
+
11
+
```bash
12
+
# SSH into your VPS
13
+
ssh user@your-vps
14
+
15
+
# Navigate to the project directory
16
+
cd /path/to/PublicAppView
17
+
18
+
# Pull latest code from git
19
+
git pull
20
+
21
+
# Rebuild and restart the affected service
22
+
docker-compose build python-backfill-worker
23
+
docker-compose up -d python-backfill-worker
24
+
25
+
# Watch logs to verify the fix
26
+
docker-compose logs -f python-backfill-worker
27
+
```
28
+
29
+
### For App (Node.js) Changes
30
+
31
+
If you modified TypeScript/JavaScript files:
32
+
33
+
```bash
34
+
# Rebuild and restart app service
35
+
docker-compose build app
36
+
docker-compose up -d app
37
+
38
+
# Watch logs
39
+
docker-compose logs -f app
40
+
```
41
+
42
+
### Rebuild Everything
43
+
44
+
If you're not sure which services changed:
45
+
46
+
```bash
47
+
# Pull latest code
48
+
git pull
49
+
50
+
# Rebuild all services
51
+
docker-compose build
52
+
53
+
# Restart all services
54
+
docker-compose up -d
55
+
56
+
# Watch all logs
57
+
docker-compose logs -f
58
+
```
59
+
60
+
## Verifying the SQL Fix
61
+
62
+
After deploying the `unified_worker.py` fix:
63
+
64
+
1. **Rebuild the service:**
65
+
```bash
66
+
docker-compose build python-backfill-worker
67
+
docker-compose up -d python-backfill-worker
68
+
```
69
+
70
+
2. **Watch for errors:**
71
+
```bash
72
+
docker-compose logs -f python-backfill-worker | grep -i "error\|syntax"
73
+
```
74
+
75
+
3. **Verify database logs:**
76
+
```bash
77
+
docker-compose logs -f db | grep -i "syntax error"
78
+
```
79
+
80
+
4. **Success indicators:**
81
+
- No more `$NULL` or `$false` syntax errors
82
+
- Viewer states (likes/reposts) inserting successfully
83
+
- No "current transaction is aborted" errors
84
+
85
+
## Deployment Checklist
86
+
87
+
- [ ] Code changes committed to git
88
+
- [ ] Pushed to remote repository
89
+
- [ ] SSH'd into VPS
90
+
- [ ] Pulled latest code with `git pull`
91
+
- [ ] Rebuilt Docker images with `docker-compose build`
92
+
- [ ] Restarted services with `docker-compose up -d`
93
+
- [ ] Checked logs for errors
94
+
- [ ] Verified application is working
95
+
96
+
## Common Issues
97
+
98
+
### "Image is up to date" but code not updated
99
+
```bash
100
+
# Force rebuild without cache
101
+
docker-compose build --no-cache python-backfill-worker
102
+
docker-compose up -d python-backfill-worker
103
+
```
104
+
105
+
### Container won't stop
106
+
```bash
107
+
# Force stop and remove
108
+
docker-compose stop python-backfill-worker
109
+
docker-compose rm -f python-backfill-worker
110
+
docker-compose up -d python-backfill-worker
111
+
```
112
+
113
+
### Database connection errors after restart
114
+
```bash
115
+
# Wait for database to be healthy
116
+
docker-compose ps
117
+
118
+
# Check database logs
119
+
docker-compose logs db
120
+
121
+
# Restart dependent services
122
+
docker-compose restart python-backfill-worker
123
+
```
124
+
125
+
### Out of disk space
126
+
```bash
127
+
# Clean up old Docker images
128
+
docker system prune -a
129
+
130
+
# Clean up old containers
131
+
docker-compose down --volumes
132
+
docker-compose up -d
133
+
```
134
+
135
+
## Git Workflow
136
+
137
+
### Committing Changes
138
+
139
+
```bash
140
+
# On your local machine (in VSCode)
141
+
git add python-firehose/unified_worker.py
142
+
git commit -m "Fix: Correct SQL syntax in post_viewer_states INSERT"
143
+
git push origin main
144
+
```
145
+
146
+
### Deploying to VPS
147
+
148
+
```bash
149
+
# On VPS
150
+
git pull origin main
151
+
docker-compose build python-backfill-worker
152
+
docker-compose up -d python-backfill-worker
153
+
```
154
+
155
+
## Rollback
156
+
157
+
If something goes wrong:
158
+
159
+
```bash
160
+
# Revert to previous git commit
161
+
git log # Find the commit hash you want to revert to
162
+
git checkout <commit-hash>
163
+
164
+
# Rebuild with old code
165
+
docker-compose build
166
+
docker-compose up -d
167
+
168
+
# Or reset to latest stable
169
+
git checkout main
170
+
git pull
171
+
docker-compose build
172
+
docker-compose up -d
173
+
```
174
+
175
+
## Health Checks
176
+
177
+
Verify services are healthy:
178
+
179
+
```bash
180
+
# Check status
181
+
docker-compose ps
182
+
183
+
# All services should show "healthy" or "running"
184
+
```
185
+
186
+
## Performance Monitoring
187
+
188
+
Watch resource usage:
189
+
190
+
```bash
191
+
# Monitor container stats
192
+
docker stats
193
+
194
+
# Watch specific service
195
+
docker stats python-backfill-worker
196
+
```
+179
DEVELOPMENT.md
+179
DEVELOPMENT.md
···
···
1
+
# Local Development Guide
2
+
3
+
This guide explains how to run AppView locally for development and testing.
4
+
5
+
## Prerequisites
6
+
7
+
- Docker and Docker Compose installed
8
+
- At least 4GB RAM available for Docker
9
+
- Git for version control
10
+
11
+
## Quick Start
12
+
13
+
1. **Copy the local environment file:**
14
+
```bash
15
+
cp .env.local .env
16
+
```
17
+
18
+
2. **Start all services in development mode:**
19
+
```bash
20
+
docker-compose -f docker-compose.yml -f docker-compose.dev.yml up
21
+
```
22
+
23
+
Or run in background:
24
+
```bash
25
+
docker-compose -f docker-compose.yml -f docker-compose.dev.yml up -d
26
+
```
27
+
28
+
3. **View logs:**
29
+
```bash
30
+
docker-compose -f docker-compose.yml -f docker-compose.dev.yml logs -f
31
+
```
32
+
33
+
4. **Access the application:**
34
+
- AppView: http://localhost:5000
35
+
- PostgreSQL: localhost:5432
36
+
- Redis: localhost:6379
37
+
38
+
## Development Features
39
+
40
+
### Hot Reload
41
+
Code changes are automatically reflected in running containers:
42
+
- **Python services**: Changes to `python-firehose/*.py` apply immediately
43
+
- **Node.js app**: Changes to `src/**` require restart (working on hot-reload)
44
+
45
+
### Lower Resource Usage
46
+
Development mode uses much lower resource limits:
47
+
- Database: 2GB RAM (vs 20GB+ in production)
48
+
- Redis: 512MB RAM (vs 8GB in production)
49
+
- Python workers: 512MB-1GB each
50
+
51
+
### Debug Logging
52
+
All services run with `LOG_LEVEL=DEBUG` for detailed output.
53
+
54
+
## Common Commands
55
+
56
+
### Restart a specific service (after code changes)
57
+
```bash
58
+
docker-compose -f docker-compose.yml -f docker-compose.dev.yml restart python-backfill-worker
59
+
```
60
+
61
+
### Rebuild after dependency changes
62
+
```bash
63
+
docker-compose -f docker-compose.yml -f docker-compose.dev.yml build python-backfill-worker
64
+
docker-compose -f docker-compose.yml -f docker-compose.dev.yml up -d python-backfill-worker
65
+
```
66
+
67
+
### Stop all services
68
+
```bash
69
+
docker-compose -f docker-compose.yml -f docker-compose.dev.yml down
70
+
```
71
+
72
+
### Reset database (WARNING: deletes all data)
73
+
```bash
74
+
docker-compose -f docker-compose.yml -f docker-compose.dev.yml down -v
75
+
docker-compose -f docker-compose.yml -f docker-compose.dev.yml up -d
76
+
```
77
+
78
+
### Access database directly
79
+
```bash
80
+
docker-compose exec db psql -U postgres -d atproto
81
+
```
82
+
83
+
### Access Redis CLI
84
+
```bash
85
+
docker-compose exec redis redis-cli
86
+
```
87
+
88
+
## Service Architecture
89
+
90
+
- **db**: PostgreSQL database for storing posts, users, etc.
91
+
- **redis**: In-memory cache and message broker
92
+
- **python-firehose**: Connects to AT Protocol firehose, pushes to Redis
93
+
- **python-worker**: Consumes from Redis, writes to database
94
+
- **python-backfill-worker**: Direct firehose → database (with optional backfill)
95
+
- **app**: Node.js AppView API server
96
+
- **constellation-bridge**: Optional enhanced stats service
97
+
98
+
## Debugging Tips
99
+
100
+
### Watch Python worker logs in real-time
101
+
```bash
102
+
docker-compose -f docker-compose.yml -f docker-compose.dev.yml logs -f python-backfill-worker
103
+
```
104
+
105
+
### Check if services are healthy
106
+
```bash
107
+
docker-compose -f docker-compose.yml -f docker-compose.dev.yml ps
108
+
```
109
+
110
+
### Execute commands in a running container
111
+
```bash
112
+
docker-compose exec python-backfill-worker bash
113
+
```
114
+
115
+
### Check database connection
116
+
```bash
117
+
docker-compose exec python-backfill-worker python -c "import asyncpg; import asyncio; asyncio.run(asyncpg.connect('postgresql://postgres:password@db:5432/atproto'))"
118
+
```
119
+
120
+
## Testing the SQL Fix
121
+
122
+
After fixing the `post_viewer_states` SQL error:
123
+
124
+
1. Restart the affected service:
125
+
```bash
126
+
docker-compose -f docker-compose.yml -f docker-compose.dev.yml restart python-backfill-worker
127
+
```
128
+
129
+
2. Watch for errors:
130
+
```bash
131
+
docker-compose -f docker-compose.yml -f docker-compose.dev.yml logs -f python-backfill-worker | grep -i error
132
+
```
133
+
134
+
3. Should see no more `$NULL` or `$false` syntax errors!
135
+
136
+
## Production vs Development
137
+
138
+
Key differences between `.env.local` and production:
139
+
140
+
| Setting | Local Dev | Production |
141
+
|---------|-----------|------------|
142
+
| Database RAM | 2GB | 20GB+ |
143
+
| Redis RAM | 512MB | 8GB |
144
+
| Worker pool size | 10 | 20 |
145
+
| Logging | DEBUG | INFO |
146
+
| Backfill | Disabled | Optional |
147
+
| Constellation | Disabled | Optional |
148
+
149
+
## Troubleshooting
150
+
151
+
### Port already in use
152
+
If you see "port already in use" errors, check what's using the ports:
153
+
```bash
154
+
netstat -ano | findstr :5432
155
+
netstat -ano | findstr :5000
156
+
```
157
+
158
+
### Out of memory
159
+
Increase Docker's memory limit in Docker Desktop settings (Minimum 4GB recommended).
160
+
161
+
### Database connection refused
162
+
Wait for the database to be healthy:
163
+
```bash
164
+
docker-compose -f docker-compose.yml -f docker-compose.dev.yml logs db
165
+
```
166
+
167
+
## Next Steps
168
+
169
+
1. Make code changes in your editor (VSCode)
170
+
2. Changes to Python files apply immediately (no restart needed)
171
+
3. For dependency changes, rebuild the container
172
+
4. Test your changes locally before deploying to VPS
173
+
174
+
## Useful VSCode Extensions
175
+
176
+
- Docker (by Microsoft) - Manage containers from VSCode
177
+
- PostgreSQL (by Chris Kolkman) - Query database directly
178
+
- Python (by Microsoft) - Python development support
179
+
- GitLens - Enhanced git integration
+92
docker-compose.dev.yml
+92
docker-compose.dev.yml
···
···
1
+
# Local Development Docker Compose Configuration
2
+
# Usage: docker-compose -f docker-compose.yml -f docker-compose.dev.yml up
3
+
#
4
+
# This override file adds:
5
+
# - Volume mounts for hot-reload (code changes apply immediately)
6
+
# - Lower resource limits suitable for local development
7
+
# - Debug logging enabled
8
+
# - Ports exposed for direct access
9
+
10
+
services:
11
+
# Database with lower resource settings for local dev
12
+
db:
13
+
command: postgres -c max_connections=100 -c shared_buffers=1GB -c effective_cache_size=2GB -c work_mem=16MB -c maintenance_work_mem=256MB
14
+
ports:
15
+
- "5432:5432"
16
+
shm_size: 1gb
17
+
deploy:
18
+
resources:
19
+
limits:
20
+
memory: 2G
21
+
22
+
# Redis with lower memory for local dev
23
+
redis:
24
+
command: redis-server --maxmemory 512mb --maxmemory-policy noeviction --appendonly yes --appendfsync everysec
25
+
ports:
26
+
- "6379:6379"
27
+
28
+
# Python Firehose with hot-reload
29
+
python-firehose:
30
+
volumes:
31
+
- ./python-firehose:/app:rw
32
+
environment:
33
+
- LOG_LEVEL=DEBUG
34
+
- REDIS_MAX_STREAM_LEN=10000
35
+
deploy:
36
+
resources:
37
+
limits:
38
+
memory: 512M
39
+
40
+
# Python Worker with hot-reload
41
+
python-worker:
42
+
volumes:
43
+
- ./python-firehose:/app:rw
44
+
environment:
45
+
- LOG_LEVEL=DEBUG
46
+
- DB_POOL_SIZE=10
47
+
- BATCH_SIZE=5
48
+
- PARALLEL_CONSUMERS=2
49
+
deploy:
50
+
resources:
51
+
limits:
52
+
memory: 1G
53
+
54
+
# Python Unified Worker with hot-reload (the one we just fixed!)
55
+
python-backfill-worker:
56
+
volumes:
57
+
- ./python-firehose:/app:rw
58
+
environment:
59
+
- LOG_LEVEL=DEBUG
60
+
- DB_POOL_SIZE=10
61
+
- BACKFILL_DAYS=0
62
+
deploy:
63
+
resources:
64
+
limits:
65
+
memory: 1G
66
+
67
+
# App service with hot-reload
68
+
app:
69
+
volumes:
70
+
- ./appview-signing-key.json:/app/appview-signing-key.json:ro
71
+
- ./appview-private.pem:/app/appview-private.pem:ro
72
+
- ./public/did.json:/app/public/did.json:ro
73
+
- ./oauth-keyset.json:/app/oauth-keyset.json:ro
74
+
- ./src:/app/src:rw
75
+
- ./public:/app/public:rw
76
+
environment:
77
+
- LOG_LEVEL=DEBUG
78
+
- DB_POOL_SIZE=20
79
+
- NODE_ENV=development
80
+
ports:
81
+
- "5000:5000"
82
+
deploy:
83
+
resources:
84
+
limits:
85
+
memory: 2G
86
+
87
+
# Constellation bridge with lower limits
88
+
constellation-bridge:
89
+
deploy:
90
+
resources:
91
+
limits:
92
+
memory: 256M
+2
python-firehose/Dockerfile.unified
+2
python-firehose/Dockerfile.unified
···
21
COPY did_resolver.py .
22
COPY pds_data_fetcher.py .
23
COPY label_service.py .
24
25
# Set environment defaults
26
ENV RELAY_URL=wss://bsky.network
27
ENV DATABASE_URL=postgresql://postgres:password@db:5432/atproto
28
ENV DB_POOL_SIZE=20
29
ENV LOG_LEVEL=INFO
30
31
# Health check
32
HEALTHCHECK --interval=30s --timeout=10s --start-period=40s --retries=3 \
···
21
COPY did_resolver.py .
22
COPY pds_data_fetcher.py .
23
COPY label_service.py .
24
+
COPY verify_code.py .
25
26
# Set environment defaults
27
ENV RELAY_URL=wss://bsky.network
28
ENV DATABASE_URL=postgresql://postgres:password@db:5432/atproto
29
ENV DB_POOL_SIZE=20
30
ENV LOG_LEVEL=INFO
31
+
ENV PYTHONDONTWRITEBYTECODE=1
32
33
# Health check
34
HEALTHCHECK --interval=30s --timeout=10s --start-period=40s --retries=3 \
+104
python-firehose/test_sql_generation.py
+104
python-firehose/test_sql_generation.py
···
···
1
+
#!/usr/bin/env python3
2
+
"""Test script to verify SQL generation for post_viewer_states"""
3
+
4
+
def test_sql_generation():
5
+
"""Test the SQL generation logic"""
6
+
7
+
test_cases = [
8
+
{
9
+
'name': 'Like only',
10
+
'like_uri': 'at://did:plc:test/app.bsky.feed.like/abc123',
11
+
'repost_uri': None,
12
+
'bookmarked': False
13
+
},
14
+
{
15
+
'name': 'Repost only',
16
+
'like_uri': None,
17
+
'repost_uri': 'at://did:plc:test/app.bsky.feed.repost/xyz789',
18
+
'bookmarked': False
19
+
},
20
+
{
21
+
'name': 'Both like and repost',
22
+
'like_uri': 'at://did:plc:test/app.bsky.feed.like/abc123',
23
+
'repost_uri': 'at://did:plc:test/app.bsky.feed.repost/xyz789',
24
+
'bookmarked': False
25
+
},
26
+
{
27
+
'name': 'Nothing set',
28
+
'like_uri': None,
29
+
'repost_uri': None,
30
+
'bookmarked': False
31
+
},
32
+
{
33
+
'name': 'Bookmarked',
34
+
'like_uri': None,
35
+
'repost_uri': None,
36
+
'bookmarked': True
37
+
}
38
+
]
39
+
40
+
for test in test_cases:
41
+
print(f"\n{'='*60}")
42
+
print(f"Test Case: {test['name']}")
43
+
print(f"{'='*60}")
44
+
45
+
like_uri = test['like_uri']
46
+
repost_uri = test['repost_uri']
47
+
bookmarked = test['bookmarked']
48
+
49
+
# Simulate the function logic
50
+
updates = []
51
+
insert_params = ['at://post/uri', 'did:plc:viewer']
52
+
param_idx = 3
53
+
54
+
# Build VALUES clause with proper parameter handling
55
+
like_param = f'${param_idx}' if like_uri else 'NULL'
56
+
if like_uri:
57
+
insert_params.append(like_uri)
58
+
param_idx += 1
59
+
60
+
repost_param = f'${param_idx}' if repost_uri else 'NULL'
61
+
if repost_uri:
62
+
insert_params.append(repost_uri)
63
+
param_idx += 1
64
+
65
+
bookmarked_value = 'true' if bookmarked else 'false'
66
+
67
+
# Build update clauses
68
+
update_idx = 3
69
+
if like_uri:
70
+
updates.append(f'like_uri = ${update_idx}')
71
+
update_idx += 1
72
+
73
+
if repost_uri:
74
+
updates.append(f'repost_uri = ${update_idx}')
75
+
update_idx += 1
76
+
77
+
if bookmarked:
78
+
updates.append('bookmarked = true')
79
+
80
+
# Generate SQL
81
+
sql = f"""
82
+
INSERT INTO post_viewer_states (post_uri, viewer_did, like_uri, repost_uri, bookmarked, thread_muted, reply_disabled, embedding_disabled, pinned)
83
+
VALUES ($1, $2, {like_param}, {repost_param}, {bookmarked_value}, false, false, false, false)
84
+
ON CONFLICT (post_uri, viewer_did) DO UPDATE SET
85
+
{', '.join(updates) if updates else 'like_uri = post_viewer_states.like_uri'}
86
+
"""
87
+
88
+
print(f"\nGenerated SQL:")
89
+
print(sql.strip())
90
+
print(f"\nParameters: {insert_params}")
91
+
print(f"Parameter count: {len(insert_params)}")
92
+
93
+
# Check for errors
94
+
if '$NULL' in sql or '$false' in sql or '$true' in sql:
95
+
print("\n❌ ERROR: Found invalid parameter placeholder!")
96
+
else:
97
+
print("\n✅ SQL looks correct!")
98
+
99
+
if __name__ == '__main__':
100
+
print("Testing SQL Generation for post_viewer_states")
101
+
print("="*60)
102
+
test_sql_generation()
103
+
print("\n" + "="*60)
104
+
print("Test complete!")
+3
python-firehose/unified_worker.py
+3
python-firehose/unified_worker.py
···
783
784
bookmarked_value = 'true' if bookmarked else 'false'
785
786
+
# Debug logging to trace the bug
787
+
logger.debug(f"[VIEWER_STATE_DEBUG] like_param={repr(like_param)}, repost_param={repr(repost_param)}, bookmarked_value={repr(bookmarked_value)}, insert_params_count={len(insert_params)}")
788
+
789
# Build update clauses
790
update_idx = 3
791
update_params = [post_uri, viewer_did]
+57
python-firehose/verify_code.py
+57
python-firehose/verify_code.py
···
···
1
+
#!/usr/bin/env python3
2
+
"""
3
+
Verification script - run this inside the Docker container to check if the fix is applied
4
+
Usage: docker-compose exec python-backfill-worker python verify_code.py
5
+
"""
6
+
7
+
import inspect
8
+
import importlib.util
9
+
10
+
def check_code():
11
+
"""Check if the fixed code is present"""
12
+
print("="*60)
13
+
print("CODE VERIFICATION SCRIPT")
14
+
print("="*60)
15
+
16
+
# Load the module
17
+
spec = importlib.util.spec_from_file_location("unified_worker", "/app/unified_worker.py")
18
+
module = importlib.util.module_from_spec(spec)
19
+
20
+
try:
21
+
spec.loader.exec_module(module)
22
+
print("\n✓ Module loaded successfully")
23
+
except Exception as e:
24
+
print(f"\n✗ Failed to load module: {e}")
25
+
return
26
+
27
+
# Get the source code of create_post_viewer_state
28
+
try:
29
+
source = inspect.getsource(module.UnifiedWorker.create_post_viewer_state)
30
+
print("\n" + "="*60)
31
+
print("SOURCE CODE OF create_post_viewer_state:")
32
+
print("="*60)
33
+
print(source[:1000]) # Print first 1000 chars
34
+
35
+
# Check for the problematic patterns
36
+
if "$NULL" in source or "$false" in source or ".replace" in source:
37
+
print("\n❌ OLD CODE DETECTED!")
38
+
print("The container is running the OLD buggy code.")
39
+
print("Found problematic patterns:")
40
+
if "$NULL" in source:
41
+
print(" - Found '$NULL'")
42
+
if "$false" in source:
43
+
print(" - Found '$false'")
44
+
if ".replace" in source:
45
+
print(" - Found '.replace'")
46
+
elif "{like_param}" in source and "{repost_param}" in source:
47
+
print("\n✅ NEW CODE DETECTED!")
48
+
print("The container is running the FIXED code.")
49
+
else:
50
+
print("\n⚠ UNKNOWN CODE VERSION")
51
+
print("Cannot determine if this is the old or new code.")
52
+
53
+
except Exception as e:
54
+
print(f"\n✗ Failed to get source: {e}")
55
+
56
+
if __name__ == '__main__':
57
+
check_code()