Blob Storage Configuration Guide#

This guide covers configuring blob storage for your PDS to store images, videos, and other media files.

Overview#

PDS stores two types of data:

Structured Data (SQLite): Posts, profiles, relationships
Blobs (Object Storage): Images, videos, avatars, banners

This document focuses on blob storage configuration.

Storage Options Comparison#

DigitalOcean Spaces ✅ Recommended#

Pros:

S3-compatible API (easy integration)
Simple pricing: $5/month for 250GB storage + bandwidth
Includes CDN at no extra cost
Easy to set up from DO dashboard
Good performance
Scales to petabytes

Cons:

Limited to DO regions
Less features than AWS S3

Best For:

Small to medium deployments (1-100 users)
Cost-conscious hosting
Simple setup requirements

Pricing:

Base: $5/month (250GB storage + 1TB bandwidth)
Additional storage: $0.02/GB/month
Additional bandwidth: $0.01/GB (after 1TB)

Google Cloud Storage#

Pros:

Highly scalable
Advanced features (lifecycle, versioning, analytics)
Global network
Integrated with GCP services
Better for large scale

Cons:

More complex pricing
Requires GCP account setup
More configuration needed
Higher cost at small scale

Best For:

Large deployments (100+ users)
Need advanced features
Already using GCP

Pricing:

Standard storage: $0.020/GB/month
Network egress: $0.12/GB (NA/EU)
Operations: $0.05 per 10,000 writes

Local Disk (Development Only)#

Pros:

Simple setup
No external dependencies
No extra cost

Cons:

No redundancy
No CDN
Limited by disk size
Not suitable for production
Difficult to scale

Best For:

Development and testing only
Never for production

DigitalOcean Spaces Setup#

1. Create a Space#

Via DigitalOcean Dashboard:

Go to Spaces: https://cloud.digitalocean.com/spaces
Click "Create a Space"
Choose region (closest to users):
- nyc3 - New York (US East)
- sfo3 - San Francisco (US West)
- fra1 - Frankfurt (Europe)
- sgp1 - Singapore (Asia)
Name: aesthetic-pds-blobs
Enable CDN: ✅ Yes (included in price)
File Listing: 🚫 Restricted (important for privacy)
Click "Create Space"

Via CLI (doctl):

# Install doctl first
brew install doctl  # or appropriate package manager

# Authenticate
doctl auth init

# Create space (CDN enabled by default)
doctl spaces create aesthetic-pds-blobs --region nyc3

2. Generate Access Keys#

Via Dashboard:

Go to API → Spaces Keys
Click "Generate New Key"
Name: aesthetic-pds-production
Save the key and secret (shown only once!)

Store in vault:

# Add to aesthetic-computer-vault/at/.env
echo "SPACES_KEY=DO00XXXXXXXXXX" >> aesthetic-computer-vault/at/.env
echo "SPACES_SECRET=xxxxxxxxxxxxxxxxxxxx" >> aesthetic-computer-vault/at/.env

3. Configure PDS#

Add to /pds/pds.env on your server:

# Blob Storage - DigitalOcean Spaces
PDS_BLOBSTORE_DISK_LOCATION=s3

# S3-compatible endpoint
PDS_BLOBSTORE_S3_ENDPOINT=https://nyc3.digitaloceanspaces.com

# Space name (bucket)
PDS_BLOBSTORE_S3_BUCKET=aesthetic-pds-blobs

# Access credentials
PDS_BLOBSTORE_S3_ACCESS_KEY_ID=DO00XXXXXXXXXX
PDS_BLOBSTORE_S3_SECRET_ACCESS_KEY=xxxxxxxxxxxxxxxxxxxx

# Region
PDS_BLOBSTORE_S3_REGION=nyc3

# Force path style (required for Spaces)
PDS_BLOBSTORE_S3_FORCE_PATH_STYLE=true

4. Set CORS Policy (If Needed)#

For direct browser uploads:

[
  {
    "AllowedOrigins": ["https://pds.aesthetic.computer"],
    "AllowedMethods": ["PUT", "POST", "GET", "HEAD"],
    "AllowedHeaders": ["*"],
    "MaxAgeSeconds": 3600
  }
]

Apply via doctl:

# Save CORS policy to file
cat > cors.json << 'EOF'
[
  {
    "AllowedOrigins": ["https://pds.aesthetic.computer"],
    "AllowedMethods": ["PUT", "POST", "GET", "HEAD"],
    "AllowedHeaders": ["*"],
    "MaxAgeSeconds": 3600
  }
]
EOF

# Apply to space
doctl spaces cors set aesthetic-pds-blobs --config cors.json

5. Enable CDN (Included)#

Spaces CDN is automatically enabled. Verify:

# Get Space info
doctl spaces list

# CDN endpoint will be shown
# Format: aesthetic-pds-blobs.nyc3.cdn.digitaloceanspaces.com

6. Test Upload#

# Install s3cmd
apt install s3cmd

# Configure
cat > ~/.s3cfg << EOF
[default]
access_key = DO00XXXXXXXXXX
secret_key = xxxxxxxxxxxxxxxxxxxx
host_base = nyc3.digitaloceanspaces.com
host_bucket = %(bucket)s.nyc3.digitaloceanspaces.com
use_https = True
EOF

# Test upload
echo "test" > test.txt
s3cmd put test.txt s3://aesthetic-pds-blobs/test.txt

# Test download
s3cmd get s3://aesthetic-pds-blobs/test.txt

# Delete test file
s3cmd del s3://aesthetic-pds-blobs/test.txt

Google Cloud Storage Setup#

1. Create GCS Bucket#

# Install gcloud CLI
curl https://sdk.cloud.google.com | bash

# Initialize and authenticate
gcloud init

# Create bucket
gcloud storage buckets create gs://aesthetic-pds-blobs \
  --location=us-east1 \
  --uniform-bucket-level-access \
  --public-access-prevention

# Enable versioning (optional)
gcloud storage buckets update gs://aesthetic-pds-blobs \
  --versioning

2. Create Service Account#

# Create service account
gcloud iam service-accounts create pds-blob-storage \
  --display-name="PDS Blob Storage"

# Grant permissions
gcloud storage buckets add-iam-policy-binding gs://aesthetic-pds-blobs \
  --member="serviceAccount:pds-blob-storage@aesthetic-computer.iam.gserviceaccount.com" \
  --role="roles/storage.objectAdmin"

# Create key
gcloud iam service-accounts keys create pds-gcs-key.json \
  --iam-account=pds-blob-storage@aesthetic-computer.iam.gserviceaccount.com

3. Configure PDS#

# In /pds/pds.env
PDS_BLOBSTORE_DISK_LOCATION=gcs
PDS_BLOBSTORE_GCS_BUCKET=aesthetic-pds-blobs
PDS_BLOBSTORE_GCS_PROJECT_ID=aesthetic-computer
GOOGLE_APPLICATION_CREDENTIALS=/pds/gcs-service-account.json

Copy the service account key to server:

scp pds-gcs-key.json root@pds.aesthetic.computer:/pds/gcs-service-account.json

4. Set CORS Policy#

# Create CORS config
cat > cors.json << 'EOF'
[
  {
    "origin": ["https://pds.aesthetic.computer"],
    "method": ["GET", "HEAD", "PUT", "POST"],
    "responseHeader": ["Content-Type"],
    "maxAgeSeconds": 3600
  }
]
EOF

# Apply CORS
gcloud storage buckets update gs://aesthetic-pds-blobs \
  --cors-file=cors.json

5. Enable CDN (Optional)#

# Create backend bucket
gcloud compute backend-buckets create aesthetic-pds-blobs-backend \
  --gcs-bucket-name=aesthetic-pds-blobs \
  --enable-cdn

# Set cache TTL
gcloud compute backend-buckets update aesthetic-pds-blobs-backend \
  --cache-mode=CACHE_ALL_STATIC \
  --default-ttl=3600

Storage Management Scripts#

Monitor Storage Usage#

For DigitalOcean Spaces:

#!/bin/bash
# scripts/check-storage-usage.sh

SPACE_NAME="aesthetic-pds-blobs"

# Get total size
SIZE=$(s3cmd du s3://$SPACE_NAME | awk '{print $1}')
SIZE_GB=$(echo "scale=2; $SIZE/1024/1024/1024" | bc)

echo "Space: $SPACE_NAME"
echo "Size: ${SIZE_GB}GB"
echo "Cost: \$$(echo "scale=2; if($SIZE_GB > 250) ($SIZE_GB - 250) * 0.02 + 5 else 5" | bc)/month"

# Get object count
COUNT=$(s3cmd ls -r s3://$SPACE_NAME | wc -l)
echo "Objects: $COUNT"

Backup Blobs#

#!/bin/bash
# scripts/backup-blobs.sh

SOURCE="s3://aesthetic-pds-blobs"
DEST="s3://aesthetic-pds-backups/blobs-$(date +%Y%m%d)"

# Sync to backup bucket
s3cmd sync $SOURCE $DEST

echo "Backed up blobs to $DEST"

Clean Old Test Files#

#!/bin/bash
# scripts/clean-test-blobs.sh

# Delete files older than 30 days with "test" in name
s3cmd ls -r s3://aesthetic-pds-blobs | \
  grep "test" | \
  awk '{if (NF==4) print $4}' | \
  while read file; do
    s3cmd del "$file"
  done

Lifecycle Policies#

DigitalOcean Spaces#

Spaces supports lifecycle rules for automatic cleanup:

<!-- lifecycle.xml -->
<LifecycleConfiguration>
  <Rule>
    <ID>delete-old-drafts</ID>
    <Status>Enabled</Status>
    <Prefix>drafts/</Prefix>
    <Expiration>
      <Days>30</Days>
    </Expiration>
  </Rule>
  <Rule>
    <ID>transition-old-videos</ID>
    <Status>Enabled</Status>
    <Prefix>videos/</Prefix>
    <Expiration>
      <Days>365</Days>
    </Expiration>
  </Rule>
</LifecycleConfiguration>

Apply:

s3cmd setlifecycle lifecycle.xml s3://aesthetic-pds-blobs

Google Cloud Storage#

# Create lifecycle config
cat > lifecycle.json << 'EOF'
{
  "lifecycle": {
    "rule": [
      {
        "action": {"type": "Delete"},
        "condition": {
          "age": 365,
          "matchesPrefix": ["videos/"]
        }
      }
    ]
  }
}
EOF

# Apply lifecycle
gcloud storage buckets update gs://aesthetic-pds-blobs \
  --lifecycle-file=lifecycle.json

Security Best Practices#

1. Restrict Access#

DO Spaces:

Set "File Listing" to "Restricted"
Use signed URLs for private content
Rotate access keys annually

GCS:

Use uniform bucket-level access
Enable public access prevention
Use service accounts, not user accounts

2. Encrypt at Rest#

Both DO Spaces and GCS encrypt at rest by default:

✅ DO Spaces: AES-256 encryption (automatic)
✅ GCS: AES-256 encryption (automatic)

For additional security:

# Client-side encryption
PDS_BLOBSTORE_ENCRYPTION_KEY=<32-byte-key>

3. Access Logging#

Enable access logs:

# DO Spaces
s3cmd logging s3://aesthetic-pds-blobs s3://aesthetic-pds-logs

# GCS
gcloud storage buckets update gs://aesthetic-pds-blobs \
  --log-bucket=aesthetic-pds-logs \
  --log-object-prefix=access-logs/

4. Versioning#

Protect against accidental deletion:

# DO Spaces
s3cmd setversioning enable s3://aesthetic-pds-blobs

# GCS
gcloud storage buckets update gs://aesthetic-pds-blobs \
  --versioning

Cost Optimization#

1. Compress Images#

# In PDS config
PDS_BLOBSTORE_COMPRESSION_ENABLED=true
PDS_BLOBSTORE_MAX_SIZE_BYTES=5242880  # 5MB

2. Use CDN#

Reduces origin requests
Lowers bandwidth costs
Improves performance

3. Set Cache Headers#

# Cache static assets
PDS_BLOBSTORE_CACHE_CONTROL="public, max-age=31536000"

4. Monitor Usage#

Set up alerts:

# Alert if storage > 200GB
if [ $SIZE_GB -gt 200 ]; then
  echo "Storage exceeds 200GB: ${SIZE_GB}GB" | mail -s "PDS Storage Alert" admin@aesthetic.computer
fi

Troubleshooting#

Connection Errors#

# Test connectivity
curl -I https://nyc3.digitaloceanspaces.com

# Test authentication
s3cmd ls s3://aesthetic-pds-blobs

Upload Failures#

Check PDS logs:

docker logs pds | grep -i blob

Common issues:

Wrong credentials
Bucket doesn't exist
CORS misconfigured
Network blocked

Slow Performance#

Check CDN is enabled
Verify region is close to users
Monitor bandwidth usage
Check server CPU/memory

Migration Between Providers#

Spaces to GCS#

# Install rclone
curl https://rclone.org/install.sh | bash

# Configure both remotes
rclone config

# Sync data
rclone sync spaces:aesthetic-pds-blobs gcs:aesthetic-pds-blobs --progress

Verify Migration#

# Check file counts
rclone size spaces:aesthetic-pds-blobs
rclone size gcs:aesthetic-pds-blobs

# Compare checksums
rclone check spaces:aesthetic-pds-blobs gcs:aesthetic-pds-blobs

Next Steps#

Return to README.md for overview
Review DEPLOYMENT.md for full deployment
Review MAINTENANCE.md for ongoing management

Last Updated: October 9, 2025 Recommended: DigitalOcean Spaces for initial deployment