Scripts Architecture#

✅ Final Organization (November 2024)#

After investigating script placement and dependency resolution, we arrived at this clean architecture:

Location: scripts/ (workspace root)#

Why root level?

• Logical separation: deployment code in system/, operational tools in scripts/
• Clear organization: not buried in implementation directories
• Consistent with existing at/scripts/ pattern

Dependency Strategy: Shared Dependencies#

Problem: Scripts need packages like @atproto/api, mongodb, etc. that are installed in system/node_modules/

Solution: Install same packages at root level

• Added to root package.json dependencies
• Small duplication (fine architecturally)
• Node.js walks up from scripts/atproto/ → scripts/ → root → finds node_modules/
• Same packages in system/ for Netlify functions (they need bundled deps)

Dependencies shared:

{
  "@atproto/api": "^0.17.2",
  "@aws-sdk/client-s3": "^3.896.0", 
  "mongodb": "^6.20.0",
  "dotenv": "^16.4.7",
  "node-fetch": "^3.3.2"
}

Import Paths#

Scripts use relative paths to access backend modules:

// From scripts/atproto/*.mjs
import { connect } from "../../system/backend/database.mjs";
import { createMediaRecord } from "../../system/backend/media-atproto.mjs";

// From scripts/recovery/*.mjs  
import { connect } from "../../system/backend/database.mjs";
import { userIDFromHandle } from "../../system/backend/authorization.mjs";

Path breakdown: ../../system/backend/

• ../ - up from scripts/atproto/ to scripts/
• ../ - up from scripts/ to workspace root
• system/backend/ - into backend module directory

Environment Variables#

Scripts load .env from system/.env:

import { config } from 'dotenv';
config({ path: './system/.env' });  // Relative to workspace root

Must run from workspace root so ./system/.env resolves correctly.

🏗️ Directory Structure#

/workspaces/aesthetic-computer/
├── scripts/                    # Operational scripts (NEW!)
│   ├── README.md              # Omakase menu for LLM
│   ├── SETUP.md               # Setup instructions
│   ├── ARCHITECTURE.md        # This file
│   ├── atproto/               # ATProto sync/backfill tools
│   │   ├── sync-atproto.mjs
│   │   ├── backfill-kidlisp-rkeys.mjs
│   │   └── ...
│   └── recovery/              # Data recovery tools
│       ├── recover-deleted-tapes.mjs
│       └── check-tape-zips-in-spaces.mjs
│
├── system/                     # Deployment code
│   ├── backend/               # Reusable modules
│   │   ├── database.mjs
│   │   ├── media-atproto.mjs
│   │   ├── authorization.mjs
│   │   └── ...
│   ├── netlify/
│   │   └── functions/         # Netlify edge functions
│   ├── .env                   # Secrets (in .gitignore)
│   ├── node_modules/          # System dependencies
│   └── package.json
│
├── at/scripts/atproto/        # External AT Protocol tools
│   └── ...                    # Federation, lexicon publishing, etc.
│
├── node_modules/              # Root dependencies (includes script deps)
└── package.json               # Root packages (includes script deps)

🎯 Design Principles#

1. Separation of Concerns

   • system/ = deployment infrastructure (Netlify functions, edge compute)
   • scripts/ = operational tools (maintenance, sync, recovery)
   • at/scripts/ = external protocol tools (federation with other PDS)

2. Shared Dependencies

   • Accept small duplication for cleaner architecture
   • Root level for scripts, system level for functions
   • Both can use same versions (kept in sync)

3. Backend Modules as Library

   • system/backend/*.mjs = reusable utilities
   • Used by: Netlify functions, operational scripts, AT tools
   • Never create duplicate implementations

4. LLM-Friendly Documentation

   • README.md = "Omakase menu" of available scripts
   • SETUP.md = Getting started, troubleshooting
   • ARCHITECTURE.md = Design decisions (this file)

🚀 Usage Pattern#

# 1. Ensure dependencies installed
npm install  # (at root)

# 2. Ensure secrets available
ls system/.env  # or copy from vault

# 3. Run scripts from root
node scripts/atproto/sync-atproto.mjs live --kidlisp-only
node scripts/recovery/check-tape-zips-in-spaces.mjs @jeffrey

🔄 Alternatives Considered#

❌ Keep in system/backend/#

• Pros: Natural access to system/node_modules
• Cons: Mixed with library code, unclear which are operations vs utilities

❌ Move to system/scripts/#

• Pros: Still in system where deps are
• Cons: Scripts aren't really "system" deployment code

❌ Use NODE_PATH hack#

• Pros: Scripts could stay anywhere
• Cons: Hacky, requires wrapper scripts, confusing

✅ Root scripts/ + shared deps (CHOSEN)#

• Pros: Clean separation, logical organization, minimal duplication
• Cons: Some deps duplicated (acceptable tradeoff)

📝 Commit Strategy#

Group related changes:

1. Script organization (moves, README, SETUP)
2. Dependency sharing (package.json updates)
3. Netlify config fixes (@atproto/api externals)

Message:

Organize operational scripts with shared dependencies

- Move provisional scripts from system/backend/ to scripts/
- Add atproto/ (sync/backfill) and recovery/ subdirectories  
- Share dependencies: install @atproto/api, mongodb, etc. at root
- Fix netlify.toml: add @atproto/api to 4 function externals
- Add documentation: README (omakase menu), SETUP, ARCHITECTURE

All kidlisp/tapes/moods synced to ATProto successfully