Import your Last.fm and Spotify listening history to the AT Protocol network using the fm.teal.alpha.feed.play lexicon.
0
fork

Configure Feed

Select the types of activity you want to include in your feed.

feat(cli): add combined import mode, structured logging, and v0.4.0 CLI overhaul

- introduce combined Last.fm + Spotify import with deduplication
- add sync and deduplicate modes
- replace legacy flags with unified --mode interface (backwards compatible)
- add structured, colour-coded logging with verbose/quiet levels
- improve batching, rate limiting, and graceful shutdown
- update docs, CLI help, and bump version to v0.4.0

+1216 -374
+197 -34
README.md
··· 6 6 7 7 ## Features 8 8 9 + - ✅ **Structured Logging**: Color-coded output with debug/verbose modes 9 10 - ✅ **Batch Operations**: Uses `com.atproto.repo.applyWrites` for efficient batch publishing (up to 200 records per call) 10 11 - ✅ **Spotify Support**: Import from Spotify Extended Streaming History (JSON format) 12 + - ✅ **Combined Import**: Merge Last.fm and Spotify exports, automatically deduplicating overlapping plays 11 13 - ✅ **TID-based Record Keys**: Records use timestamp-based identifiers for chronological ordering 12 14 - ✅ **Re-Sync Mode**: Check existing Teal records and only import new scrobbles (no duplicates!) 13 15 - ✅ **Rate Limiting**: Automatically limits imports to 1K records per day to prevent rate limiting your entire PDS ··· 44 46 45 47 ## Usage 46 48 47 - ### Re-Sync Mode (NEW!) 49 + ### Combined Import Mode 50 + 51 + Merge your Last.fm and Spotify listening history into a single, deduplicated import: 52 + 53 + ```bash 54 + # Preview the merged import 55 + npm start -- -i lastfm.csv --spotify-input spotify-export/ -m combined --dry-run 56 + 57 + # Perform the combined import 58 + npm start -- -i lastfm.csv --spotify-input spotify-export/ -m combined -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -y 59 + ``` 60 + 61 + Combined mode will: 62 + 1. Parse both Last.fm CSV and Spotify JSON exports 63 + 2. Normalize track names and artist names for comparison 64 + 3. Identify duplicate plays (same track within 5 minutes) 65 + 4. Choose the best version of each play: 66 + - Prefers Last.fm records with MusicBrainz IDs 67 + - Otherwise prefers Spotify for better metadata quality 68 + 5. Merge into a single chronological timeline 69 + 6. Show detailed statistics about the merge 70 + 71 + This is perfect for: 72 + - Getting complete listening history from both services 73 + - Filling gaps where one service was used more than the other 74 + - Ensuring the best metadata quality for each play 75 + - Avoiding duplicate entries when both services tracked the same play 76 + 77 + **Example Output:** 78 + ``` 79 + 📊 Merge Statistics 80 + ═══════════════════════════════════════════ 81 + Last.fm records: 15,234 82 + Spotify records: 8,567 83 + Total before merge: 23,801 84 + 85 + Duplicates removed: 3,421 86 + Last.fm unique: 11,813 87 + Spotify unique: 5,146 88 + 89 + Final merged total: 16,959 90 + 91 + Date range: 92 + First: 2015-03-15 10:23:45 93 + Last: 2025-01-07 14:32:11 94 + ═══════════════════════════════════════════ 95 + ``` 96 + 97 + ### Re-Sync Mode 48 98 49 99 If you've already imported scrobbles before and want to sync your Last.fm export with Teal without creating duplicates: 50 100 51 101 ```bash 52 102 # Preview what will be synced 53 - npm start -- -f lastfm.csv -i alice.bsky.social -p xxxx-xxxx-xxxx-xxxx --sync --dry-run 103 + npm start -- -i lastfm.csv -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -m sync --dry-run 54 104 55 105 # Perform the sync 56 - npm start -- -f lastfm.csv -i alice.bsky.social -p xxxx-xxxx-xxxx-xxxx --sync -y 106 + npm start -- -i lastfm.csv -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -m sync -y 57 107 ``` 58 108 59 109 Sync mode will: ··· 70 120 71 121 **Note:** Sync mode requires authentication even in dry-run mode to fetch existing records. 72 122 123 + ### Remove Duplicates Mode 124 + 125 + If you accidentally imported duplicate records, you can clean them up: 126 + 127 + ```bash 128 + # Preview duplicates (dry run) 129 + npm start -- -m deduplicate -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx --dry-run 130 + 131 + # Remove duplicates (keeps first occurrence) 132 + npm start -- -m deduplicate -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx 133 + ``` 134 + 135 + This will: 136 + 1. Fetch all existing records from Teal 137 + 2. Identify duplicate plays (same track, artist, and timestamp) 138 + 3. Keep the first occurrence of each duplicate 139 + 4. Delete the rest 140 + 73 141 ### Interactive Mode 74 142 75 143 The simplest way to use the importer - just run it and follow the prompts: ··· 84 152 85 153 ```bash 86 154 # Full automation (Last.fm) 87 - npm start -- -f lastfm.csv -i alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -y 155 + npm start -- -i lastfm.csv -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -y 88 156 89 157 # Import from Spotify (single file) 90 - npm start -- -f Streaming_History_Audio_2021-2023_0.json --spotify -i alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -y 158 + npm start -- -i Streaming_History_Audio_2021-2023_0.json -m spotify -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -y 91 159 92 160 # Import from Spotify (directory with multiple files - recommended) 93 - npm start -- -f '/path/to/Spotify Extended Streaming History' --spotify -i alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -y 161 + npm start -- -i '/path/to/Spotify Extended Streaming History' -m spotify -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -y 162 + 163 + # Combined import (merge Last.fm and Spotify) 164 + npm start -- -i lastfm.csv --spotify-input '/path/to/Spotify Extended Streaming History' -m combined -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -y 94 165 95 166 # Preview without publishing 96 - npm start -- -f lastfm.csv --dry-run 167 + npm start -- -i lastfm.csv --dry-run 97 168 98 - # Preview Spotify import 99 - npm start -- -f '/path/to/Spotify Extended Streaming History' --spotify --dry-run 169 + # Preview with verbose debug output 170 + npm start -- -i lastfm.csv --dry-run -v 171 + 172 + # Quiet mode (only warnings and errors) 173 + npm start -- -i lastfm.csv -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -q -y 100 174 101 175 # Custom batch settings (advanced users) 102 - npm start -- -f lastfm.csv -i alice.bsky.social -b 20 -d 3000 176 + npm start -- -i lastfm.csv -h alice.bsky.social -b 20 -d 3000 103 177 104 178 # Process newest tracks first 105 - npm start -- -f lastfm.csv -i alice.bsky.social -r -y 179 + npm start -- -i lastfm.csv -h alice.bsky.social -r -y 106 180 ``` 107 181 108 182 ## Command Line Options 109 183 184 + ### Authentication 185 + | Option | Short | Description | 186 + |--------|-------|-------------| 187 + | `--handle <handle>` | `-h` | ATProto handle or DID (e.g., alice.bsky.social) | 188 + | `--password <pass>` | `-p` | ATProto app password | 189 + 190 + ### Input 191 + | Option | Short | Description | 192 + |--------|-------|-------------| 193 + | `--input <path>` | `-i` | Path to Last.fm CSV or Spotify JSON file/directory | 194 + | `--spotify-input <path>` | | Path to Spotify export (for combined mode) | 195 + 196 + ### Mode 110 197 | Option | Short | Description | Default | 111 198 |--------|-------|-------------|---------| 112 - | `--help` | `-h` | Show help message | - | 113 - | `--file <path>` | `-f` | Path to Last.fm CSV or Spotify JSON file/directory | (prompted) | 114 - | `--spotify` | | Import from Spotify JSON export instead of Last.fm | false | 115 - | `--identifier <id>` | `-i` | ATProto handle or DID | (prompted) | 116 - | `--password <pass>` | `-p` | ATProto app password | (prompted) | 199 + | `--mode <mode>` | `-m` | Import mode | `lastfm` | 200 + 201 + **Available modes:** 202 + - `lastfm` - Import Last.fm export only 203 + - `spotify` - Import Spotify export only 204 + - `combined` - Merge Last.fm + Spotify exports 205 + - `sync` - Skip existing records (sync mode) 206 + - `deduplicate` - Remove duplicate records 207 + 208 + ### Batch Configuration 209 + | Option | Short | Description | Default | 210 + |--------|-------|-------------|---------| 117 211 | `--batch-size <num>` | `-b` | Records per batch | Auto-calculated | 118 - | `--batch-delay <ms>` | `-d` | Delay between batches in ms | 2000 (min: 1000) | 119 - | `--yes` | `-y` | Skip confirmation prompt | false | 120 - | `--dry-run` | `-n` | Preview without publishing | false | 121 - | `--reverse-chronological` | `-r` | Process newest first | false (oldest first) | 122 - | `--sync` | `-s` | Re-sync mode: only import new records | false | 212 + | `--batch-delay <ms>` | `-d` | Delay between batches in ms | 500 (min: 500) | 213 + 214 + ### Import Options 215 + | Option | Short | Description | Default | 216 + |--------|-------|-------------|---------| 217 + | `--reverse` | `-r` | Process newest first | false (oldest first) | 218 + | `--yes` | `-y` | Skip confirmation prompts | false | 219 + | `--dry-run` | | Preview without importing | false | 220 + 221 + ### Output 222 + | Option | Short | Description | Default | 223 + |--------|-------|-------------|---------| 224 + | `--verbose` | `-v` | Enable debug logging | false | 225 + | `--quiet` | `-q` | Suppress non-essential output | false | 226 + | `--help` | | Show help message | - | 227 + 228 + ### Legacy Flags (Backwards Compatible) 229 + 230 + For backwards compatibility, the following old flags still work: 231 + - `--file` → Use `--input` instead 232 + - `--identifier` → Use `--handle` instead 233 + - `--spotify-file` → Use `--spotify-input` instead 234 + - `--reverse-chronological` → Use `--reverse` instead 235 + - `--spotify` → Use `--mode spotify` instead 236 + - `--combined` → Use `--mode combined` instead 237 + - `--sync` → Use `--mode sync` instead 238 + - `--remove-duplicates` → Use `--mode deduplicate` instead 123 239 124 240 ### Batch Settings 125 241 126 242 The importer automatically calculates optimal batch settings based on your total record count and rate limits. You generally **don't need** to specify batch settings unless you have specific requirements. 127 243 128 244 **Automatic behavior:** 129 - - For imports < 1K records: Uses default settings (50 records/batch, 2s delay) 245 + - For imports < 1K records: Uses default settings (200 records/batch, 500ms delay) 130 246 - For imports > 1K records: Automatically calculates settings to spread across multiple days 131 247 132 248 **Manual override** (advanced): 133 249 - `--batch-size`: Number of records processed per batch (1-200, PDS maximum) 134 - - `--batch-delay`: Milliseconds to wait between batches (min: 1000) 250 + - `--batch-delay`: Milliseconds to wait between batches (min: 500) 135 251 136 252 ⚠️ Lower delays increase speed but risk hitting rate limits. The automatic calculation is recommended. 137 253 254 + ## Logging and Output 255 + 256 + The importer includes a structured logging system with color-coded output: 257 + 258 + - **Green (✓)**: Success messages 259 + - **Cyan (→)**: Progress updates 260 + - **Yellow (⚠️)**: Warnings 261 + - **Red (✗)**: Errors 262 + - **Bold Red (🛑)**: Fatal errors 263 + 264 + ### Verbosity Levels 265 + 266 + **Default Mode**: Shows standard operational messages 267 + ```bash 268 + npm start -- -i lastfm.csv -h alice.bsky.social -p pass 269 + ``` 270 + 271 + **Verbose Mode** (`-v`): Shows detailed debug information including batch timing, API calls, etc. 272 + ```bash 273 + npm start -- -i lastfm.csv -h alice.bsky.social -p pass -v 274 + ``` 275 + 276 + **Quiet Mode** (`-q`): Only shows warnings and errors 277 + ```bash 278 + npm start -- -i lastfm.csv -h alice.bsky.social -p pass -q 279 + ``` 280 + 138 281 ## Getting Your Data 139 282 140 283 ### Last.fm Export ··· 167 310 - **trackName**: The name of the track 168 311 - **artists**: Array of artist objects (requires `artistName`, optional `artistMbId` for Last.fm) 169 312 - **playedTime**: ISO 8601 timestamp of when you listened 170 - - **submissionClientAgent**: Identifies this importer (`lastfm-importer/v0.3.0`) 313 + - **submissionClientAgent**: Identifies this importer (`lastfm-importer/v0.4.0`) 171 314 - **musicServiceBaseDomain**: Set to `last.fm` or `spotify.com` depending on source 172 315 173 316 ### Optional Fields (when available) ··· 194 337 "recordingMbId": "3a390ad3-fe56-45f2-a073-bebc45d6bde1", 195 338 "playedTime": "2025-11-13T23:49:36Z", 196 339 "originUrl": "https://www.last.fm/music/Cjbeards/_/Paint+My+Masterpiece", 197 - "submissionClientAgent": "lastfm-importer/v0.3.0", 340 + "submissionClientAgent": "lastfm-importer/v0.4.0", 198 341 "musicServiceBaseDomain": "last.fm" 199 342 } 200 343 ``` ··· 212 355 "releaseName": "Twenty", 213 356 "playedTime": "2021-09-09T10:34:08Z", 214 357 "originUrl": "https://open.spotify.com/track/3gZqDJkMZipOYCRjlHWgOV", 215 - "submissionClientAgent": "lastfm-importer/v0.3.0", 358 + "submissionClientAgent": "lastfm-importer/v0.4.0", 216 359 "musicServiceBaseDomain": "spotify.com" 217 360 } 218 361 ``` ··· 221 364 222 365 By default, records are processed **oldest first** (chronological order). This means your earliest scrobbles will appear first in your ATProto feed. 223 366 224 - Use the `--reverse-chronological` or `-r` flag to process **newest first** instead. 367 + Use the `--reverse` or `-r` flag to process **newest first** instead. 225 368 226 369 ## Multi-Day Imports 227 370 ··· 254 397 Preview what will be imported without actually publishing: 255 398 256 399 ```bash 257 - npm start -- -f lastfm.csv --dry-run 400 + npm start -- -i lastfm.csv --dry-run 258 401 ``` 259 402 260 403 Dry run shows: ··· 283 426 ├── src/ 284 427 │ ├── lib/ 285 428 │ │ ├── auth.ts # Authentication & identity resolution 286 - │ │ ├── cli.ts # Command line argument parsing 429 + │ │ ├── cli.ts # Command line interface & argument parsing 287 430 │ │ ├── csv.ts # CSV parsing & record conversion 288 - │ │ └── publisher.ts # Batch publishing with rate limiting 431 + │ │ ├── publisher.ts # Batch publishing with rate limiting 432 + │ │ ├── spotify.ts # Spotify JSON parsing 433 + │ │ ├── merge.ts # Combined import deduplication 434 + │ │ └── sync.ts # Re-sync mode & duplicate detection 289 435 │ ├── utils/ 436 + │ │ ├── logger.ts # Structured logging system (NEW!) 290 437 │ │ ├── helpers.ts # Utility functions (timing, formatting) 291 438 │ │ ├── input.ts # User input handling (prompts, passwords) 292 - │ │ └── rate-limiter.ts # Rate limiting calculations 439 + │ │ ├── rate-limiter.ts # Rate limiting calculations 440 + │ │ ├── killswitch.ts # Graceful shutdown handling 441 + │ │ ├── tid.ts # TID generation from timestamps 442 + │ │ └── ui.ts # UI elements (spinners, progress bars) 293 443 │ ├── config.ts # Configuration constants 294 444 │ └── types.ts # TypeScript type definitions 295 445 ├── lexicons/ # fm.teal.alpha lexicon definitions ··· 298 448 │ └── play.json # Play record schema 299 449 ├── package.json 300 450 ├── tsconfig.json 451 + ├── CLI_IMPROVEMENTS.md # Detailed CLI documentation 301 452 └── README.md 302 453 ``` 303 454 ··· 328 479 1. Calculates safe daily limit (90% of 1K = 900 records/day) 329 480 2. Determines how many days needed for your import 330 481 3. Calculates optimal batch size and delay to spread records evenly 331 - 4. Enforces minimum 1 second delay between batches 482 + 4. Enforces minimum 500ms delay between batches 332 483 5. Shows clear schedule before starting 333 484 334 485 ### Record Processing ··· 397 548 - Check progress messages - large imports take time 398 549 - Multi-day imports pause for 24 hours between days 399 550 - You can safely stop (Ctrl+C) and resume later 551 + - Use `--verbose` flag to see detailed progress 552 + 553 + ### Too much output 554 + - Use `--quiet` flag to suppress non-essential messages 555 + - Only warnings and errors will be shown 556 + 557 + ### Need more details 558 + - Use `--verbose` flag to see debug-level information 559 + - Shows batch timing, API calls, and detailed progress 400 560 401 561 ## Contributing 402 562 ··· 406 566 3. Make your changes with tests 407 567 4. Submit a pull request 408 568 569 + See `CLI_IMPROVEMENTS.md` for developer documentation on the logging system and CLI structure. 570 + 409 571 ## License 410 572 411 - MIT License - See LICENSE file for details 573 + AGPL-3.0-only - See LICENCE file for details 412 574 413 575 ## Credits 414 576 ··· 416 578 - CSV parsing via [csv-parse](https://www.npmjs.com/package/csv-parse) 417 579 - Identity resolution via [Slingshot](https://slingshot.danner.cloud) 418 580 - Follows the `fm.teal.alpha` lexicon standard 581 + - Colored output via [chalk](https://www.npmjs.com/package/chalk) 419 582 420 583 --- 421 584 422 - **Note**: This tool is for personal use. Respect the terms of service and rate limits when exporting your data. 585 + **Note**: This tool is for personal use. Respect the terms of service and rate limits when exporting your data.
+4 -4
package-lock.json
··· 1 1 { 2 2 "name": "lastfm-importer", 3 - "version": "0.3.0", 3 + "version": "0.4.0", 4 4 "lockfileVersion": 3, 5 5 "requires": true, 6 6 "packages": { 7 7 "": { 8 8 "name": "lastfm-importer", 9 - "version": "0.3.0", 9 + "version": "0.4.0", 10 10 "license": "AGPL-3.0-only", 11 11 "dependencies": { 12 12 "@atproto/api": "^0.13.35", ··· 330 330 "integrity": "sha512-tsaTIkKW9b4N+AEj+SVA+WhJzV7/zMhcSu78mLKWSk7cXMOSHsBKFWUs0fWwq8QyK3MgJBQRX6Gbi4kYbdvGkQ==", 331 331 "license": "MIT", 332 332 "dependencies": { 333 - "emoji-regex": "^10.3.0", 334 - "get-east-asian-width": "^1.0.0", 333 + "emoji-regex": "^10.4.0", 334 + "get-east-asian-width": "^0.4.0", 335 335 "strip-ansi": "^7.1.0" 336 336 }, 337 337 "engines": {
+1 -1
package.json
··· 1 1 { 2 2 "name": "lastfm-importer", 3 - "version": "0.3.0", 3 + "version": "0.4.0", 4 4 "description": "Import Last.fm scrobbles to ATProto with rate limiting", 5 5 "type": "module", 6 6 "main": "./dist/index.js",
+15 -2
src/config.ts
··· 16 16 // Record type 17 17 export const RECORD_TYPE = 'fm.teal.alpha.feed.play'; 18 18 19 - // Client agent 20 - export const CLIENT_AGENT = `lastfm-importer/v0.3.0 (${process.platform}; Node/${process.version})`; 19 + // Build client agent string 20 + export function buildClientAgent() { 21 + const PLATFORM_LABELS: Record<string, string> = { 22 + darwin: 'macOS', 23 + linux: 'Linux', 24 + win32: 'Windows', 25 + }; 26 + 27 + const platform = 28 + PLATFORM_LABELS[process.platform] ?? process.platform; 29 + 30 + return `lastfm-importer/v0.4.0 (${platform}; Node/${process.version})`; 31 + } 32 + 33 + const CLIENT_AGENT = buildClientAgent(); 21 34 22 35 // Default batch configuration - aggressive defaults for maximum speed 23 36 // Will dynamically adjust based on success/failure
+385 -215
src/lib/cli.ts
··· 1 + #!/usr/bin/env node 2 + 1 3 import { parseArgs } from 'node:util'; 2 - import { AtpAgent } from '@atproto/api'; // Use AtpAgent for consistency 3 - import type { PlayRecord, Config, CommandLineArgs, PublishResult } from '../types.js'; 4 - import { login } from './auth.js'; 4 + import { AtpAgent } from '@atproto/api'; 5 + import type { PlayRecord, Config, CommandLineArgs, PublishResult } from '../types.js'; 6 + import { login } from './auth.js'; 5 7 import { parseLastFmCsv, convertToPlayRecord, sortRecords } from '../lib/csv.js'; 6 - import { parseSpotifyJson, convertSpotifyToPlayRecord, sortSpotifyRecords } from '../lib/spotify.js'; 7 - import { publishRecordsWithApplyWrites } from './publisher.js'; 8 - import { prompt } from '../utils/input.js'; 9 - import config from '../config.js'; 10 - import { calculateOptimalBatchSize, showRateLimitInfo } from '../utils/helpers.js'; 11 - import { fetchExistingRecords, filterNewRecords, displaySyncStats, removeDuplicates } from './sync.js'; 8 + import { parseSpotifyJson, convertSpotifyToPlayRecord, sortSpotifyRecords } from '../lib/spotify.js'; 9 + import { parseCombinedExports } from '../lib/merge.js'; 10 + import { publishRecordsWithApplyWrites } from './publisher.js'; 11 + import { prompt } from '../utils/input.js'; 12 + import config from '../config.js'; 13 + import { calculateOptimalBatchSize } from '../utils/helpers.js'; 14 + import { fetchExistingRecords, filterNewRecords, displaySyncStats, removeDuplicates } from './sync.js'; 15 + import { Logger, LogLevel, setGlobalLogger, log } from '../utils/logger.js'; 12 16 13 17 /** 14 18 * Show help message 15 19 */ 16 20 export function showHelp(): void { 17 - console.log(` 18 - Last.fm to ATProto Importer v0.3.0 21 + console.log(` 22 + ${'\x1b[1m'}Last.fm to ATProto Importer v0.4.0${'\x1b[0m'} 23 + 24 + ${'\x1b[1m'}USAGE:${'\x1b[0m'} 25 + npm start [options] 26 + lastfm-import [options] 27 + 28 + ${'\x1b[1m'}AUTHENTICATION:${'\x1b[0m'} 29 + -h, --handle <handle> ATProto handle or DID (e.g., user.bsky.social) 30 + -p, --password <password> ATProto app password 31 + 32 + ${'\x1b[1m'}INPUT:${'\x1b[0m'} 33 + -i, --input <path> Path to Last.fm CSV or Spotify JSON export 34 + --spotify-input <path> Path to Spotify export (for combined mode) 19 35 20 - Usage: npm start [options] 36 + ${'\x1b[1m'}MODE:${'\x1b[0m'} 37 + -m, --mode <mode> Import mode (default: lastfm) 38 + lastfm Import Last.fm export only 39 + spotify Import Spotify export only 40 + combined Merge Last.fm + Spotify exports 41 + sync Skip existing records (sync mode) 42 + deduplicate Remove duplicate records 21 43 22 - Options: 23 - -h, --help Show this help message 24 - -f, --file <path> Path to Last.fm CSV or Spotify JSON export file/directory 25 - -i, --identifier <id> ATProto handle or DID 26 - -p, --password <pass> ATProto app password 27 - -b, --batch-size <num> Number of records per batch (auto-calculated if not set) 44 + ${'\x1b[1m'}BATCH CONFIGURATION:${'\x1b[0m'} 45 + -b, --batch-size <number> Records per batch (default: auto-calculated) 28 46 -d, --batch-delay <ms> Delay between batches in ms (default: 500, min: 500) 29 - -y, --yes Skip confirmation prompt 30 - -n, --dry-run Preview records without publishing 31 - -r, --reverse-chronological Process newest first (default: oldest first) 32 - -s, --sync Re-sync mode: check existing Teal records and only import new ones 33 - --spotify Import from Spotify JSON export instead of Last.fm CSV 34 - --remove-duplicates Remove duplicate records from Teal (keeps first occurrence) 47 + 48 + ${'\x1b[1m'}IMPORT OPTIONS:${'\x1b[0m'} 49 + -r, --reverse Process newest records first (default: oldest first) 50 + -y, --yes Skip confirmation prompts 51 + --dry-run Preview without importing 52 + 53 + ${'\x1b[1m'}OUTPUT:${'\x1b[0m'} 54 + -v, --verbose Enable verbose logging (debug level) 55 + -q, --quiet Suppress non-essential output 56 + --help Show this help message 57 + 58 + ${'\x1b[1m'}EXAMPLES:${'\x1b[0m'} 59 + 60 + ${'\x1b[2m'}# Import Last.fm export${'\x1b[0m'} 61 + npm start -- -i lastfm-export.csv -h user.bsky.social -p app-password 62 + 63 + ${'\x1b[2m'}# Import Spotify export${'\x1b[0m'} 64 + npm start -- -i spotify-export/ -m spotify -h user.bsky.social -p app-password 65 + 66 + ${'\x1b[2m'}# Combined import (merge both sources)${'\x1b[0m'} 67 + npm start -- -i lastfm.csv --spotify-input spotify/ -m combined -h user.bsky.social -p pass 68 + 69 + ${'\x1b[2m'}# Sync mode (only import new records)${'\x1b[0m'} 70 + npm start -- -i lastfm.csv -m sync -h user.bsky.social -p app-password 71 + 72 + ${'\x1b[2m'}# Dry run with verbose logging${'\x1b[0m'} 73 + npm start -- -i lastfm.csv --dry-run -v 74 + 75 + ${'\x1b[2m'}# Remove duplicate records${'\x1b[0m'} 76 + npm start -- -m deduplicate -h user.bsky.social -p app-password 77 + 78 + ${'\x1b[1m'}NOTES:${'\x1b[0m'} 79 + • Rate limits: Max 10,000 records/day to avoid PDS rate limiting 80 + • Import will auto-pause between days for large datasets 81 + • Press Ctrl+C during import to stop gracefully after current batch 82 + • Sync mode requires authentication even with --dry-run 83 + 84 + ${'\x1b[1m'}MORE INFO:${'\x1b[0m'} 85 + Repository: https://github.com/yourusername/atproto-lastfm-importer 86 + Issues: https://github.com/yourusername/atproto-lastfm-importer/issues 35 87 `); 36 88 } 37 89 ··· 39 91 * Parse command line arguments 40 92 */ 41 93 export function parseCommandLineArgs(): CommandLineArgs { 42 - // The options definition is identical to the CommandLineArgs keys 43 - const options = { 44 - help: { type: 'boolean', short: 'h', default: false }, 45 - file: { type: 'string', short: 'f' }, 46 - identifier: { type: 'string', short: 'i' }, 47 - password: { type: 'string', short: 'p' }, 48 - 'batch-size': { type: 'string', short: 'b' }, 49 - 'batch-delay': { type: 'string', short: 'd' }, 50 - yes: { type: 'boolean', short: 'y', default: false }, 51 - 'dry-run': { type: 'boolean', short: 'n', default: false }, 52 - 'reverse-chronological': { type: 'boolean', short: 'r', default: false }, 53 - sync: { type: 'boolean', short: 's', default: false }, 54 - spotify: { type: 'boolean', default: false }, 55 - 'remove-duplicates': { type: 'boolean', default: false }, 56 - } as const; 94 + const options = { 95 + // Help 96 + help: { type: 'boolean', default: false }, 97 + 98 + // Authentication 99 + handle: { type: 'string', short: 'h' }, 100 + password: { type: 'string', short: 'p' }, 101 + 102 + // Input 103 + input: { type: 'string', short: 'i' }, 104 + 'spotify-input': { type: 'string' }, 105 + 106 + // Mode 107 + mode: { type: 'string', short: 'm' }, 108 + 109 + // Batch configuration 110 + 'batch-size': { type: 'string', short: 'b' }, 111 + 'batch-delay': { type: 'string', short: 'd' }, 112 + 113 + // Import options 114 + reverse: { type: 'boolean', short: 'r', default: false }, 115 + yes: { type: 'boolean', short: 'y', default: false }, 116 + 'dry-run': { type: 'boolean', default: false }, 117 + 118 + // Output 119 + verbose: { type: 'boolean', short: 'v', default: false }, 120 + quiet: { type: 'boolean', short: 'q', default: false }, 57 121 58 - try { 59 - const { values } = parseArgs({ options, allowPositionals: false }); 60 - return values as CommandLineArgs; 61 - } catch (error) { 62 - const err = error as Error; 63 - console.error('Error parsing arguments:', err.message); 64 - showHelp(); 65 - process.exit(1); 122 + // Legacy flags for backwards compatibility (hidden from help) 123 + file: { type: 'string', short: 'f' }, // Maps to --input 124 + 'spotify-file': { type: 'string' }, // Maps to --spotify-input 125 + identifier: { type: 'string' }, // Maps to --handle 126 + 'reverse-chronological': { type: 'boolean' }, // Maps to --reverse 127 + sync: { type: 'boolean', short: 's' }, // Maps to --mode sync 128 + spotify: { type: 'boolean' }, // Maps to --mode spotify 129 + combined: { type: 'boolean' }, // Maps to --mode combined 130 + 'remove-duplicates': { type: 'boolean' }, // Maps to --mode deduplicate 131 + } as const; 132 + 133 + try { 134 + const { values } = parseArgs({ options, allowPositionals: false }); 135 + 136 + // Handle legacy flag mappings 137 + const normalizedArgs: CommandLineArgs = { 138 + help: values.help, 139 + handle: values.handle || values.identifier, 140 + password: values.password, 141 + input: values.input || values.file, 142 + 'spotify-input': values['spotify-input'] || values['spotify-file'], 143 + 'batch-size': values['batch-size'], 144 + 'batch-delay': values['batch-delay'], 145 + reverse: values.reverse || values['reverse-chronological'], 146 + yes: values.yes, 147 + 'dry-run': values['dry-run'], 148 + verbose: values.verbose, 149 + quiet: values.quiet, 150 + }; 151 + 152 + // Determine mode from new --mode flag or legacy flags 153 + if (values.mode) { 154 + normalizedArgs.mode = values.mode; 155 + } else if (values['remove-duplicates']) { 156 + normalizedArgs.mode = 'deduplicate'; 157 + } else if (values.combined) { 158 + normalizedArgs.mode = 'combined'; 159 + } else if (values.sync) { 160 + normalizedArgs.mode = 'sync'; 161 + } else if (values.spotify) { 162 + normalizedArgs.mode = 'spotify'; 163 + } else { 164 + normalizedArgs.mode = 'lastfm'; // default 66 165 } 166 + 167 + return normalizedArgs; 168 + } catch (error) { 169 + const err = error as Error; 170 + console.error('Error parsing arguments:', err.message); 171 + showHelp(); 172 + process.exit(1); 173 + } 174 + } 175 + 176 + /** 177 + * Validate and normalize mode 178 + */ 179 + function validateMode(mode: string): 'lastfm' | 'spotify' | 'combined' | 'sync' | 'deduplicate' { 180 + const validModes = ['lastfm', 'spotify', 'combined', 'sync', 'deduplicate']; 181 + const normalized = mode.toLowerCase(); 182 + 183 + if (!validModes.includes(normalized)) { 184 + throw new Error( 185 + `Invalid mode: ${mode}. Must be one of: ${validModes.join(', ')}` 186 + ); 187 + } 188 + 189 + return normalized as 'lastfm' | 'spotify' | 'combined' | 'sync' | 'deduplicate'; 67 190 } 68 191 69 192 /** 70 193 * The full, real implementation of the CLI 71 194 */ 72 195 export async function runCLI(): Promise<void> { 73 - try { 74 - const args = parseCommandLineArgs(); 75 - const cfg = config as Config; // Use a constant for the typed config 196 + try { 197 + const args = parseCommandLineArgs(); 198 + const cfg = config as Config; 76 199 77 - if (args.help) { 78 - showHelp(); 79 - return; 80 - } 200 + // Setup logging 201 + const logger = new Logger( 202 + args.quiet ? LogLevel.WARN : 203 + args.verbose ? LogLevel.DEBUG : 204 + LogLevel.INFO 205 + ); 206 + setGlobalLogger(logger); 81 207 82 - if (!args.file) { 83 - throw new Error('Missing required argument: -f, --file <path>'); 84 - } 208 + if (args.help) { 209 + showHelp(); 210 + return; 211 + } 85 212 86 - const dryRun = args['dry-run'] ?? false; 87 - const syncMode = args.sync ?? false; 88 - const removeDuplicatesMode = args['remove-duplicates'] ?? false; 89 - let agent: AtpAgent | null = null; 213 + // Validate and normalize mode 214 + const mode = validateMode(args.mode || 'lastfm'); 215 + const dryRun = args['dry-run'] ?? false; 216 + let agent: AtpAgent | null = null; 90 217 91 - // Remove duplicates mode - requires authentication but not file 92 - if (removeDuplicatesMode) { 93 - if (!args.identifier || !args.password) { 94 - throw new Error('Missing required arguments for login: -i (identifier) and -p (password)'); 95 - } 96 - 97 - agent = await login(args.identifier, args.password, cfg.SLINGSHOT_RESOLVER) as AtpAgent; 98 - 99 - // Check for duplicates first 100 - const result = await removeDuplicates(agent, cfg, true); // Always dry-run first to show info 101 - 102 - if (result.totalDuplicates === 0) { 103 - return; // No duplicates, exit early 104 - } 105 - 106 - // Ask for confirmation if not in dry-run mode 107 - if (!dryRun && !(args.yes ?? false)) { 108 - console.log(`⚠️ WARNING: This will permanently delete ${result.totalDuplicates} duplicate records from Teal.`); 109 - console.log(' The first occurrence of each duplicate will be kept.\n'); 110 - const answer = await prompt('Are you sure you want to continue? (y/N) '); 111 - if (answer.toLowerCase() !== 'y') { 112 - console.log('Duplicate removal cancelled by user.'); 113 - process.exit(0); 114 - } 115 - 116 - // Actually remove duplicates 117 - await removeDuplicates(agent, cfg, false); 118 - console.log('🎉 Duplicate removal complete!\n'); 119 - } else if (dryRun) { 120 - console.log('DRY RUN: No records were actually removed.\n'); 121 - console.log('Remove --dry-run flag to actually delete duplicates.\n'); 122 - } 123 - 124 - return; 125 - } 218 + log.debug(`Mode: ${mode}`); 219 + log.debug(`Dry run: ${dryRun}`); 220 + log.debug(`Log level: ${args.verbose ? 'DEBUG' : args.quiet ? 'WARN' : 'INFO'}`); 126 221 127 - // 1. Get Authentication (required for sync mode, even in dry-run) 128 - if (!dryRun || syncMode) { 129 - if (!args.identifier || !args.password) { 130 - throw new Error('Missing required arguments for login: -i (identifier) and -p (password)'); 131 - } 132 - // Assume login returns AtpAgent, as per the type fix 133 - agent = await login(args.identifier, args.password, cfg.SLINGSHOT_RESOLVER) as AtpAgent; 134 - } 222 + // Validate mode-specific requirements 223 + if (mode === 'combined') { 224 + if (!args.input || !args['spotify-input']) { 225 + throw new Error('Combined mode requires both --input (Last.fm) and --spotify-input (Spotify)'); 226 + } 227 + } else if (mode !== 'deduplicate' && !args.input) { 228 + throw new Error('Missing required argument: --input <path>'); 229 + } 135 230 136 - // 2. Parse and Prepare Records 137 - const useSpotify = args.spotify ?? false; 138 - let records: PlayRecord[]; 139 - let rawRecordCount: number; 140 - 141 - if (useSpotify) { 142 - console.log('📀 Importing from Spotify export...\n'); 143 - const spotifyRecords = parseSpotifyJson(args.file); 144 - rawRecordCount = spotifyRecords.length; 145 - records = spotifyRecords.map(record => convertSpotifyToPlayRecord(record, cfg)); 146 - } else { 147 - console.log('📀 Importing from Last.fm CSV export...\n'); 148 - const csvRecords = parseLastFmCsv(args.file); 149 - rawRecordCount = csvRecords.length; 150 - records = csvRecords.map(record => convertToPlayRecord(record, cfg)); 151 - } 152 - 153 - // 2.5. Sync Mode: Fetch existing records and filter duplicates 154 - if (syncMode && agent) { 155 - const originalRecords = [...records]; // Save before filtering 156 - const existingRecords = await fetchExistingRecords(agent, cfg); 157 - records = filterNewRecords(records, existingRecords); 158 - 159 - if (records.length === 0) { 160 - console.log('✓ All records already exist in Teal. Nothing to import!'); 161 - process.exit(0); 162 - } 163 - 164 - displaySyncStats(originalRecords, existingRecords, records); 165 - } 166 - 167 - const totalRecords = records.length; 168 - 169 - const reverseChronological = args['reverse-chronological'] ?? false; 170 - const sortedRecords = useSpotify 171 - ? sortSpotifyRecords(records, reverseChronological) 172 - : sortRecords(records, reverseChronological); 173 - 174 - // 3. Determine Batching parameters 175 - let batchDelay = cfg.DEFAULT_BATCH_DELAY; 176 - if (args['batch-delay']) { 177 - const delay = parseInt(args['batch-delay'], 10); 178 - if (isNaN(delay)) { 179 - throw new Error(`Invalid batch delay value: ${args['batch-delay']}`); 180 - } 181 - // Enforce minimum delay 182 - batchDelay = Math.max(delay, cfg.MIN_BATCH_DELAY); 183 - } 184 - 185 - let batchSize: number; 186 - if (args['batch-size']) { 187 - batchSize = parseInt(args['batch-size'], 10); 188 - if (isNaN(batchSize) || batchSize <= 0) { 189 - throw new Error(`Invalid batch size value: ${args['batch-size']}`); 190 - } 191 - } else { 192 - // Calculate optimal batch size if not provided 193 - batchSize = calculateOptimalBatchSize(totalRecords, batchDelay, cfg); 194 - } 231 + // Deduplicate mode 232 + if (mode === 'deduplicate') { 233 + if (!args.handle || !args.password) { 234 + throw new Error('Deduplicate mode requires --handle and --password'); 235 + } 195 236 196 - // 4. Show Rate Limiting Information 197 - const recordsPerDay = cfg.RECORDS_PER_DAY_LIMIT * cfg.SAFETY_MARGIN; 198 - const estimatedDays = Math.ceil(totalRecords / recordsPerDay); 237 + log.section('Remove Duplicate Records'); 238 + agent = await login(args.handle, args.password, cfg.SLINGSHOT_RESOLVER) as AtpAgent; 199 239 200 - // Updated call to match the expected signature in showRateLimitInfo (from previous response) 201 - showRateLimitInfo( 202 - totalRecords, 203 - batchSize, 204 - batchDelay, 205 - estimatedDays, 206 - cfg.RECORDS_PER_DAY_LIMIT, 207 - ); 240 + const result = await removeDuplicates(agent, cfg, true); 208 241 209 - // 5. Confirmation Prompt 210 - if (!dryRun && !(args.yes ?? false)) { 211 - if (syncMode) { 212 - console.log(`\nReady to publish ${totalRecords.toLocaleString()} NEW records (${rawRecordCount - totalRecords} duplicates skipped).`); 213 - } else { 214 - console.log(`\nReady to publish ${totalRecords.toLocaleString()} records.`); 215 - } 216 - const answer = await prompt('Do you want to continue? (y/N) '); 217 - if (answer.toLowerCase() !== 'y') { 218 - console.log('Import cancelled by user.'); 219 - process.exit(0); 220 - } 242 + if (result.totalDuplicates === 0) { 243 + return; 244 + } 245 + 246 + if (!dryRun && !args.yes) { 247 + log.warn(`This will permanently delete ${result.totalDuplicates} duplicate records from Teal.`); 248 + log.info('The first occurrence of each duplicate will be kept.'); 249 + log.blank(); 250 + const answer = await prompt('Are you sure you want to continue? (y/N) '); 251 + if (answer.toLowerCase() !== 'y') { 252 + log.info('Duplicate removal cancelled by user.'); 253 + process.exit(0); 221 254 } 222 - 223 - // 6. Publish Records 224 - const result: PublishResult = await publishRecordsWithApplyWrites( 225 - agent, 226 - sortedRecords, 227 - batchSize, 228 - batchDelay, 229 - cfg, 230 - dryRun, 231 - syncMode 232 - ); 255 + 256 + await removeDuplicates(agent, cfg, false); 257 + log.success('Duplicate removal complete!'); 258 + } else if (dryRun) { 259 + log.info('DRY RUN: No records were actually removed.'); 260 + log.info('Remove --dry-run flag to actually delete duplicates.'); 261 + } 262 + 263 + return; 264 + } 265 + 266 + // Authentication (required for sync mode, even in dry-run) 267 + if (!dryRun || mode === 'sync') { 268 + if (!args.handle || !args.password) { 269 + throw new Error('Missing required arguments: --handle and --password'); 270 + } 271 + log.debug('Authenticating...'); 272 + agent = await login(args.handle, args.password, cfg.SLINGSHOT_RESOLVER) as AtpAgent; 273 + log.debug('Authentication successful'); 274 + } 233 275 234 - // 7. Final Output 235 - if (result.cancelled) { 236 - console.log(`\nImport stopped gracefully. ${result.successCount} records processed.`); 237 - } else if (dryRun) { 238 - console.log(`\nDRY RUN COMPLETE${syncMode ? ' (SYNC MODE)' : ''}. No records were published.`); 239 - } else { 240 - console.log(`\n🎉 ${syncMode ? 'Sync' : 'Import'} Complete!`); 241 - console.log(`Total records processed: ${result.successCount.toLocaleString()} (${result.errorCount.toLocaleString()} failed)`); 242 - if (syncMode) { 243 - console.log(`Duplicates skipped: ${rawRecordCount - totalRecords}`); 244 - } 276 + // Parse and prepare records 277 + log.section('Loading Records'); 278 + let records: PlayRecord[]; 279 + let rawRecordCount: number; 280 + 281 + if (mode === 'combined') { 282 + log.info('Merging Last.fm and Spotify exports...'); 283 + records = parseCombinedExports(args.input!, args['spotify-input']!, cfg); 284 + rawRecordCount = records.length; 285 + } else if (mode === 'spotify') { 286 + log.info('Importing from Spotify export...'); 287 + const spotifyRecords = parseSpotifyJson(args.input!); 288 + rawRecordCount = spotifyRecords.length; 289 + records = spotifyRecords.map(record => convertSpotifyToPlayRecord(record, cfg)); 290 + } else { 291 + log.info('Importing from Last.fm CSV export...'); 292 + const csvRecords = parseLastFmCsv(args.input!); 293 + rawRecordCount = csvRecords.length; 294 + records = csvRecords.map(record => convertToPlayRecord(record, cfg)); 295 + } 296 + 297 + log.success(`Loaded ${rawRecordCount.toLocaleString()} records`); 298 + 299 + // Sync mode: filter existing records 300 + if (mode === 'sync' && agent) { 301 + log.section('Sync Mode'); 302 + log.info('Checking for existing records...'); 303 + const originalRecords = [...records]; 304 + const existingRecords = await fetchExistingRecords(agent, cfg); 305 + records = filterNewRecords(records, existingRecords); 306 + 307 + if (records.length === 0) { 308 + log.success('All records already exist in Teal. Nothing to import!'); 309 + process.exit(0); 310 + } 311 + 312 + displaySyncStats(originalRecords, existingRecords, records); 313 + } 314 + 315 + const totalRecords = records.length; 316 + 317 + // Sort records (skip for combined mode as it already sorts) 318 + if (mode !== 'combined') { 319 + log.debug(`Sorting records (reverse: ${args.reverse})...`); 320 + records = mode === 'spotify' 321 + ? sortSpotifyRecords(records, args.reverse ?? false) 322 + : sortRecords(records, args.reverse ?? false); 323 + } 324 + 325 + // Determine batch parameters 326 + log.section('Batch Configuration'); 327 + 328 + let batchDelay = cfg.DEFAULT_BATCH_DELAY; 329 + if (args['batch-delay']) { 330 + const delay = parseInt(args['batch-delay'], 10); 331 + if (isNaN(delay)) { 332 + throw new Error(`Invalid batch delay: ${args['batch-delay']}`); 333 + } 334 + batchDelay = Math.max(delay, cfg.MIN_BATCH_DELAY); 335 + if (delay < cfg.MIN_BATCH_DELAY) { 336 + log.warn(`Batch delay increased to minimum: ${cfg.MIN_BATCH_DELAY}ms`); 337 + } 338 + } 339 + 340 + let batchSize: number; 341 + if (args['batch-size']) { 342 + batchSize = parseInt(args['batch-size'], 10); 343 + if (isNaN(batchSize) || batchSize <= 0) { 344 + throw new Error(`Invalid batch size: ${args['batch-size']}`); 345 + } 346 + log.info(`Using manual batch size: ${batchSize} records`); 347 + } else { 348 + batchSize = calculateOptimalBatchSize(totalRecords, batchDelay, cfg); 349 + log.info(`Using auto-calculated batch size: ${batchSize} records`); 350 + } 351 + 352 + log.info(`Batch delay: ${batchDelay}ms`); 353 + 354 + // Show rate limiting information 355 + log.section('Import Configuration'); 356 + log.info(`Total records: ${totalRecords.toLocaleString()}`); 357 + log.info(`Batch size: ${batchSize} records`); 358 + log.info(`Batch delay: ${batchDelay}ms`); 359 + 360 + const recordsPerDay = cfg.RECORDS_PER_DAY_LIMIT * cfg.SAFETY_MARGIN; 361 + const estimatedDays = Math.ceil(totalRecords / recordsPerDay); 362 + 363 + if (estimatedDays > 1) { 364 + log.info(`Duration: ${estimatedDays} days (${recordsPerDay.toLocaleString()} records/day limit)`); 365 + log.warn('Large import will span multiple days with automatic pauses'); 366 + } 367 + 368 + log.blank(); 369 + 370 + // Confirmation prompt 371 + if (!dryRun && !args.yes) { 372 + const modeLabel = mode === 'combined' ? 'merged' : mode === 'sync' ? 'new' : ''; 373 + const skippedInfo = mode === 'sync' ? ` (${rawRecordCount - totalRecords} skipped)` : ''; 374 + log.raw(`Ready to publish ${totalRecords.toLocaleString()} ${modeLabel} records${skippedInfo}`); 375 + const answer = await prompt('Continue? (y/N) '); 376 + if (answer.toLowerCase() !== 'y') { 377 + log.info('Cancelled by user.'); 378 + process.exit(0); 379 + } 380 + log.blank(); 381 + } 382 + 383 + // Publish records 384 + log.section('Publishing Records'); 385 + const result: PublishResult = await publishRecordsWithApplyWrites( 386 + agent, 387 + records, 388 + batchSize, 389 + batchDelay, 390 + cfg, 391 + dryRun, 392 + mode === 'sync' || mode === 'combined' 393 + ); 394 + 395 + // Final output 396 + log.blank(); 397 + if (result.cancelled) { 398 + log.warn(`Stopped: ${result.successCount.toLocaleString()} processed`); 399 + } else if (dryRun) { 400 + const modeLabel = mode === 'combined' ? 'COMBINED' : mode === 'sync' ? 'SYNC' : ''; 401 + log.success(`Dry run complete${modeLabel ? ` (${modeLabel})` : ''}`); 402 + } else { 403 + const modeLabel = mode === 'combined' ? 'Combined' : mode === 'sync' ? 'Sync' : 'Import'; 404 + log.success(`${modeLabel} complete!`); 405 + log.info(`Processed: ${result.successCount.toLocaleString()} (${result.errorCount.toLocaleString()} failed)`); 406 + if (mode === 'sync' || mode === 'combined') { 407 + const skipped = rawRecordCount - totalRecords; 408 + if (skipped > 0) { 409 + log.info(`Skipped: ${skipped.toLocaleString()} duplicates`); 245 410 } 411 + } 412 + } 246 413 247 - } catch (error) { 248 - // Handle fatal errors 249 - const err = error as Error; 250 - console.error('\n🛑 A fatal error occurred:'); 251 - console.error(err.message); 252 - process.exit(1); 414 + } catch (error) { 415 + const err = error as Error; 416 + log.blank(); 417 + log.fatal('A fatal error occurred:'); 418 + log.error(err.message); 419 + if (log.getLevel() <= LogLevel.DEBUG) { 420 + console.error(err.stack); 253 421 } 254 - } 422 + process.exit(1); 423 + } 424 + }
+240
src/lib/merge.ts
··· 1 + import type { PlayRecord, Config } from '../types.js'; 2 + import { parseLastFmCsv, convertToPlayRecord } from './csv.js'; 3 + import { parseSpotifyJson, convertSpotifyToPlayRecord } from './spotify.js'; 4 + import { formatDate } from '../utils/helpers.js'; 5 + import { log } from '../utils/logger.js'; 6 + 7 + /** 8 + * Normalized record for comparison 9 + */ 10 + interface NormalizedRecord { 11 + original: PlayRecord; 12 + normalizedTrack: string; 13 + normalizedArtist: string; 14 + timestamp: number; 15 + source: 'lastfm' | 'spotify'; 16 + } 17 + 18 + /** 19 + * Normalize a string for comparison (lowercase, remove extra spaces, punctuation) 20 + */ 21 + function normalizeString(str: string): string { 22 + return str 23 + .toLowerCase() 24 + .replace(/[^\w\s]/g, '') // Remove punctuation 25 + .replace(/\s+/g, ' ') // Normalize whitespace 26 + .trim(); 27 + } 28 + 29 + /** 30 + * Check if two records represent the same play 31 + * They match if they have: 32 + * 1. The same timestamp (within 5 minutes) 33 + * 2. The same normalized track name 34 + * 3. The same normalized artist name 35 + */ 36 + function areRecordsDuplicates(a: NormalizedRecord, b: NormalizedRecord): boolean { 37 + // Check timestamp within 5 minutes (300000 ms) 38 + const timeDiff = Math.abs(a.timestamp - b.timestamp); 39 + if (timeDiff > 300000) { 40 + return false; 41 + } 42 + 43 + // Check normalized track and artist 44 + return ( 45 + a.normalizedTrack === b.normalizedTrack && 46 + a.normalizedArtist === b.normalizedArtist 47 + ); 48 + } 49 + 50 + /** 51 + * Choose the better record between two duplicates 52 + * Prefer Last.fm if it has MusicBrainz IDs, otherwise prefer Spotify 53 + */ 54 + function chooseBetterRecord(a: NormalizedRecord, b: NormalizedRecord): PlayRecord { 55 + // Prefer Last.fm if it has any MusicBrainz IDs 56 + const aHasMbIds = a.source === 'lastfm' && ( 57 + a.original.recordingMbId || 58 + a.original.releaseMbId || 59 + (a.original.artists[0]?.artistMbId) 60 + ); 61 + 62 + const bHasMbIds = b.source === 'lastfm' && ( 63 + b.original.recordingMbId || 64 + b.original.releaseMbId || 65 + (b.original.artists[0]?.artistMbId) 66 + ); 67 + 68 + if (aHasMbIds && !bHasMbIds) return a.original; 69 + if (bHasMbIds && !aHasMbIds) return b.original; 70 + 71 + // Otherwise prefer Spotify for its better metadata quality 72 + if (a.source === 'spotify') return a.original; 73 + if (b.source === 'spotify') return b.original; 74 + 75 + // Default to first record 76 + return a.original; 77 + } 78 + 79 + /** 80 + * Merge and deduplicate records from multiple sources 81 + */ 82 + function mergeRecords( 83 + lastfmRecords: PlayRecord[], 84 + spotifyRecords: PlayRecord[] 85 + ): { merged: PlayRecord[]; stats: MergeStats } { 86 + log.info('Merging Last.fm and Spotify exports...'); 87 + log.blank(); 88 + 89 + const stats: MergeStats = { 90 + lastfmTotal: lastfmRecords.length, 91 + spotifyTotal: spotifyRecords.length, 92 + duplicatesRemoved: 0, 93 + lastfmUnique: 0, 94 + spotifyUnique: 0, 95 + mergedTotal: 0, 96 + }; 97 + 98 + // Normalize all records 99 + const normalizedLastFm: NormalizedRecord[] = lastfmRecords.map(record => ({ 100 + original: record, 101 + normalizedTrack: normalizeString(record.trackName), 102 + normalizedArtist: normalizeString(record.artists[0]?.artistName || ''), 103 + timestamp: new Date(record.playedTime).getTime(), 104 + source: 'lastfm' as const, 105 + })); 106 + 107 + const normalizedSpotify: NormalizedRecord[] = spotifyRecords.map(record => ({ 108 + original: record, 109 + normalizedTrack: normalizeString(record.trackName), 110 + normalizedArtist: normalizeString(record.artists[0]?.artistName || ''), 111 + timestamp: new Date(record.playedTime).getTime(), 112 + source: 'spotify' as const, 113 + })); 114 + 115 + // Combine all records 116 + const allRecords = [...normalizedLastFm, ...normalizedSpotify]; 117 + 118 + // Sort by timestamp 119 + allRecords.sort((a, b) => a.timestamp - b.timestamp); 120 + 121 + // Deduplicate 122 + const uniqueRecords: PlayRecord[] = []; 123 + const seen = new Set<string>(); 124 + 125 + for (const record of allRecords) { 126 + // Create a key for duplicate detection 127 + const key = `${record.normalizedTrack}|${record.normalizedArtist}|${Math.floor(record.timestamp / 60000)}`; // Round to minute 128 + 129 + if (seen.has(key)) { 130 + // Find the existing record to compare 131 + const existingIndex = uniqueRecords.findIndex(r => { 132 + const normalized: NormalizedRecord = { 133 + original: r, 134 + normalizedTrack: normalizeString(r.trackName), 135 + normalizedArtist: normalizeString(r.artists[0]?.artistName || ''), 136 + timestamp: new Date(r.playedTime).getTime(), 137 + source: r.musicServiceBaseDomain === 'last.fm' ? 'lastfm' : 'spotify', 138 + }; 139 + return areRecordsDuplicates(record, normalized); 140 + }); 141 + 142 + if (existingIndex !== -1) { 143 + // This is a duplicate - choose the better one 144 + const existing: NormalizedRecord = { 145 + original: uniqueRecords[existingIndex], 146 + normalizedTrack: normalizeString(uniqueRecords[existingIndex].trackName), 147 + normalizedArtist: normalizeString(uniqueRecords[existingIndex].artists[0]?.artistName || ''), 148 + timestamp: new Date(uniqueRecords[existingIndex].playedTime).getTime(), 149 + source: uniqueRecords[existingIndex].musicServiceBaseDomain === 'last.fm' ? 'lastfm' : 'spotify', 150 + }; 151 + 152 + uniqueRecords[existingIndex] = chooseBetterRecord(existing, record); 153 + stats.duplicatesRemoved++; 154 + continue; 155 + } 156 + } 157 + 158 + seen.add(key); 159 + uniqueRecords.push(record.original); 160 + 161 + // Track source statistics 162 + if (record.source === 'lastfm') { 163 + stats.lastfmUnique++; 164 + } else { 165 + stats.spotifyUnique++; 166 + } 167 + } 168 + 169 + stats.mergedTotal = uniqueRecords.length; 170 + 171 + // Sort final records chronologically 172 + uniqueRecords.sort((a, b) => { 173 + const timeA = new Date(a.playedTime).getTime(); 174 + const timeB = new Date(b.playedTime).getTime(); 175 + return timeA - timeB; 176 + }); 177 + 178 + return { merged: uniqueRecords, stats }; 179 + } 180 + 181 + /** 182 + * Statistics about the merge operation 183 + */ 184 + export interface MergeStats { 185 + lastfmTotal: number; 186 + spotifyTotal: number; 187 + duplicatesRemoved: number; 188 + lastfmUnique: number; 189 + spotifyUnique: number; 190 + mergedTotal: number; 191 + } 192 + 193 + /** 194 + * Display merge statistics 195 + */ 196 + function displayMergeStats(stats: MergeStats, merged: PlayRecord[]): void { 197 + log.blank(); 198 + log.section('Merge Statistics'); 199 + log.info(`Last.fm: ${stats.lastfmTotal.toLocaleString()} records`); 200 + log.info(`Spotify: ${stats.spotifyTotal.toLocaleString()} records`); 201 + log.info(`Duplicates: ${stats.duplicatesRemoved.toLocaleString()} removed`); 202 + log.info(`Result: ${stats.mergedTotal.toLocaleString()} unique records`); 203 + 204 + if (merged.length > 0) { 205 + const firstPlay = formatDate(merged[0].playedTime); 206 + const lastPlay = formatDate(merged[merged.length - 1].playedTime); 207 + log.info(`Range: ${firstPlay} to ${lastPlay}`); 208 + } 209 + log.blank(); 210 + } 211 + 212 + /** 213 + * Parse and merge Last.fm and Spotify exports 214 + */ 215 + export function parseCombinedExports( 216 + lastfmPath: string, 217 + spotifyPath: string, 218 + config: Config 219 + ): PlayRecord[] { 220 + log.section('Combined Import Mode'); 221 + log.blank(); 222 + 223 + // Parse Last.fm 224 + log.info('Parsing Last.fm export...'); 225 + const lastfmCsvRecords = parseLastFmCsv(lastfmPath); 226 + const lastfmRecords = lastfmCsvRecords.map(r => convertToPlayRecord(r, config)); 227 + 228 + // Parse Spotify 229 + log.info('Parsing Spotify export...'); 230 + const spotifyJsonRecords = parseSpotifyJson(spotifyPath); 231 + const spotifyRecords = spotifyJsonRecords.map(r => convertSpotifyToPlayRecord(r, config)); 232 + 233 + // Merge and deduplicate 234 + const { merged, stats } = mergeRecords(lastfmRecords, spotifyRecords); 235 + 236 + // Display statistics 237 + displayMergeStats(stats, merged); 238 + 239 + return merged; 240 + }
+88 -67
src/lib/publisher.ts
··· 9 9 } from '../utils/rate-limiter.js'; 10 10 import { generateTIDFromISO } from '../utils/tid.js'; 11 11 import type { PlayRecord, Config, PublishResult } from '../types.js'; 12 + import { log } from '../utils/logger.js'; 12 13 13 14 /** 14 15 * Maximum operations allowed per applyWrites call ··· 51 52 let consecutiveFailures = 0; 52 53 const MAX_CONSECUTIVE_FAILURES = 3; 53 54 54 - console.log(`\n🚀 Starting adaptive import with aggressive settings...`); 55 - console.log(` Initial batch size: ${currentBatchSize} records`); 56 - console.log(` Initial delay: ${currentBatchDelay}ms`); 57 - console.log(` Will automatically adjust based on server response\n`); 55 + log.section('Adaptive Import'); 56 + log.info(`Initial batch size: ${currentBatchSize} records`); 57 + log.info(`Initial delay: ${currentBatchDelay}ms`); 58 + log.info(`Will automatically adjust based on server response`); 59 + log.blank(); 60 + log.info(`Publishing ${totalRecords.toLocaleString()} records using adaptive batching...`); 61 + log.warn('Press Ctrl+C to stop gracefully after current batch'); 62 + log.blank(); 58 63 59 64 let successCount = 0; 60 65 let errorCount = 0; 61 66 const startTime = Date.now(); 62 - 63 - console.log(`Publishing ${totalRecords} records using adaptive batching...`); 64 - console.log(`\n🚨 Press Ctrl+C to stop gracefully after current batch\n`); 65 67 66 68 let i = 0; 67 69 while (i < totalRecords) { ··· 74 76 const batchNum = Math.floor(i / currentBatchSize) + 1; 75 77 const progress = ((i / totalRecords) * 100).toFixed(1); 76 78 77 - console.log( 79 + log.progress( 78 80 `[${progress}%] Batch ${batchNum} (records ${i + 1}-${Math.min(i + currentBatchSize, totalRecords)}) [size: ${currentBatchSize}, delay: ${currentBatchDelay}ms]` 79 81 ); 80 82 ··· 102 104 consecutiveFailures = 0; 103 105 104 106 const batchDuration = Date.now() - batchStartTime; 105 - console.log( 106 - ` ✓ Complete in ${batchDuration}ms (${batchSuccessCount} successful)` 107 - ); 107 + log.debug(`Batch complete in ${batchDuration}ms (${batchSuccessCount} successful)`); 108 108 109 109 // Speed up if we're doing well (after 5 consecutive successes) 110 110 if (consecutiveSuccesses >= 5 && currentBatchDelay > config.MIN_BATCH_DELAY) { ··· 114 114 Math.floor(currentBatchDelay * 0.8) 115 115 ); 116 116 if (oldDelay !== currentBatchDelay) { 117 - console.log(` ⚡ Speeding up! Delay: ${oldDelay}ms → ${currentBatchDelay}ms`); 117 + log.info(`⚡ Speeding up! Delay: ${oldDelay}ms → ${currentBatchDelay}ms`); 118 118 } 119 119 consecutiveSuccesses = 0; 120 120 } ··· 132 132 consecutiveSuccesses = 0; 133 133 134 134 if (isRateLimitError) { 135 - console.error(` ⚠️ Rate limit hit! Backing off...`); 135 + log.warn('Rate limit hit! Backing off...'); 136 136 137 137 // Exponential backoff 138 138 const backoffMultiplier = Math.pow(2, consecutiveFailures); ··· 147 147 10 // Minimum 10 records 148 148 ); 149 149 150 - console.log(` 📉 Adjusted: batch size → ${currentBatchSize}, delay → ${currentBatchDelay}ms`); 151 - console.log(` ⏳ Waiting ${currentBatchDelay}ms before retry...`); 150 + log.info(`📉 Adjusted: batch size → ${currentBatchSize}, delay → ${currentBatchDelay}ms`); 151 + log.info(`⏳ Waiting ${currentBatchDelay}ms before retry...`); 152 152 153 153 await new Promise((resolve) => setTimeout(resolve, currentBatchDelay)); 154 154 ··· 158 158 } else { 159 159 // Other error - log and continue 160 160 errorCount += batch.length; 161 - console.error(` ✗ Batch failed: ${err.message}`); 161 + log.error(`Batch failed: ${err.message}`); 162 162 163 163 // Log failed records 164 164 batch.slice(0, 3).forEach((record) => { 165 - console.error(` - ${record.trackName} by ${record.artists[0]?.artistName}`); 165 + log.debug(`Failed: ${record.trackName} by ${record.artists[0]?.artistName}`); 166 166 }); 167 167 if (batch.length > 3) { 168 - console.error(` ... and ${batch.length - 3} more`); 168 + log.debug(`... and ${batch.length - 3} more failed`); 169 169 } 170 170 171 171 // If too many consecutive failures, slow down 172 172 if (consecutiveFailures >= MAX_CONSECUTIVE_FAILURES) { 173 173 currentBatchDelay = Math.min(currentBatchDelay * 2, 10000); 174 174 currentBatchSize = Math.max(Math.floor(currentBatchSize / 2), 10); 175 - console.log(` 📉 Multiple failures: adjusted to ${currentBatchSize} records, ${currentBatchDelay}ms delay`); 175 + log.warn(`📉 Multiple failures: adjusted to ${currentBatchSize} records, ${currentBatchDelay}ms delay`); 176 176 } 177 177 178 178 i += batch.length; // Skip failed batch ··· 184 184 const remainingRecords = totalRecords - i; 185 185 const estimatedRemaining = remainingRecords / Math.max(recordsPerSecond, 1); 186 186 187 - console.log( 188 - ` ⏱ Elapsed: ${elapsed} | Speed: ${recordsPerSecond.toFixed(1)} rec/s | Remaining: ~${formatDuration(estimatedRemaining * 1000)}\n` 187 + log.debug( 188 + `Elapsed: ${elapsed} | Speed: ${recordsPerSecond.toFixed(1)} rec/s | Remaining: ~${formatDuration(estimatedRemaining * 1000)}` 189 189 ); 190 + log.blank(); 190 191 191 192 // Check again before waiting 192 193 if (isImportCancelled()) { ··· 236 237 ); 237 238 238 239 if (rateLimitParams.estimatedDays > 1) { 239 - const dailySchedule = calculateDailySchedule( 240 - totalRecords, 241 - batchSize, 242 - batchDelay, 243 - rateLimitParams.recordsPerDay 244 - ); 240 + // Only show daily schedule in verbose/debug mode 241 + if (log.getLevel() <= 0) { // DEBUG level 242 + const dailySchedule = calculateDailySchedule( 243 + totalRecords, 244 + batchSize, 245 + batchDelay, 246 + rateLimitParams.recordsPerDay 247 + ); 245 248 246 - console.log('📅 Multi-Day Import Schedule:\n'); 247 - dailySchedule.forEach((day) => { 248 - console.log(` Day ${day.day}:`); 249 - console.log(` Records ${day.recordsStart + 1}-${day.recordsEnd} (${day.recordsCount} total)`); 250 - if (day.pauseAfter) { 251 - console.log(` → Pause 24h after completion`); 252 - } 253 - }); 254 - console.log(''); 249 + console.log('📅 Multi-Day Import Schedule:\n'); 250 + dailySchedule.forEach((day) => { 251 + console.log(` Day ${day.day}:`); 252 + console.log(` Records ${day.recordsStart + 1}-${day.recordsEnd} (${day.recordsCount} total)`); 253 + if (day.pauseAfter) { 254 + console.log(` → Pause 24h after completion`); 255 + } 256 + }); 257 + console.log(''); 258 + } 255 259 } 256 260 } 257 261 258 - console.log(`\n=== DRY RUN MODE ${syncMode ? '(SYNC)' : ''} ===`); 262 + log.section(`DRY RUN MODE ${syncMode ? '(SYNC)' : ''}`); 259 263 if (syncMode) { 260 - console.log(`Sync mode enabled: Only new records will be published`); 264 + log.info('Sync mode: Only new records will be published'); 261 265 } 262 - console.log(`Would publish ${totalRecords} records using applyWrites`); 263 - console.log(`Batch size: ${Math.min(batchSize, MAX_APPLY_WRITES_OPS)} records per applyWrites call`); 264 - 266 + log.info(`Total: ${totalRecords.toLocaleString()} records`); 267 + log.info(`Batch: ${Math.min(batchSize, MAX_APPLY_WRITES_OPS)} records per call`); 268 + 265 269 if (rateLimitParams.estimatedDays > 1) { 266 - console.log( 267 - `Import would span ${rateLimitParams.estimatedDays} days with automatic pauses\n` 268 - ); 270 + log.info(`Duration: ${rateLimitParams.estimatedDays} days with automatic pauses`); 269 271 } else { 270 - console.log(`Estimated time: ${formatDuration(Math.ceil(totalRecords / batchSize) * batchDelay)}\n`); 272 + log.info(`Time: ~${formatDuration(Math.ceil(totalRecords / batchSize) * batchDelay)}`); 271 273 } 274 + log.blank(); 272 275 273 276 // Show first 5 records as preview 274 277 const previewCount = Math.min(5, totalRecords); 275 - console.log(`Preview of first ${previewCount} records (in processing order):\n`); 278 + log.info(`Preview (first ${previewCount} records):`); 279 + log.blank(); 276 280 277 281 for (let i = 0; i < previewCount; i++) { 278 282 const record = records[i]; 279 - console.log(`${i + 1}. ${record.artists[0]?.artistName} - ${record.trackName}`); 280 - console.log(` Album: ${record.releaseName || 'N/A'}`); 281 - console.log(` Played: ${formatDate(record.playedTime, true)}`); 282 - console.log(` URL: ${record.originUrl}`); 283 - 284 - // Show MusicBrainz IDs if available 285 - const mbids = []; 286 - if (record.artists[0]?.artistMbId) 287 - mbids.push(`Artist: ${record.artists[0].artistMbId}`); 288 - if (record.recordingMbId) mbids.push(`Recording: ${record.recordingMbId}`); 289 - if (record.releaseMbId) mbids.push(`Release: ${record.releaseMbId}`); 290 - 283 + const artistName = record.artists[0]?.artistName || 'Unknown Artist'; 284 + 285 + log.raw(`${i + 1}. ${artistName} - ${record.trackName}`); 286 + 287 + // Album/Release 288 + if (record.releaseName) { 289 + log.raw(` Album: ${record.releaseName}`); 290 + } 291 + 292 + // Timestamp 293 + log.raw(` Played: ${formatDate(record.playedTime, true)}`); 294 + 295 + // Source and URL 296 + log.raw(` Source: ${record.musicServiceBaseDomain}`); 297 + log.raw(` URL: ${record.originUrl}`); 298 + 299 + // MusicBrainz IDs (if available) 300 + const mbids: string[] = []; 301 + if (record.artists[0]?.artistMbId) mbids.push(`Artist: ${record.artists[0].artistMbId}`); 302 + if (record.recordingMbId) mbids.push(`Track: ${record.recordingMbId}`); 303 + if (record.releaseMbId) mbids.push(`Album: ${record.releaseMbId}`); 304 + 291 305 if (mbids.length > 0) { 292 - console.log(` MBIDs: ${mbids.join(', ')}`); 306 + log.raw(` MusicBrainz IDs: ${mbids.join(', ')}`); 293 307 } 294 - console.log(''); 308 + 309 + // Record metadata 310 + log.raw(` Record Type: ${record.$type}`); 311 + log.raw(` Client: ${record.submissionClientAgent}`); 312 + 313 + log.blank(); 295 314 } 296 315 297 316 if (totalRecords > previewCount) { 298 - console.log(`... and ${totalRecords - previewCount} more records\n`); 317 + log.info(`... and ${(totalRecords - previewCount).toLocaleString()} more records`); 318 + log.blank(); 299 319 } 300 320 301 - console.log('=== DRY RUN COMPLETE ==='); 302 - console.log('No records were actually published.'); 303 - console.log('Remove --dry-run flag to publish for real.\n'); 321 + log.section('DRY RUN COMPLETE'); 322 + log.info('No records were published.'); 323 + log.info('Remove --dry-run to publish for real.'); 304 324 305 325 return { successCount: totalRecords, errorCount: 0, cancelled: false }; 306 326 } ··· 313 333 errorCount: number, 314 334 totalRecords: number 315 335 ): PublishResult { 316 - console.log(`\n🛑 Import cancelled by user`); 317 - console.log(` Processed: ${successCount}/${totalRecords} records`); 318 - console.log(` Remaining: ${totalRecords - successCount} records\n`); 336 + log.blank(); 337 + log.warn('Import cancelled by user'); 338 + log.info(`Processed: ${successCount.toLocaleString()}/${totalRecords.toLocaleString()} records`); 339 + log.info(`Remaining: ${(totalRecords - successCount).toLocaleString()} records`); 319 340 return { successCount, errorCount, cancelled: true }; 320 341 }
+33 -36
src/lib/sync.ts
··· 2 2 import type { PlayRecord, Config } from '../types.js'; 3 3 import { formatDate, formatDateRange } from '../utils/helpers.js'; 4 4 import * as ui from '../utils/ui.js'; 5 + import { log } from '../utils/logger.js'; 5 6 6 7 interface ExistingRecord { 7 8 uri: string; ··· 22 23 agent: AtpAgent, 23 24 config: Config 24 25 ): Promise<Map<string, ExistingRecord>> { 25 - console.log('\n=== Fetching Existing Teal Records ==='); 26 + log.section('Fetching Existing Teal Records'); 26 27 const { RECORD_TYPE } = config; 27 28 const did = agent.session?.did; 28 29 ··· 62 63 63 64 // Show progress 64 65 if (totalFetched % 500 === 0 && totalFetched > 0) { 65 - console.log(` Fetched ${totalFetched} records...`); 66 + log.progress(`Fetched ${totalFetched.toLocaleString()} records...`); 66 67 } 67 68 } while (cursor); 68 69 69 - console.log(`✓ Found ${existingRecords.size} existing records\n`); 70 + log.success(`Found ${existingRecords.size.toLocaleString()} existing records`); 71 + log.blank(); 70 72 return existingRecords; 71 73 } catch (error) { 72 74 const err = error as Error; 73 - console.error('✗ Failed to fetch existing records:', err.message); 75 + log.error(`Failed to fetch existing records: ${err.message}`); 74 76 throw error; 75 77 } 76 78 } ··· 154 156 lastfmRecords: PlayRecord[], 155 157 existingRecords: Map<string, ExistingRecord> 156 158 ): PlayRecord[] { 157 - console.log('\n=== Identifying New Records ==='); 159 + log.section('Identifying New Records'); 158 160 159 161 const newRecords: PlayRecord[] = []; 160 162 const duplicates: PlayRecord[] = []; ··· 168 170 } 169 171 } 170 172 171 - console.log(` Total Last.fm records: ${lastfmRecords.length}`); 172 - console.log(` Already in Teal: ${duplicates.length}`); 173 - console.log(` New records to import: ${newRecords.length}\n`); 173 + log.info(`Total: ${lastfmRecords.length.toLocaleString()} records`); 174 + log.info(`Existing: ${duplicates.length.toLocaleString()} already in Teal`); 175 + log.info(`New: ${newRecords.length.toLocaleString()} to import`); 176 + log.blank(); 174 177 175 - // Show some examples of duplicates if any 176 - if (duplicates.length > 0 && duplicates.length <= 5) { 177 - console.log('Examples of existing records (skipped):'); 178 - duplicates.slice(0, 5).forEach((record, i) => { 179 - console.log(` ${i + 1}. ${record.artists[0]?.artistName} - ${record.trackName}`); 180 - console.log(` Played: ${formatDate(record.playedTime, true)}`); 178 + // Show some examples of duplicates if any (only in verbose mode) 179 + if (log.getLevel() <= 0 && duplicates.length > 0) { // DEBUG level 180 + const exampleCount = Math.min(3, duplicates.length); 181 + log.debug('Examples of existing records (skipped):'); 182 + duplicates.slice(0, exampleCount).forEach((record, i) => { 183 + log.debug(` ${i + 1}. ${record.artists[0]?.artistName} - ${record.trackName}`); 184 + log.debug(` ${formatDate(record.playedTime, true)}`); 181 185 }); 182 - console.log(''); 183 - } else if (duplicates.length > 5) { 184 - console.log('Examples of existing records (skipped):'); 185 - duplicates.slice(0, 5).forEach((record, i) => { 186 - console.log(` ${i + 1}. ${record.artists[0]?.artistName} - ${record.trackName}`); 187 - console.log(` Played: ${formatDate(record.playedTime, true)}`); 188 - }); 189 - console.log(` ... and ${duplicates.length - 5} more duplicates\n`); 186 + if (duplicates.length > exampleCount) { 187 + log.debug(` ... and ${(duplicates.length - exampleCount).toLocaleString()} more`); 188 + } 189 + log.blank(); 190 190 } 191 191 192 192 return newRecords; ··· 219 219 const existingArray = Array.from(existingRecords.values()).map(r => r.value); 220 220 const existingRange = getRecordTimeRange(existingArray); 221 221 222 - console.log('=== Sync Statistics ==='); 223 - console.log(`Last.fm Export:`); 224 - console.log(` Total records: ${lastfmRecords.length}`); 222 + log.section('Sync Statistics'); 223 + log.info(`Last.fm export: ${lastfmRecords.length.toLocaleString()} records`); 225 224 if (lastfmRange) { 226 - console.log(` Date range: ${formatDateRange(lastfmRange.earliest, lastfmRange.latest)}`); 225 + log.info(` Range: ${formatDateRange(lastfmRange.earliest, lastfmRange.latest)}`); 227 226 } 228 - console.log(''); 227 + log.blank(); 229 228 230 - console.log(`Teal (Current):`); 231 - console.log(` Total records: ${existingRecords.size}`); 229 + log.info(`Teal current: ${existingRecords.size.toLocaleString()} records`); 232 230 if (existingRange) { 233 - console.log(` Date range: ${formatDateRange(existingRange.earliest, existingRange.latest)}`); 231 + log.info(` Range: ${formatDateRange(existingRange.earliest, existingRange.latest)}`); 234 232 } 235 - console.log(''); 233 + log.blank(); 236 234 237 - console.log(`Sync Result:`); 238 - console.log(` Records to import: ${newRecords.length}`); 239 - console.log(` Duplicates skipped: ${lastfmRecords.length - newRecords.length}`); 240 - console.log(` Match rate: ${((1 - newRecords.length / lastfmRecords.length) * 100).toFixed(1)}%`); 241 - console.log(''); 235 + log.info(`New to import: ${newRecords.length.toLocaleString()}`); 236 + log.info(`Duplicates: ${(lastfmRecords.length - newRecords.length).toLocaleString()} skipped`); 237 + log.info(`Match rate: ${((1 - newRecords.length / lastfmRecords.length) * 100).toFixed(1)}%`); 238 + log.blank(); 242 239 } 243 240 244 241 /**
+27 -3
src/types.ts
··· 35 35 } 36 36 37 37 export interface CommandLineArgs { 38 + // Help 38 39 help?: boolean; 39 - file?: string; 40 - identifier?: string; 40 + 41 + // Authentication 42 + handle?: string; 41 43 password?: string; 44 + 45 + // Input 46 + input?: string; 47 + 'spotify-input'?: string; 48 + 49 + // Mode 50 + mode?: string; 51 + 52 + // Batch configuration 42 53 'batch-size'?: string; 43 54 'batch-delay'?: string; 55 + 56 + // Import options 57 + reverse?: boolean; 44 58 yes?: boolean; 45 59 'dry-run'?: boolean; 60 + 61 + // Output 62 + verbose?: boolean; 63 + quiet?: boolean; 64 + 65 + // Legacy flags (for backwards compatibility) 66 + file?: string; 67 + 'spotify-file'?: string; 68 + identifier?: string; 46 69 'reverse-chronological'?: boolean; 47 70 sync?: boolean; 48 71 spotify?: boolean; 72 + combined?: boolean; 49 73 'remove-duplicates'?: boolean; 50 74 } 51 75 ··· 72 96 SLINGSHOT_RESOLVER: string; 73 97 74 98 RECORD_TYPE: string; 75 - } 99 + }
+3
src/utils/helpers.ts
··· 129 129 130 130 /** 131 131 * Logs rate limiting and batching information to the console. 132 + * Note: This function cannot import log from logger.ts to avoid circular dependencies, 133 + * so it uses console.log directly. The CLI controls the log level, so this output 134 + * is appropriately controlled. 132 135 */ 133 136 export function showRateLimitInfo( 134 137 totalRecords: number,
+216
src/utils/logger.ts
··· 1 + /** 2 + * Structured logging utility with log levels, formatting, and file logging 3 + */ 4 + import chalk from 'chalk'; 5 + import fs from 'fs'; 6 + import path from 'path'; 7 + 8 + export enum LogLevel { 9 + DEBUG = 0, 10 + INFO = 1, 11 + WARN = 2, 12 + ERROR = 3, 13 + SILENT = 4, 14 + } 15 + 16 + export class Logger { 17 + private level: LogLevel; 18 + private prefix: string; 19 + private logFile: string | null = null; 20 + private logStream: fs.WriteStream | null = null; 21 + 22 + constructor(level: LogLevel = LogLevel.INFO, prefix: string = '') { 23 + this.level = level; 24 + this.prefix = prefix; 25 + } 26 + 27 + setLevel(level: LogLevel): void { 28 + this.level = level; 29 + } 30 + 31 + getLevel(): LogLevel { 32 + return this.level; 33 + } 34 + 35 + /** 36 + * Enable logging to file 37 + */ 38 + enableFileLogging(logDir: string = 'logs'): void { 39 + try { 40 + // Create logs directory if it doesn't exist 41 + const logsPath = path.resolve(process.cwd(), logDir); 42 + if (!fs.existsSync(logsPath)) { 43 + fs.mkdirSync(logsPath, { recursive: true }); 44 + } 45 + 46 + // Create log filename with timestamp 47 + const timestamp = new Date().toISOString().replace(/[:.]/g, '-').slice(0, -5); 48 + this.logFile = path.join(logsPath, `import-${timestamp}.log`); 49 + 50 + // Create write stream 51 + this.logStream = fs.createWriteStream(this.logFile, { flags: 'a' }); 52 + 53 + this.writeToFile(`Log started at ${new Date().toISOString()}`); 54 + this.writeToFile(`Log file: ${this.logFile}`); 55 + this.writeToFile('='.repeat(80)); 56 + } catch (error) { 57 + console.error('Failed to enable file logging:', error); 58 + } 59 + } 60 + 61 + /** 62 + * Get the current log file path 63 + */ 64 + getLogFile(): string | null { 65 + return this.logFile; 66 + } 67 + 68 + /** 69 + * Close the log file stream 70 + */ 71 + closeLogFile(): void { 72 + if (this.logStream) { 73 + this.writeToFile('='.repeat(80)); 74 + this.writeToFile(`Log ended at ${new Date().toISOString()}`); 75 + this.logStream.end(); 76 + this.logStream = null; 77 + } 78 + } 79 + 80 + /** 81 + * Write to log file (without formatting) 82 + */ 83 + private writeToFile(message: string): void { 84 + if (this.logStream) { 85 + const timestamp = new Date().toISOString(); 86 + this.logStream.write(`[${timestamp}] ${message}\n`); 87 + } 88 + } 89 + 90 + private shouldLog(level: LogLevel): boolean { 91 + return level >= this.level; 92 + } 93 + 94 + private formatMessage(message: string, prefix?: string): string { 95 + const finalPrefix = prefix || this.prefix; 96 + return finalPrefix ? `${finalPrefix} ${message}` : message; 97 + } 98 + 99 + private logToFile(level: string, message: string): void { 100 + if (this.logStream) { 101 + const cleanMessage = message.replace(/\x1b\[[0-9;]*m/g, ''); // Remove ANSI codes 102 + this.writeToFile(`[${level}] ${cleanMessage}`); 103 + } 104 + } 105 + 106 + debug(message: string, ...args: any[]): void { 107 + if (this.shouldLog(LogLevel.DEBUG)) { 108 + const formatted = chalk.gray(`[DEBUG] ${this.formatMessage(message)}`); 109 + console.log(formatted, ...args); 110 + this.logToFile('DEBUG', this.formatMessage(message)); 111 + } 112 + } 113 + 114 + info(message: string, ...args: any[]): void { 115 + if (this.shouldLog(LogLevel.INFO)) { 116 + const formatted = this.formatMessage(message); 117 + console.log(formatted, ...args); 118 + this.logToFile('INFO', formatted); 119 + } 120 + } 121 + 122 + success(message: string, ...args: any[]): void { 123 + if (this.shouldLog(LogLevel.INFO)) { 124 + const formatted = chalk.green(`✓ ${this.formatMessage(message)}`); 125 + console.log(formatted, ...args); 126 + this.logToFile('SUCCESS', this.formatMessage(message)); 127 + } 128 + } 129 + 130 + warn(message: string, ...args: any[]): void { 131 + if (this.shouldLog(LogLevel.WARN)) { 132 + const formatted = chalk.yellow(`⚠️ ${this.formatMessage(message)}`); 133 + console.warn(formatted, ...args); 134 + this.logToFile('WARN', this.formatMessage(message)); 135 + } 136 + } 137 + 138 + error(message: string, ...args: any[]): void { 139 + if (this.shouldLog(LogLevel.ERROR)) { 140 + const formatted = chalk.red(`✗ ${this.formatMessage(message)}`); 141 + console.error(formatted, ...args); 142 + this.logToFile('ERROR', this.formatMessage(message)); 143 + } 144 + } 145 + 146 + fatal(message: string, ...args: any[]): void { 147 + if (this.shouldLog(LogLevel.ERROR)) { 148 + const formatted = chalk.red.bold(`🛑 ${this.formatMessage(message)}`); 149 + console.error(formatted, ...args); 150 + this.logToFile('FATAL', this.formatMessage(message)); 151 + } 152 + } 153 + 154 + // Progress and status messages (always shown unless SILENT) 155 + progress(message: string, ...args: any[]): void { 156 + if (this.level < LogLevel.SILENT) { 157 + const formatted = chalk.cyan(`→ ${this.formatMessage(message)}`); 158 + console.log(formatted, ...args); 159 + this.logToFile('PROGRESS', this.formatMessage(message)); 160 + } 161 + } 162 + 163 + // Section headers 164 + section(title: string): void { 165 + if (this.shouldLog(LogLevel.INFO)) { 166 + const formatted = chalk.bold(`\n=== ${title} ===`); 167 + console.log(formatted); 168 + this.logToFile('SECTION', title); 169 + } 170 + } 171 + 172 + // Blank line 173 + blank(): void { 174 + if (this.level < LogLevel.SILENT) { 175 + console.log(''); 176 + } 177 + } 178 + 179 + // Raw output (bypasses log level for important user-facing info) 180 + raw(message: string, ...args: any[]): void { 181 + if (this.level < LogLevel.SILENT) { 182 + console.log(message, ...args); 183 + this.logToFile('INFO', message); 184 + } 185 + } 186 + } 187 + 188 + // Global logger instance 189 + let globalLogger: Logger = new Logger(LogLevel.INFO); 190 + 191 + export function setGlobalLogger(logger: Logger): void { 192 + globalLogger = logger; 193 + } 194 + 195 + export function getLogger(): Logger { 196 + return globalLogger; 197 + } 198 + 199 + // Convenience exports 200 + export const log = { 201 + debug: (msg: string, ...args: any[]) => globalLogger.debug(msg, ...args), 202 + info: (msg: string, ...args: any[]) => globalLogger.info(msg, ...args), 203 + success: (msg: string, ...args: any[]) => globalLogger.success(msg, ...args), 204 + warn: (msg: string, ...args: any[]) => globalLogger.warn(msg, ...args), 205 + error: (msg: string, ...args: any[]) => globalLogger.error(msg, ...args), 206 + fatal: (msg: string, ...args: any[]) => globalLogger.fatal(msg, ...args), 207 + progress: (msg: string, ...args: any[]) => globalLogger.progress(msg, ...args), 208 + section: (title: string) => globalLogger.section(title), 209 + blank: () => globalLogger.blank(), 210 + raw: (msg: string, ...args: any[]) => globalLogger.raw(msg, ...args), 211 + setLevel: (level: LogLevel) => globalLogger.setLevel(level), 212 + getLevel: () => globalLogger.getLevel(), 213 + enableFileLogging: (logDir?: string) => globalLogger.enableFileLogging(logDir), 214 + getLogFile: () => globalLogger.getLogFile(), 215 + closeLogFile: () => globalLogger.closeLogFile(), 216 + };
+7 -12
src/utils/rate-limiter.ts
··· 127 127 * Display rate limit warning 128 128 */ 129 129 export function displayRateLimitWarning(): void { 130 - console.log('\n⚠️ ═══════════════════════════════════════════════════════════════════════════════'); 131 - console.log('⚠️ IMPORTANT: Bluesky AppView Rate Limits'); 132 - console.log('⚠️ ═══════════════════════════════════════════════════════════════════════════════'); 133 - console.log('⚠️'); 134 - console.log('⚠️ Exceeding 10K records per day can rate limit your ENTIRE PDS on Bluesky\'s'); 135 - console.log('⚠️ AppView. This affects ALL users on your PDS, not just your account!'); 136 - console.log('⚠️'); 137 - console.log('⚠️ This importer automatically limits imports to 1K records per day by default'); 138 - console.log('⚠️ with automatic batching and pauses to stay within safe limits.'); 139 - console.log('⚠️'); 140 - console.log('⚠️ See: https://docs.bsky.app/blog/rate-limits-pds-v3'); 141 - console.log('⚠️ ═══════════════════════════════════════════════════════════════════════════════\n'); 130 + console.log(''); 131 + console.log('⚠️ IMPORTANT: Rate Limits'); 132 + console.log(' Exceeding 10K records/day can rate limit your ENTIRE PDS.'); 133 + console.log(' This affects ALL users on your PDS, not just your account.'); 134 + console.log(' Import automatically limits to 10K records/day with pauses.'); 135 + console.log(' See: https://docs.bsky.app/blog/rate-limits-pds-v3'); 136 + console.log(''); 142 137 } 143 138 144 139 /**