Microkernel based hobby OS

VFS Plan

Changed files
+1290
docs
+1290
docs/VFS_PLAN.md
··· 1 + # The Path-Keepers: AethelOS Virtual File System Architecture 2 + 3 + **Status:** Planned 4 + **Priority:** High 5 + **Dependencies:** Block device driver, World-Tree design 6 + **Estimated Timeline:** 4-6 weeks 7 + **Version:** 1.0 8 + **Last Updated:** January 2025 9 + 10 + --- 11 + 12 + ## Table of Contents 13 + 14 + 1. [Vision](#vision) 15 + 2. [The Fundamental Insight](#the-fundamental-insight) 16 + 3. [Architecture Overview](#architecture-overview) 17 + 4. [VFS Abstraction Layer](#vfs-abstraction-layer) 18 + 5. [Filesystem Implementations](#filesystem-implementations) 19 + 6. [Integration with World-Tree](#integration-with-world-tree) 20 + 7. [Implementation Phases](#implementation-phases) 21 + 8. [Multi-Boot Support](#multi-boot-support) 22 + 9. [Testing Strategy](#testing-strategy) 23 + 10. [Timeline and Milestones](#timeline-and-milestones) 24 + 25 + --- 26 + 27 + ## Vision 28 + 29 + > *"The World-Tree does not care whether its roots touch ext4, FAT32, or NTFS. It draws nourishment from all soils, transforming base storage into living knowledge."* 30 + 31 + ### The Problem 32 + 33 + World-Tree needs persistent storage for: 34 + - Content-addressed objects (Git-like blobs) 35 + - Metadata indices (queryable attributes) 36 + - Commit graphs (version history) 37 + - Pruning logs (space management) 38 + 39 + **Bad approach:** Build a custom filesystem from scratch 40 + - 6-12 months of development 41 + - High risk of data corruption bugs 42 + - Reinventing solved problems 43 + - No interoperability with existing systems 44 + 45 + **Good approach:** Abstract storage behind a clean interface 46 + - Support multiple proven filesystems 47 + - Reuse battle-tested code 48 + - Enable multi-boot configurations 49 + - Focus on World-Tree's unique features 50 + 51 + ### The Philosophy 52 + 53 + AethelOS embraces **pragmatic abstraction**: 54 + - Use existing filesystems as storage layers 55 + - Build unique features on top (queries, versioning, metadata) 56 + - Enable interoperability (install on Linux/Windows partitions) 57 + - Keep options open (can add custom FS later if needed) 58 + 59 + **Git proves this works:** Git doesn't have its own filesystem. It works brilliantly on ext4, NTFS, APFS, FAT32, and even network filesystems. Git's innovation is its *model* (content-addressing, versioning), not its storage layer. 60 + 61 + **AethelOS follows the same pattern:** World-Tree's innovation is *queries and metadata*, not block allocation. 62 + 63 + --- 64 + 65 + ## The Fundamental Insight 66 + 67 + ### What Makes World-Tree Unique (Build This) 68 + 69 + - ✅ **Query-based interface** - `seek scrolls where creator is "Elara"` 70 + - ✅ **Rich metadata** - Essence, creator, genesis time, connections 71 + - ✅ **Content versioning** - Every version preserved, queryable by time 72 + - ✅ **Temporal queries** - See file as it existed 3 days ago 73 + - ✅ **Pruning policies** - Intelligent space management 74 + - ✅ **No path hierarchy** - Files are database objects 75 + 76 + ### What's Commodity (Reuse This) 77 + 78 + - ❌ Block allocation algorithms 79 + - ❌ Directory structures and B-trees 80 + - ❌ Journal recovery and crash consistency 81 + - ❌ Disk I/O optimization 82 + - ❌ File permissions and ownership 83 + - ❌ Inode management 84 + 85 + **ALL of World-Tree's unique features work on top of ANY filesystem.** 86 + 87 + --- 88 + 89 + ## Architecture Overview 90 + 91 + ### The Stack 92 + 93 + ``` 94 + ┌─────────────────────────────────────────────────────┐ 95 + │ Eldarin Shell / User Applications │ 96 + │ > seek scrolls where essence="Scroll" │ 97 + └─────────────────────────────────────────────────────┘ 98 + 99 + ┌─────────────────────────────────────────────────────┐ 100 + │ World-Tree Grove (Query & Versioning Layer) │ 101 + │ - Query language parser and executor │ 102 + │ - Metadata index (SQLite or custom) │ 103 + │ - Version graph management │ 104 + │ - Pruning policy engine │ 105 + └─────────────────────────────────────────────────────┘ 106 + 107 + ┌─────────────────────────────────────────────────────┐ 108 + │ Object Store (Git-like Content Addressing) │ 109 + │ - SHA-256 content addressing │ 110 + │ - Blob storage and deduplication │ 111 + │ - Reference management │ 112 + │ - Commit graph │ 113 + └─────────────────────────────────────────────────────┘ 114 + 115 + ┌─────────────────────────────────────────────────────┐ 116 + │ VFS Layer (Filesystem Abstraction) ← YOU BUILD THIS│ 117 + │ trait FileSystem { │ 118 + │ fn read(path) -> Vec<u8>; │ 119 + │ fn write(path, data); │ 120 + │ fn list_dir(path) -> Vec<Entry>; │ 121 + │ } │ 122 + └─────────────────────────────────────────────────────┘ 123 + 124 + ┌──────────────┬──────────────┬──────────────┬────────┐ 125 + │ FAT32 │ ext4 │ NTFS │ Custom │ 126 + │ Driver │ Driver │ Driver │ (future)| 127 + │ (Week 1) │ (Week 3) │ (Week 5) │ │ 128 + └──────────────┴──────────────┴──────────────┴────────┘ 129 + 130 + ┌─────────────────────────────────────────────────────┐ 131 + │ Block Device Layer │ 132 + │ - Read/write sectors │ 133 + │ - Partition table parsing │ 134 + │ - IDE/SATA/NVMe drivers │ 135 + └─────────────────────────────────────────────────────┘ 136 + 137 + ┌─────────────────────────────────────────────────────┐ 138 + │ Physical Disk Hardware │ 139 + └─────────────────────────────────────────────────────┘ 140 + ``` 141 + 142 + ### Key Insight 143 + 144 + **World-Tree doesn't call FAT32/ext4 directly.** It calls the VFS layer, which dispatches to the appropriate driver. This means: 145 + 146 + - ✅ World-Tree code is filesystem-agnostic 147 + - ✅ Easy to add new filesystem support 148 + - ✅ Can test with mock filesystems 149 + - ✅ Can switch filesystems at runtime 150 + - ✅ Can use multiple filesystems simultaneously 151 + 152 + --- 153 + 154 + ## VFS Abstraction Layer 155 + 156 + ### Core Trait 157 + 158 + ```rust 159 + // heartwood/src/vfs/mod.rs 160 + 161 + use alloc::vec::Vec; 162 + use alloc::string::String; 163 + use alloc::boxed::Box; 164 + 165 + /// Virtual File System trait - the abstraction that makes everything work 166 + pub trait FileSystem: Send + Sync { 167 + /// Filesystem type name (for debugging/logging) 168 + fn name(&self) -> &str; 169 + 170 + /// Read entire file into memory 171 + fn read(&self, path: &Path) -> Result<Vec<u8>, FsError>; 172 + 173 + /// Write entire file (create or overwrite) 174 + fn write(&self, path: &Path, data: &[u8]) -> Result<(), FsError>; 175 + 176 + /// Delete file or empty directory 177 + fn remove(&self, path: &Path) -> Result<(), FsError>; 178 + 179 + /// Create directory (and parents if needed) 180 + fn create_dir(&self, path: &Path) -> Result<(), FsError>; 181 + 182 + /// List directory contents 183 + fn read_dir(&self, path: &Path) -> Result<Vec<DirEntry>, FsError>; 184 + 185 + /// Get file metadata 186 + fn stat(&self, path: &Path) -> Result<FileStat, FsError>; 187 + 188 + /// Check if path exists 189 + fn exists(&self, path: &Path) -> bool { 190 + self.stat(path).is_ok() 191 + } 192 + 193 + /// Sync all pending writes to disk 194 + fn sync(&self) -> Result<(), FsError>; 195 + } 196 + 197 + /// Path type (Unix-style, even on filesystems that use backslashes) 198 + #[derive(Debug, Clone, PartialEq, Eq)] 199 + pub struct Path { 200 + inner: String, 201 + } 202 + 203 + impl Path { 204 + pub fn new(s: &str) -> Self { 205 + // Normalize to forward slashes 206 + Self { inner: s.replace('\\', "/") } 207 + } 208 + 209 + pub fn as_str(&self) -> &str { 210 + &self.inner 211 + } 212 + 213 + pub fn join(&self, component: &str) -> Self { 214 + Self::new(&format!("{}/{}", self.inner.trim_end_matches('/'), component)) 215 + } 216 + } 217 + 218 + /// Directory entry 219 + #[derive(Debug, Clone)] 220 + pub struct DirEntry { 221 + pub name: String, 222 + pub is_dir: bool, 223 + } 224 + 225 + /// File metadata 226 + #[derive(Debug, Clone)] 227 + pub struct FileStat { 228 + pub size: u64, 229 + pub is_dir: bool, 230 + pub created: Option<u64>, // Unix timestamp 231 + pub modified: Option<u64>, 232 + } 233 + 234 + /// Filesystem errors 235 + #[derive(Debug, Clone, Copy, PartialEq, Eq)] 236 + pub enum FsError { 237 + NotFound, 238 + AlreadyExists, 239 + PermissionDenied, 240 + NotADirectory, 241 + IsADirectory, 242 + InvalidPath, 243 + IoError, 244 + OutOfSpace, 245 + ReadOnly, 246 + } 247 + ``` 248 + 249 + ### VFS Manager 250 + 251 + ```rust 252 + // heartwood/src/vfs/manager.rs 253 + 254 + use alloc::collections::BTreeMap; 255 + use alloc::boxed::Box; 256 + 257 + /// Manages multiple mounted filesystems 258 + pub struct VfsManager { 259 + filesystems: BTreeMap<String, Box<dyn FileSystem>>, 260 + } 261 + 262 + impl VfsManager { 263 + pub fn new() -> Self { 264 + Self { 265 + filesystems: BTreeMap::new(), 266 + } 267 + } 268 + 269 + /// Mount a filesystem at a mount point 270 + pub fn mount(&mut self, mount_point: &str, fs: Box<dyn FileSystem>) -> Result<(), VfsError> { 271 + if self.filesystems.contains_key(mount_point) { 272 + return Err(VfsError::AlreadyMounted); 273 + } 274 + 275 + crate::println!("◈ Mounting {} at /{}", fs.name(), mount_point); 276 + self.filesystems.insert(mount_point.to_string(), fs); 277 + Ok(()) 278 + } 279 + 280 + /// Unmount a filesystem 281 + pub fn unmount(&mut self, mount_point: &str) -> Result<(), VfsError> { 282 + self.filesystems.remove(mount_point) 283 + .map(|_| ()) 284 + .ok_or(VfsError::NotMounted) 285 + } 286 + 287 + /// Get filesystem by mount point 288 + pub fn get(&self, mount_point: &str) -> Option<&dyn FileSystem> { 289 + self.filesystems.get(mount_point).map(|b| &**b) 290 + } 291 + 292 + /// Get mutable filesystem reference 293 + pub fn get_mut(&mut self, mount_point: &str) -> Option<&mut dyn FileSystem> { 294 + self.filesystems.get_mut(mount_point).map(|b| &mut **b) 295 + } 296 + 297 + /// List all mount points 298 + pub fn mounts(&self) -> Vec<&str> { 299 + self.filesystems.keys().map(|s| s.as_str()).collect() 300 + } 301 + 302 + /// Resolve a path to (mount_point, filesystem, relative_path) 303 + pub fn resolve(&self, path: &Path) -> Option<(&str, &dyn FileSystem, Path)> { 304 + let path_str = path.as_str().trim_start_matches('/'); 305 + 306 + // Find longest matching mount point 307 + for (mount_point, fs) in self.filesystems.iter().rev() { 308 + if path_str.starts_with(mount_point) { 309 + let relative = path_str.strip_prefix(mount_point) 310 + .unwrap_or("") 311 + .trim_start_matches('/'); 312 + return Some((mount_point, &**fs, Path::new(relative))); 313 + } 314 + } 315 + 316 + None 317 + } 318 + } 319 + 320 + /// Global VFS instance (initialized during boot) 321 + static VFS: InterruptSafeLock<Option<VfsManager>> = InterruptSafeLock::new(None); 322 + 323 + pub fn init() { 324 + *VFS.lock() = Some(VfsManager::new()); 325 + crate::println!("◈ VFS layer initialized"); 326 + } 327 + 328 + pub fn vfs() -> &'static InterruptSafeLock<Option<VfsManager>> { 329 + &VFS 330 + } 331 + ``` 332 + 333 + --- 334 + 335 + ## Filesystem Implementations 336 + 337 + ### 1. FAT32 - The Foundation (Week 1) 338 + 339 + **Why start here:** 340 + - ✅ Simple specification (easy to understand) 341 + - ✅ Universal compatibility (every OS can read it) 342 + - ✅ Pure Rust crate available (`fatfs`) 343 + - ✅ Good for boot media (USB sticks) 344 + - ✅ No complex features (no journaling, permissions, etc.) 345 + 346 + **Implementation:** 347 + 348 + ```rust 349 + // heartwood/src/vfs/fat32.rs 350 + 351 + use fatfs::{FileSystem as FatFileSystem, FsOptions}; 352 + 353 + pub struct Fat32 { 354 + fs: FatFileSystem<BlockDevice>, 355 + read_only: bool, 356 + } 357 + 358 + impl Fat32 { 359 + pub fn new(device: BlockDevice) -> Result<Self, FsError> { 360 + let fs = FatFileSystem::new(device, FsOptions::new()) 361 + .map_err(|_| FsError::IoError)?; 362 + 363 + Ok(Self { 364 + fs, 365 + read_only: false, 366 + }) 367 + } 368 + } 369 + 370 + impl FileSystem for Fat32 { 371 + fn name(&self) -> &str { 372 + "FAT32" 373 + } 374 + 375 + fn read(&self, path: &Path) -> Result<Vec<u8>, FsError> { 376 + let root_dir = self.fs.root_dir(); 377 + let mut file = root_dir.open_file(path.as_str()) 378 + .map_err(|_| FsError::NotFound)?; 379 + 380 + let mut buf = Vec::new(); 381 + file.read_to_end(&mut buf) 382 + .map_err(|_| FsError::IoError)?; 383 + 384 + Ok(buf) 385 + } 386 + 387 + fn write(&self, path: &Path, data: &[u8]) -> Result<(), FsError> { 388 + if self.read_only { 389 + return Err(FsError::ReadOnly); 390 + } 391 + 392 + let root_dir = self.fs.root_dir(); 393 + let mut file = root_dir.create_file(path.as_str()) 394 + .map_err(|_| FsError::IoError)?; 395 + 396 + file.write_all(data) 397 + .map_err(|_| FsError::IoError)?; 398 + 399 + Ok(()) 400 + } 401 + 402 + fn read_dir(&self, path: &Path) -> Result<Vec<DirEntry>, FsError> { 403 + let root_dir = self.fs.root_dir(); 404 + let dir = if path.as_str().is_empty() || path.as_str() == "/" { 405 + root_dir 406 + } else { 407 + root_dir.open_dir(path.as_str()) 408 + .map_err(|_| FsError::NotFound)? 409 + }; 410 + 411 + let mut entries = Vec::new(); 412 + for entry in dir.iter() { 413 + let entry = entry.map_err(|_| FsError::IoError)?; 414 + entries.push(DirEntry { 415 + name: entry.file_name(), 416 + is_dir: entry.is_dir(), 417 + }); 418 + } 419 + 420 + Ok(entries) 421 + } 422 + 423 + fn stat(&self, path: &Path) -> Result<FileStat, FsError> { 424 + let root_dir = self.fs.root_dir(); 425 + let file = root_dir.open_file(path.as_str()) 426 + .map_err(|_| FsError::NotFound)?; 427 + 428 + Ok(FileStat { 429 + size: file.len(), 430 + is_dir: false, 431 + created: None, // FAT32 has timestamps but we'll skip for now 432 + modified: None, 433 + }) 434 + } 435 + 436 + fn create_dir(&self, path: &Path) -> Result<(), FsError> { 437 + if self.read_only { 438 + return Err(FsError::ReadOnly); 439 + } 440 + 441 + let root_dir = self.fs.root_dir(); 442 + root_dir.create_dir(path.as_str()) 443 + .map_err(|_| FsError::IoError)?; 444 + 445 + Ok(()) 446 + } 447 + 448 + fn remove(&self, path: &Path) -> Result<(), FsError> { 449 + if self.read_only { 450 + return Err(FsError::ReadOnly); 451 + } 452 + 453 + let root_dir = self.fs.root_dir(); 454 + root_dir.remove(path.as_str()) 455 + .map_err(|_| FsError::NotFound)?; 456 + 457 + Ok(()) 458 + } 459 + 460 + fn sync(&self) -> Result<(), FsError> { 461 + // FAT32 crate handles sync automatically 462 + Ok(()) 463 + } 464 + } 465 + ``` 466 + 467 + **Dependencies:** 468 + ```toml 469 + [dependencies] 470 + fatfs = { version = "0.4", default-features = false } 471 + ``` 472 + 473 + **Time estimate:** 1 week (including block device integration) 474 + 475 + --- 476 + 477 + ### 2. ext4 - Linux Compatibility (Week 3-4) 478 + 479 + **Why ext4:** 480 + - ✅ Most common Linux filesystem 481 + - ✅ Journaling (crash-safe) 482 + - ✅ Good performance 483 + - ✅ Can install AethelOS on existing Linux partitions 484 + 485 + **Implementation options:** 486 + 487 + **Option A: Port lwext4 (Recommended)** 488 + - Small C library (~15,000 lines) 489 + - BSD license 490 + - Supports ext2/ext3/ext4 491 + - Well-tested 492 + 493 + **Option B: Use ext4-rs (If mature enough)** 494 + - Pure Rust implementation 495 + - Still in development 496 + - May not support all ext4 features yet 497 + 498 + **Skeleton:** 499 + 500 + ```rust 501 + // heartwood/src/vfs/ext4.rs 502 + 503 + // Using lwext4 via FFI (to be implemented) 504 + pub struct Ext4 { 505 + // lwext4 file descriptor 506 + fs: ext4_sys::Ext4FileSystem, 507 + } 508 + 509 + impl FileSystem for Ext4 { 510 + fn name(&self) -> &str { 511 + "ext4" 512 + } 513 + 514 + fn read(&self, path: &Path) -> Result<Vec<u8>, FsError> { 515 + // Call lwext4 functions via FFI 516 + todo!("Implement ext4 read") 517 + } 518 + 519 + // ... similar pattern to FAT32 520 + } 521 + ``` 522 + 523 + **Time estimate:** 2-4 weeks (including lwext4 integration and testing) 524 + 525 + --- 526 + 527 + ### 3. NTFS - Windows Interop (Week 5-6) 528 + 529 + **Why NTFS:** 530 + - ✅ Read Windows partitions 531 + - ✅ Share data with Windows dual-boot 532 + - ✅ Access Documents, Pictures, etc. from AethelOS 533 + 534 + **Implementation:** 535 + 536 + ```rust 537 + // heartwood/src/vfs/ntfs.rs 538 + 539 + use ntfs::{Ntfs, NtfsFile}; 540 + 541 + pub struct NtfsFs { 542 + ntfs: Ntfs, 543 + read_only: bool, // Write support is complex, start read-only 544 + } 545 + 546 + impl FileSystem for NtfsFs { 547 + fn name(&self) -> &str { 548 + "NTFS" 549 + } 550 + 551 + fn read(&self, path: &Path) -> Result<Vec<u8>, FsError> { 552 + // Use ntfs crate to read file 553 + todo!("Implement NTFS read") 554 + } 555 + 556 + fn write(&self, _path: &Path, _data: &[u8]) -> Result<(), FsError> { 557 + // Read-only for now 558 + Err(FsError::ReadOnly) 559 + } 560 + 561 + // ... implement other methods 562 + } 563 + ``` 564 + 565 + **Dependencies:** 566 + ```toml 567 + [dependencies] 568 + ntfs = { version = "0.4", default-features = false } 569 + ``` 570 + 571 + **Time estimate:** 1 week for read-only, 4+ weeks for write support 572 + 573 + --- 574 + 575 + ### 4. Mock Filesystem - Testing (Immediate) 576 + 577 + ```rust 578 + // heartwood/src/vfs/mock.rs 579 + 580 + use alloc::collections::BTreeMap; 581 + 582 + /// In-memory filesystem for testing 583 + pub struct MockFs { 584 + files: BTreeMap<String, Vec<u8>>, 585 + dirs: BTreeMap<String, ()>, 586 + } 587 + 588 + impl MockFs { 589 + pub fn new() -> Self { 590 + let mut fs = Self { 591 + files: BTreeMap::new(), 592 + dirs: BTreeMap::new(), 593 + }; 594 + fs.dirs.insert("/".to_string(), ()); 595 + fs 596 + } 597 + } 598 + 599 + impl FileSystem for MockFs { 600 + fn name(&self) -> &str { 601 + "MockFS" 602 + } 603 + 604 + fn read(&self, path: &Path) -> Result<Vec<u8>, FsError> { 605 + self.files.get(path.as_str()) 606 + .cloned() 607 + .ok_or(FsError::NotFound) 608 + } 609 + 610 + fn write(&self, path: &Path, data: &[u8]) -> Result<(), FsError> { 611 + self.files.insert(path.as_str().to_string(), data.to_vec()); 612 + Ok(()) 613 + } 614 + 615 + fn read_dir(&self, path: &Path) -> Result<Vec<DirEntry>, FsError> { 616 + let prefix = format!("{}/", path.as_str().trim_end_matches('/')); 617 + let mut entries = Vec::new(); 618 + 619 + for key in self.files.keys() { 620 + if key.starts_with(&prefix) { 621 + let name = key.strip_prefix(&prefix).unwrap(); 622 + if !name.contains('/') { 623 + entries.push(DirEntry { 624 + name: name.to_string(), 625 + is_dir: false, 626 + }); 627 + } 628 + } 629 + } 630 + 631 + Ok(entries) 632 + } 633 + 634 + fn stat(&self, path: &Path) -> Result<FileStat, FsError> { 635 + if let Some(data) = self.files.get(path.as_str()) { 636 + Ok(FileStat { 637 + size: data.len() as u64, 638 + is_dir: false, 639 + created: None, 640 + modified: None, 641 + }) 642 + } else if self.dirs.contains_key(path.as_str()) { 643 + Ok(FileStat { 644 + size: 0, 645 + is_dir: true, 646 + created: None, 647 + modified: None, 648 + }) 649 + } else { 650 + Err(FsError::NotFound) 651 + } 652 + } 653 + 654 + fn create_dir(&self, path: &Path) -> Result<(), FsError> { 655 + self.dirs.insert(path.as_str().to_string(), ()); 656 + Ok(()) 657 + } 658 + 659 + fn remove(&self, path: &Path) -> Result<(), FsError> { 660 + self.files.remove(path.as_str()) 661 + .ok_or(FsError::NotFound)?; 662 + Ok(()) 663 + } 664 + 665 + fn sync(&self) -> Result<(), FsError> { 666 + Ok(()) 667 + } 668 + } 669 + ``` 670 + 671 + **Usage in tests:** 672 + 673 + ```rust 674 + #[test] 675 + fn test_world_tree_storage() { 676 + let mock_fs = MockFs::new(); 677 + let tree = WorldTree::new(Box::new(mock_fs)); 678 + 679 + let hash = tree.store_scroll(b"Test content", "TestUser"); 680 + let data = tree.retrieve(&hash).unwrap(); 681 + 682 + assert_eq!(data, b"Test content"); 683 + } 684 + ``` 685 + 686 + --- 687 + 688 + ## Integration with World-Tree 689 + 690 + ### Object Store on VFS 691 + 692 + ```rust 693 + // groves/world-tree_grove/src/object_store.rs 694 + 695 + use sha2::{Sha256, Digest}; 696 + use crate::vfs::FileSystem; 697 + 698 + pub type Hash = [u8; 32]; 699 + 700 + pub struct ObjectStore { 701 + fs: Box<dyn FileSystem>, 702 + root: String, // e.g., "/world-tree/objects" 703 + } 704 + 705 + impl ObjectStore { 706 + pub fn new(fs: Box<dyn FileSystem>, root: &str) -> Self { 707 + Self { 708 + fs, 709 + root: root.to_string(), 710 + } 711 + } 712 + 713 + /// Store a blob (Git-like) 714 + pub fn write_blob(&mut self, data: &[u8]) -> Result<Hash, StoreError> { 715 + // Compute SHA-256 716 + let hash: Hash = Sha256::digest(data).into(); 717 + 718 + // Git-like path: objects/ab/cd1234... 719 + let hash_hex = hex::encode(hash); 720 + let path = format!("{}/{}/{}", 721 + self.root, 722 + &hash_hex[0..2], 723 + &hash_hex[2..] 724 + ); 725 + 726 + // Ensure directory exists 727 + let dir = format!("{}/{}", self.root, &hash_hex[0..2]); 728 + let _ = self.fs.create_dir(&Path::new(&dir)); 729 + 730 + // Write file 731 + self.fs.write(&Path::new(&path), data)?; 732 + 733 + Ok(hash) 734 + } 735 + 736 + /// Read a blob 737 + pub fn read_blob(&self, hash: &Hash) -> Result<Vec<u8>, StoreError> { 738 + let hash_hex = hex::encode(hash); 739 + let path = format!("{}/{}/{}", 740 + self.root, 741 + &hash_hex[0..2], 742 + &hash_hex[2..] 743 + ); 744 + 745 + self.fs.read(&Path::new(&path)) 746 + .map_err(|_| StoreError::NotFound) 747 + } 748 + 749 + /// Check if blob exists 750 + pub fn has_blob(&self, hash: &Hash) -> bool { 751 + let hash_hex = hex::encode(hash); 752 + let path = format!("{}/{}/{}", 753 + self.root, 754 + &hash_hex[0..2], 755 + &hash_hex[2..] 756 + ); 757 + 758 + self.fs.exists(&Path::new(&path)) 759 + } 760 + } 761 + ``` 762 + 763 + ### World-Tree on VFS 764 + 765 + ```rust 766 + // groves/world-tree_grove/src/lib.rs 767 + 768 + pub struct WorldTree { 769 + object_store: ObjectStore, 770 + metadata_db: MetadataDb, // Could be SQLite or custom 771 + } 772 + 773 + impl WorldTree { 774 + /// Create World-Tree on any filesystem 775 + pub fn new(fs: Box<dyn FileSystem>) -> Result<Self, TreeError> { 776 + // Initialize object store 777 + let object_store = ObjectStore::new(fs.clone(), "/world-tree/objects"); 778 + 779 + // Initialize metadata database 780 + let metadata_db = MetadataDb::new(fs, "/world-tree/metadata.db")?; 781 + 782 + Ok(Self { 783 + object_store, 784 + metadata_db, 785 + }) 786 + } 787 + 788 + /// Store a scroll (file with metadata) 789 + pub fn store_scroll(&mut self, content: &[u8], creator: &str) -> Result<Hash, TreeError> { 790 + // Store content (Git-like) 791 + let hash = self.object_store.write_blob(content)?; 792 + 793 + // Store metadata 794 + self.metadata_db.insert(hash, Metadata { 795 + essence: "Scroll".to_string(), 796 + creator: creator.to_string(), 797 + genesis_time: now(), 798 + connections: vec![], 799 + })?; 800 + 801 + Ok(hash) 802 + } 803 + 804 + /// Query interface 805 + pub fn seek(&self) -> QueryBuilder { 806 + QueryBuilder::new(&self.metadata_db) 807 + } 808 + } 809 + ``` 810 + 811 + --- 812 + 813 + ## Implementation Phases 814 + 815 + ### Phase 1: VFS Abstraction (Week 1) 816 + 817 + **Goals:** 818 + - ✅ Define VFS trait 819 + - ✅ Implement VfsManager 820 + - ✅ Create MockFs for testing 821 + - ✅ Write comprehensive tests 822 + 823 + **Deliverables:** 824 + - `heartwood/src/vfs/mod.rs` - Core trait 825 + - `heartwood/src/vfs/manager.rs` - VFS manager 826 + - `heartwood/src/vfs/mock.rs` - Mock filesystem 827 + - `heartwood/src/vfs/tests.rs` - Test suite 828 + 829 + **Success criteria:** 830 + - All VFS operations work with MockFs 831 + - Can mount/unmount filesystems 832 + - Path resolution works correctly 833 + 834 + --- 835 + 836 + ### Phase 2: FAT32 Support (Week 2) 837 + 838 + **Goals:** 839 + - ✅ Integrate `fatfs` crate 840 + - ✅ Implement FileSystem trait for FAT32 841 + - ✅ Add block device abstraction 842 + - ✅ Test with real FAT32 image 843 + 844 + **Deliverables:** 845 + - `heartwood/src/vfs/fat32.rs` - FAT32 driver 846 + - `heartwood/src/block_device.rs` - Block device interface 847 + - FAT32 test image for QEMU 848 + 849 + **Success criteria:** 850 + - Can read files from FAT32 partition 851 + - Can write files to FAT32 partition 852 + - Can list directories 853 + - Works in QEMU with test disk image 854 + 855 + --- 856 + 857 + ### Phase 3: World-Tree Integration (Week 2-3) 858 + 859 + **Goals:** 860 + - ✅ Port Object Store to use VFS 861 + - ✅ Create metadata database on VFS 862 + - ✅ Implement Git-like storage layout 863 + - ✅ Add Eldarin commands for testing 864 + 865 + **Deliverables:** 866 + - `groves/world-tree_grove/src/object_store.rs` 867 + - `groves/world-tree_grove/src/metadata.rs` 868 + - Eldarin commands: `wt-store`, `wt-read`, `wt-seek` 869 + 870 + **Success criteria:** 871 + - Can store blobs on FAT32 872 + - Can retrieve blobs by hash 873 + - Metadata persists across reboots 874 + - Simple queries work 875 + 876 + --- 877 + 878 + ### Phase 4: ext4 Support (Week 4-5) 879 + 880 + **Goals:** 881 + - ✅ Port lwext4 or integrate ext4-rs 882 + - ✅ Implement FileSystem trait for ext4 883 + - ✅ Test journaling and crash recovery 884 + - ✅ Benchmark performance vs FAT32 885 + 886 + **Deliverables:** 887 + - `heartwood/src/vfs/ext4.rs` - ext4 driver 888 + - ext4 test images 889 + - Performance benchmarks 890 + 891 + **Success criteria:** 892 + - Can read/write ext4 partitions 893 + - Journaling works (test with forced crashes) 894 + - Performance is acceptable 895 + - Can install AethelOS on Linux partition 896 + 897 + --- 898 + 899 + ### Phase 5: NTFS Support (Week 6) 900 + 901 + **Goals:** 902 + - ✅ Integrate `ntfs` crate 903 + - ✅ Implement read-only NTFS support 904 + - ✅ Test with Windows partition 905 + - ✅ Document limitations 906 + 907 + **Deliverables:** 908 + - `heartwood/src/vfs/ntfs.rs` - NTFS driver (read-only) 909 + - NTFS test images 910 + - Interop guide 911 + 912 + **Success criteria:** 913 + - Can read files from Windows partition 914 + - Can list Windows directories 915 + - Can access Documents, Pictures, etc. 916 + - Clearly documented that write is not yet supported 917 + 918 + --- 919 + 920 + ### Phase 6: Multi-Mount Setup (Week 7) 921 + 922 + **Goals:** 923 + - ✅ Support multiple mounted filesystems 924 + - ✅ Partition table parsing (MBR/GPT) 925 + - ✅ Auto-detect filesystem types 926 + - ✅ Create multi-boot examples 927 + 928 + **Deliverables:** 929 + - `heartwood/src/partition.rs` - Partition parsing 930 + - Auto-mount logic 931 + - Multi-boot guide 932 + - Example disk layouts 933 + 934 + **Success criteria:** 935 + - Can mount multiple partitions 936 + - Filesystem auto-detection works 937 + - Can dual-boot with Linux/Windows 938 + - World-Tree can span multiple filesystems 939 + 940 + --- 941 + 942 + ## Multi-Boot Support 943 + 944 + ### Example Disk Layout 945 + 946 + ``` 947 + /dev/sda (1TB SSD) 948 + ├─ sda1: EFI System Partition (FAT32, 512MB) 949 + │ └─ /EFI/BOOT/BOOTX64.EFI ← GRUB bootloader 950 + │ └─ /aethelos/heartwood.bin ← AethelOS kernel 951 + 952 + ├─ sda2: AethelOS Root (ext4, 50GB) 953 + │ └─ /world-tree/ ← World-Tree storage 954 + │ ├─ objects/ab/cd1234... ← Content blobs 955 + │ └─ metadata.db ← Query index 956 + 957 + ├─ sda3: Linux Root (ext4, 50GB) 958 + │ └─ /home/user/... ← Linux files 959 + 960 + └─ sda4: Shared Data (NTFS, 800GB) 961 + └─ /Documents/ ← Shared with Windows 962 + └─ /Pictures/ 963 + └─ /Projects/ 964 + ``` 965 + 966 + ### Boot Configuration 967 + 968 + ```bash 969 + # /boot/grub/grub.cfg 970 + 971 + menuentry "AethelOS" { 972 + insmod gzio 973 + insmod part_gpt 974 + insmod ext4 975 + 976 + set root='hd0,gpt2' # AethelOS partition 977 + 978 + multiboot2 /boot/aethelos/heartwood.bin 979 + boot 980 + } 981 + 982 + menuentry "Linux" { 983 + set root='hd0,gpt3' 984 + linux /boot/vmlinuz root=/dev/sda3 985 + initrd /boot/initrd.img 986 + } 987 + ``` 988 + 989 + ### Mount Configuration in AethelOS 990 + 991 + ```rust 992 + // During boot 993 + fn init_filesystems() -> Result<(), FsError> { 994 + let mut vfs = VfsManager::new(); 995 + 996 + // Parse partition table 997 + let disk = BlockDevice::new(0)?; // Primary disk 998 + let partitions = parse_gpt(&disk)?; 999 + 1000 + // Mount EFI partition (FAT32) 1001 + let efi_part = partitions.get(0).unwrap(); 1002 + let efi_fs = Fat32::new(efi_part.clone())?; 1003 + vfs.mount("boot", Box::new(efi_fs))?; 1004 + 1005 + // Mount AethelOS root (ext4) 1006 + let root_part = partitions.get(1).unwrap(); 1007 + let root_fs = Ext4::new(root_part.clone())?; 1008 + vfs.mount("root", Box::new(root_fs))?; 1009 + 1010 + // Mount shared data (NTFS, read-only for now) 1011 + let data_part = partitions.get(3).unwrap(); 1012 + let data_fs = NtfsFs::new(data_part.clone())?; 1013 + vfs.mount("data", Box::new(data_fs))?; 1014 + 1015 + // Initialize World-Tree on root filesystem 1016 + let root_fs = vfs.get("root").unwrap(); 1017 + let tree = WorldTree::new(Box::new(root_fs))?; 1018 + 1019 + crate::println!("◈ Filesystems mounted:"); 1020 + crate::println!(" /boot → FAT32 (EFI partition)"); 1021 + crate::println!(" /root → ext4 (AethelOS root)"); 1022 + crate::println!(" /data → NTFS (Shared data, read-only)"); 1023 + 1024 + Ok(()) 1025 + } 1026 + ``` 1027 + 1028 + --- 1029 + 1030 + ## Testing Strategy 1031 + 1032 + ### Unit Tests 1033 + 1034 + ```rust 1035 + #[cfg(test)] 1036 + mod tests { 1037 + use super::*; 1038 + 1039 + #[test] 1040 + fn test_mock_fs_read_write() { 1041 + let mut fs = MockFs::new(); 1042 + let path = Path::new("/test.txt"); 1043 + 1044 + fs.write(&path, b"Hello, World!").unwrap(); 1045 + let data = fs.read(&path).unwrap(); 1046 + 1047 + assert_eq!(data, b"Hello, World!"); 1048 + } 1049 + 1050 + #[test] 1051 + fn test_vfs_mounting() { 1052 + let mut vfs = VfsManager::new(); 1053 + let mock_fs = Box::new(MockFs::new()); 1054 + 1055 + vfs.mount("test", mock_fs).unwrap(); 1056 + 1057 + assert!(vfs.get("test").is_some()); 1058 + assert_eq!(vfs.mounts().len(), 1); 1059 + } 1060 + 1061 + #[test] 1062 + fn test_path_resolution() { 1063 + let mut vfs = VfsManager::new(); 1064 + vfs.mount("root", Box::new(MockFs::new())).unwrap(); 1065 + 1066 + let (mount, fs, rel_path) = vfs.resolve(&Path::new("/root/foo/bar.txt")).unwrap(); 1067 + 1068 + assert_eq!(mount, "root"); 1069 + assert_eq!(rel_path.as_str(), "foo/bar.txt"); 1070 + } 1071 + } 1072 + ``` 1073 + 1074 + ### Integration Tests 1075 + 1076 + ```rust 1077 + #[test] 1078 + fn test_world_tree_on_fat32() { 1079 + // Create FAT32 image 1080 + let img_path = create_test_fat32_image(); 1081 + let device = BlockDevice::from_file(&img_path).unwrap(); 1082 + let fs = Fat32::new(device).unwrap(); 1083 + 1084 + // Create World-Tree 1085 + let mut tree = WorldTree::new(Box::new(fs)).unwrap(); 1086 + 1087 + // Store a scroll 1088 + let hash = tree.store_scroll(b"Test content", "Tester").unwrap(); 1089 + 1090 + // Retrieve it 1091 + let data = tree.read_blob(&hash).unwrap(); 1092 + assert_eq!(data, b"Test content"); 1093 + 1094 + // Query by metadata 1095 + let results = tree.seek() 1096 + .where_creator("Tester") 1097 + .execute() 1098 + .unwrap(); 1099 + 1100 + assert_eq!(results.len(), 1); 1101 + assert_eq!(results[0], hash); 1102 + } 1103 + ``` 1104 + 1105 + ### Performance Tests 1106 + 1107 + ```rust 1108 + #[test] 1109 + fn bench_object_store_throughput() { 1110 + let mock_fs = Box::new(MockFs::new()); 1111 + let mut store = ObjectStore::new(mock_fs, "/objects"); 1112 + 1113 + let start = now(); 1114 + 1115 + // Write 1000 blobs 1116 + for i in 0..1000 { 1117 + let data = format!("Blob {}", i).into_bytes(); 1118 + store.write_blob(&data).unwrap(); 1119 + } 1120 + 1121 + let elapsed = now() - start; 1122 + let throughput = 1000.0 / elapsed.as_secs_f64(); 1123 + 1124 + crate::println!("Object store throughput: {:.2} blobs/sec", throughput); 1125 + } 1126 + ``` 1127 + 1128 + --- 1129 + 1130 + ## Timeline and Milestones 1131 + 1132 + ### Week 1: VFS Foundation 1133 + - **Day 1-2:** Design and implement VFS trait 1134 + - **Day 3-4:** Implement VfsManager 1135 + - **Day 5:** Create MockFs and write tests 1136 + - **Milestone:** VFS abstraction complete, all tests pass 1137 + 1138 + ### Week 2: FAT32 Support 1139 + - **Day 1-2:** Block device abstraction 1140 + - **Day 3-4:** Integrate fatfs crate 1141 + - **Day 5:** Testing with FAT32 images 1142 + - **Milestone:** Can read/write FAT32 in QEMU 1143 + 1144 + ### Week 3: World-Tree Integration 1145 + - **Day 1-2:** Port Object Store to VFS 1146 + - **Day 3-4:** Implement metadata storage 1147 + - **Day 5:** Add Eldarin commands 1148 + - **Milestone:** World-Tree works on FAT32 1149 + 1150 + ### Week 4-5: ext4 Support 1151 + - **Week 4:** Port lwext4, implement trait 1152 + - **Week 5:** Testing, journaling, crash recovery 1153 + - **Milestone:** Can use ext4 partitions 1154 + 1155 + ### Week 6: NTFS Support 1156 + - **Day 1-3:** Integrate ntfs crate 1157 + - **Day 4-5:** Testing with Windows partitions 1158 + - **Milestone:** Read-only NTFS works 1159 + 1160 + ### Week 7: Multi-Boot 1161 + - **Day 1-2:** Partition table parsing 1162 + - **Day 3-4:** Auto-mount logic 1163 + - **Day 5:** Documentation and examples 1164 + - **Milestone:** Can dual-boot with Linux/Windows 1165 + 1166 + --- 1167 + 1168 + ## Success Criteria 1169 + 1170 + ### Functional Requirements 1171 + 1172 + - ✅ VFS trait is clean and extensible 1173 + - ✅ FAT32 read/write works reliably 1174 + - ✅ ext4 read/write works with journaling 1175 + - ✅ NTFS read-only works 1176 + - ✅ World-Tree stores objects on any filesystem 1177 + - ✅ Can mount multiple filesystems simultaneously 1178 + - ✅ Filesystem auto-detection works 1179 + - ✅ Can dual-boot with other OSes 1180 + 1181 + ### Performance Requirements 1182 + 1183 + - ✅ Object store: >100 blobs/sec write throughput 1184 + - ✅ Metadata queries: <100ms for typical queries 1185 + - ✅ File read: Within 2x of Linux performance 1186 + - ✅ File write: Within 2x of Linux performance 1187 + 1188 + ### Quality Requirements 1189 + 1190 + - ✅ All unit tests pass 1191 + - ✅ All integration tests pass 1192 + - ✅ No data corruption under normal use 1193 + - ✅ Graceful handling of disk full 1194 + - ✅ Proper error messages 1195 + - ✅ Comprehensive documentation 1196 + 1197 + --- 1198 + 1199 + ## Future Enhancements 1200 + 1201 + ### Phase 8: Advanced Features (Post v1.0) 1202 + 1203 + **Write support for NTFS:** 1204 + - Complex but valuable for Windows interop 1205 + - Requires careful implementation (easy to corrupt) 1206 + 1207 + **Network filesystems:** 1208 + - NFS client (access Linux network shares) 1209 + - SMB/CIFS client (access Windows network shares) 1210 + 1211 + **Copy-on-Write filesystem:** 1212 + - Custom AethelFS optimized for World-Tree 1213 + - Native snapshots and versioning 1214 + - Compression and deduplication 1215 + 1216 + **FUSE support:** 1217 + - Let userspace implement filesystems 1218 + - Enable experimentation without kernel changes 1219 + 1220 + --- 1221 + 1222 + ## Philosophical Notes 1223 + 1224 + ### Why This Approach Works 1225 + 1226 + **Git proves it:** Git doesn't have a filesystem. It uses whatever is available. Yet it's the most successful version control system ever. 1227 + 1228 + **Databases prove it:** PostgreSQL, MySQL, SQLite—none make their own filesystems. They focus on database features. 1229 + 1230 + **Docker proves it:** Uses overlay filesystems on top of ext4/btrfs/whatever. Focuses on containers, not storage. 1231 + 1232 + **AethelOS follows this pattern:** Focus on what's unique (queries, versioning, metadata). Reuse proven infrastructure (filesystems). 1233 + 1234 + ### The Abstraction Principle 1235 + 1236 + > *"The World-Tree does not care whether it grows in ext4 soil or FAT32 sand. It adapts to its environment while maintaining its essential nature."* 1237 + 1238 + Good abstractions enable: 1239 + - **Flexibility** - Switch storage backends easily 1240 + - **Testing** - Use mocks for fast unit tests 1241 + - **Compatibility** - Work with existing systems 1242 + - **Evolution** - Can optimize later without breaking API 1243 + 1244 + ### Engineering Pragmatism 1245 + 1246 + **Build what differentiates you. Reuse what doesn't.** 1247 + 1248 + World-Tree's differentiation: 1249 + - Query-based interface ← Build this 1250 + - Rich metadata ← Build this 1251 + - Versioning model ← Build this 1252 + 1253 + Commodity infrastructure: 1254 + - Block allocation ← Reuse ext4 1255 + - Crash recovery ← Reuse journaling 1256 + - I/O optimization ← Reuse proven filesystems 1257 + 1258 + **Result:** Ship in 6 weeks instead of 6 months. 1259 + 1260 + --- 1261 + 1262 + ## Conclusion 1263 + 1264 + The VFS layer is **critical infrastructure** that unlocks: 1265 + - ✅ Multi-filesystem support (FAT32, ext4, NTFS) 1266 + - ✅ Interoperability (dual-boot, shared partitions) 1267 + - ✅ Flexibility (can optimize later) 1268 + - ✅ Testing (mock filesystems) 1269 + - ✅ Focus (build World-Tree features, not filesystem internals) 1270 + 1271 + **This is the right architecture.** It's not a compromise—it's **smart engineering.** 1272 + 1273 + --- 1274 + 1275 + *"The strongest trees don't resist the wind; they bend with it. So too does AethelOS adapt to the storage landscape, drawing strength from proven foundations."* 1276 + 1277 + **Status:** Ready to implement 1278 + **First step:** Phase 1 (VFS Abstraction, Week 1) 1279 + **Next review:** After FAT32 integration (Week 2) 1280 + 1281 + --- 1282 + 1283 + ## References 1284 + 1285 + - **Git Object Model:** https://git-scm.com/book/en/v2/Git-Internals-Git-Objects 1286 + - **fatfs crate:** https://crates.io/crates/fatfs 1287 + - **ntfs crate:** https://crates.io/crates/ntfs 1288 + - **lwext4:** https://github.com/gkostka/lwext4 1289 + - **VFS in Linux:** https://www.kernel.org/doc/html/latest/filesystems/vfs.html 1290 + - **World-Tree Plan:** [WORLD_TREE_PLAN.md](WORLD_TREE_PLAN.md)