+1290
docs/VFS_PLAN.md
+1290
docs/VFS_PLAN.md
···
1
+
# The Path-Keepers: AethelOS Virtual File System Architecture
2
+
3
+
**Status:** Planned
4
+
**Priority:** High
5
+
**Dependencies:** Block device driver, World-Tree design
6
+
**Estimated Timeline:** 4-6 weeks
7
+
**Version:** 1.0
8
+
**Last Updated:** January 2025
9
+
10
+
---
11
+
12
+
## Table of Contents
13
+
14
+
1. [Vision](#vision)
15
+
2. [The Fundamental Insight](#the-fundamental-insight)
16
+
3. [Architecture Overview](#architecture-overview)
17
+
4. [VFS Abstraction Layer](#vfs-abstraction-layer)
18
+
5. [Filesystem Implementations](#filesystem-implementations)
19
+
6. [Integration with World-Tree](#integration-with-world-tree)
20
+
7. [Implementation Phases](#implementation-phases)
21
+
8. [Multi-Boot Support](#multi-boot-support)
22
+
9. [Testing Strategy](#testing-strategy)
23
+
10. [Timeline and Milestones](#timeline-and-milestones)
24
+
25
+
---
26
+
27
+
## Vision
28
+
29
+
> *"The World-Tree does not care whether its roots touch ext4, FAT32, or NTFS. It draws nourishment from all soils, transforming base storage into living knowledge."*
30
+
31
+
### The Problem
32
+
33
+
World-Tree needs persistent storage for:
34
+
- Content-addressed objects (Git-like blobs)
35
+
- Metadata indices (queryable attributes)
36
+
- Commit graphs (version history)
37
+
- Pruning logs (space management)
38
+
39
+
**Bad approach:** Build a custom filesystem from scratch
40
+
- 6-12 months of development
41
+
- High risk of data corruption bugs
42
+
- Reinventing solved problems
43
+
- No interoperability with existing systems
44
+
45
+
**Good approach:** Abstract storage behind a clean interface
46
+
- Support multiple proven filesystems
47
+
- Reuse battle-tested code
48
+
- Enable multi-boot configurations
49
+
- Focus on World-Tree's unique features
50
+
51
+
### The Philosophy
52
+
53
+
AethelOS embraces **pragmatic abstraction**:
54
+
- Use existing filesystems as storage layers
55
+
- Build unique features on top (queries, versioning, metadata)
56
+
- Enable interoperability (install on Linux/Windows partitions)
57
+
- Keep options open (can add custom FS later if needed)
58
+
59
+
**Git proves this works:** Git doesn't have its own filesystem. It works brilliantly on ext4, NTFS, APFS, FAT32, and even network filesystems. Git's innovation is its *model* (content-addressing, versioning), not its storage layer.
60
+
61
+
**AethelOS follows the same pattern:** World-Tree's innovation is *queries and metadata*, not block allocation.
62
+
63
+
---
64
+
65
+
## The Fundamental Insight
66
+
67
+
### What Makes World-Tree Unique (Build This)
68
+
69
+
- ✅ **Query-based interface** - `seek scrolls where creator is "Elara"`
70
+
- ✅ **Rich metadata** - Essence, creator, genesis time, connections
71
+
- ✅ **Content versioning** - Every version preserved, queryable by time
72
+
- ✅ **Temporal queries** - See file as it existed 3 days ago
73
+
- ✅ **Pruning policies** - Intelligent space management
74
+
- ✅ **No path hierarchy** - Files are database objects
75
+
76
+
### What's Commodity (Reuse This)
77
+
78
+
- ❌ Block allocation algorithms
79
+
- ❌ Directory structures and B-trees
80
+
- ❌ Journal recovery and crash consistency
81
+
- ❌ Disk I/O optimization
82
+
- ❌ File permissions and ownership
83
+
- ❌ Inode management
84
+
85
+
**ALL of World-Tree's unique features work on top of ANY filesystem.**
86
+
87
+
---
88
+
89
+
## Architecture Overview
90
+
91
+
### The Stack
92
+
93
+
```
94
+
┌─────────────────────────────────────────────────────┐
95
+
│ Eldarin Shell / User Applications │
96
+
│ > seek scrolls where essence="Scroll" │
97
+
└─────────────────────────────────────────────────────┘
98
+
↓
99
+
┌─────────────────────────────────────────────────────┐
100
+
│ World-Tree Grove (Query & Versioning Layer) │
101
+
│ - Query language parser and executor │
102
+
│ - Metadata index (SQLite or custom) │
103
+
│ - Version graph management │
104
+
│ - Pruning policy engine │
105
+
└─────────────────────────────────────────────────────┘
106
+
↓
107
+
┌─────────────────────────────────────────────────────┐
108
+
│ Object Store (Git-like Content Addressing) │
109
+
│ - SHA-256 content addressing │
110
+
│ - Blob storage and deduplication │
111
+
│ - Reference management │
112
+
│ - Commit graph │
113
+
└─────────────────────────────────────────────────────┘
114
+
↓
115
+
┌─────────────────────────────────────────────────────┐
116
+
│ VFS Layer (Filesystem Abstraction) ← YOU BUILD THIS│
117
+
│ trait FileSystem { │
118
+
│ fn read(path) -> Vec<u8>; │
119
+
│ fn write(path, data); │
120
+
│ fn list_dir(path) -> Vec<Entry>; │
121
+
│ } │
122
+
└─────────────────────────────────────────────────────┘
123
+
↓
124
+
┌──────────────┬──────────────┬──────────────┬────────┐
125
+
│ FAT32 │ ext4 │ NTFS │ Custom │
126
+
│ Driver │ Driver │ Driver │ (future)|
127
+
│ (Week 1) │ (Week 3) │ (Week 5) │ │
128
+
└──────────────┴──────────────┴──────────────┴────────┘
129
+
↓
130
+
┌─────────────────────────────────────────────────────┐
131
+
│ Block Device Layer │
132
+
│ - Read/write sectors │
133
+
│ - Partition table parsing │
134
+
│ - IDE/SATA/NVMe drivers │
135
+
└─────────────────────────────────────────────────────┘
136
+
↓
137
+
┌─────────────────────────────────────────────────────┐
138
+
│ Physical Disk Hardware │
139
+
└─────────────────────────────────────────────────────┘
140
+
```
141
+
142
+
### Key Insight
143
+
144
+
**World-Tree doesn't call FAT32/ext4 directly.** It calls the VFS layer, which dispatches to the appropriate driver. This means:
145
+
146
+
- ✅ World-Tree code is filesystem-agnostic
147
+
- ✅ Easy to add new filesystem support
148
+
- ✅ Can test with mock filesystems
149
+
- ✅ Can switch filesystems at runtime
150
+
- ✅ Can use multiple filesystems simultaneously
151
+
152
+
---
153
+
154
+
## VFS Abstraction Layer
155
+
156
+
### Core Trait
157
+
158
+
```rust
159
+
// heartwood/src/vfs/mod.rs
160
+
161
+
use alloc::vec::Vec;
162
+
use alloc::string::String;
163
+
use alloc::boxed::Box;
164
+
165
+
/// Virtual File System trait - the abstraction that makes everything work
166
+
pub trait FileSystem: Send + Sync {
167
+
/// Filesystem type name (for debugging/logging)
168
+
fn name(&self) -> &str;
169
+
170
+
/// Read entire file into memory
171
+
fn read(&self, path: &Path) -> Result<Vec<u8>, FsError>;
172
+
173
+
/// Write entire file (create or overwrite)
174
+
fn write(&self, path: &Path, data: &[u8]) -> Result<(), FsError>;
175
+
176
+
/// Delete file or empty directory
177
+
fn remove(&self, path: &Path) -> Result<(), FsError>;
178
+
179
+
/// Create directory (and parents if needed)
180
+
fn create_dir(&self, path: &Path) -> Result<(), FsError>;
181
+
182
+
/// List directory contents
183
+
fn read_dir(&self, path: &Path) -> Result<Vec<DirEntry>, FsError>;
184
+
185
+
/// Get file metadata
186
+
fn stat(&self, path: &Path) -> Result<FileStat, FsError>;
187
+
188
+
/// Check if path exists
189
+
fn exists(&self, path: &Path) -> bool {
190
+
self.stat(path).is_ok()
191
+
}
192
+
193
+
/// Sync all pending writes to disk
194
+
fn sync(&self) -> Result<(), FsError>;
195
+
}
196
+
197
+
/// Path type (Unix-style, even on filesystems that use backslashes)
198
+
#[derive(Debug, Clone, PartialEq, Eq)]
199
+
pub struct Path {
200
+
inner: String,
201
+
}
202
+
203
+
impl Path {
204
+
pub fn new(s: &str) -> Self {
205
+
// Normalize to forward slashes
206
+
Self { inner: s.replace('\\', "/") }
207
+
}
208
+
209
+
pub fn as_str(&self) -> &str {
210
+
&self.inner
211
+
}
212
+
213
+
pub fn join(&self, component: &str) -> Self {
214
+
Self::new(&format!("{}/{}", self.inner.trim_end_matches('/'), component))
215
+
}
216
+
}
217
+
218
+
/// Directory entry
219
+
#[derive(Debug, Clone)]
220
+
pub struct DirEntry {
221
+
pub name: String,
222
+
pub is_dir: bool,
223
+
}
224
+
225
+
/// File metadata
226
+
#[derive(Debug, Clone)]
227
+
pub struct FileStat {
228
+
pub size: u64,
229
+
pub is_dir: bool,
230
+
pub created: Option<u64>, // Unix timestamp
231
+
pub modified: Option<u64>,
232
+
}
233
+
234
+
/// Filesystem errors
235
+
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
236
+
pub enum FsError {
237
+
NotFound,
238
+
AlreadyExists,
239
+
PermissionDenied,
240
+
NotADirectory,
241
+
IsADirectory,
242
+
InvalidPath,
243
+
IoError,
244
+
OutOfSpace,
245
+
ReadOnly,
246
+
}
247
+
```
248
+
249
+
### VFS Manager
250
+
251
+
```rust
252
+
// heartwood/src/vfs/manager.rs
253
+
254
+
use alloc::collections::BTreeMap;
255
+
use alloc::boxed::Box;
256
+
257
+
/// Manages multiple mounted filesystems
258
+
pub struct VfsManager {
259
+
filesystems: BTreeMap<String, Box<dyn FileSystem>>,
260
+
}
261
+
262
+
impl VfsManager {
263
+
pub fn new() -> Self {
264
+
Self {
265
+
filesystems: BTreeMap::new(),
266
+
}
267
+
}
268
+
269
+
/// Mount a filesystem at a mount point
270
+
pub fn mount(&mut self, mount_point: &str, fs: Box<dyn FileSystem>) -> Result<(), VfsError> {
271
+
if self.filesystems.contains_key(mount_point) {
272
+
return Err(VfsError::AlreadyMounted);
273
+
}
274
+
275
+
crate::println!("◈ Mounting {} at /{}", fs.name(), mount_point);
276
+
self.filesystems.insert(mount_point.to_string(), fs);
277
+
Ok(())
278
+
}
279
+
280
+
/// Unmount a filesystem
281
+
pub fn unmount(&mut self, mount_point: &str) -> Result<(), VfsError> {
282
+
self.filesystems.remove(mount_point)
283
+
.map(|_| ())
284
+
.ok_or(VfsError::NotMounted)
285
+
}
286
+
287
+
/// Get filesystem by mount point
288
+
pub fn get(&self, mount_point: &str) -> Option<&dyn FileSystem> {
289
+
self.filesystems.get(mount_point).map(|b| &**b)
290
+
}
291
+
292
+
/// Get mutable filesystem reference
293
+
pub fn get_mut(&mut self, mount_point: &str) -> Option<&mut dyn FileSystem> {
294
+
self.filesystems.get_mut(mount_point).map(|b| &mut **b)
295
+
}
296
+
297
+
/// List all mount points
298
+
pub fn mounts(&self) -> Vec<&str> {
299
+
self.filesystems.keys().map(|s| s.as_str()).collect()
300
+
}
301
+
302
+
/// Resolve a path to (mount_point, filesystem, relative_path)
303
+
pub fn resolve(&self, path: &Path) -> Option<(&str, &dyn FileSystem, Path)> {
304
+
let path_str = path.as_str().trim_start_matches('/');
305
+
306
+
// Find longest matching mount point
307
+
for (mount_point, fs) in self.filesystems.iter().rev() {
308
+
if path_str.starts_with(mount_point) {
309
+
let relative = path_str.strip_prefix(mount_point)
310
+
.unwrap_or("")
311
+
.trim_start_matches('/');
312
+
return Some((mount_point, &**fs, Path::new(relative)));
313
+
}
314
+
}
315
+
316
+
None
317
+
}
318
+
}
319
+
320
+
/// Global VFS instance (initialized during boot)
321
+
static VFS: InterruptSafeLock<Option<VfsManager>> = InterruptSafeLock::new(None);
322
+
323
+
pub fn init() {
324
+
*VFS.lock() = Some(VfsManager::new());
325
+
crate::println!("◈ VFS layer initialized");
326
+
}
327
+
328
+
pub fn vfs() -> &'static InterruptSafeLock<Option<VfsManager>> {
329
+
&VFS
330
+
}
331
+
```
332
+
333
+
---
334
+
335
+
## Filesystem Implementations
336
+
337
+
### 1. FAT32 - The Foundation (Week 1)
338
+
339
+
**Why start here:**
340
+
- ✅ Simple specification (easy to understand)
341
+
- ✅ Universal compatibility (every OS can read it)
342
+
- ✅ Pure Rust crate available (`fatfs`)
343
+
- ✅ Good for boot media (USB sticks)
344
+
- ✅ No complex features (no journaling, permissions, etc.)
345
+
346
+
**Implementation:**
347
+
348
+
```rust
349
+
// heartwood/src/vfs/fat32.rs
350
+
351
+
use fatfs::{FileSystem as FatFileSystem, FsOptions};
352
+
353
+
pub struct Fat32 {
354
+
fs: FatFileSystem<BlockDevice>,
355
+
read_only: bool,
356
+
}
357
+
358
+
impl Fat32 {
359
+
pub fn new(device: BlockDevice) -> Result<Self, FsError> {
360
+
let fs = FatFileSystem::new(device, FsOptions::new())
361
+
.map_err(|_| FsError::IoError)?;
362
+
363
+
Ok(Self {
364
+
fs,
365
+
read_only: false,
366
+
})
367
+
}
368
+
}
369
+
370
+
impl FileSystem for Fat32 {
371
+
fn name(&self) -> &str {
372
+
"FAT32"
373
+
}
374
+
375
+
fn read(&self, path: &Path) -> Result<Vec<u8>, FsError> {
376
+
let root_dir = self.fs.root_dir();
377
+
let mut file = root_dir.open_file(path.as_str())
378
+
.map_err(|_| FsError::NotFound)?;
379
+
380
+
let mut buf = Vec::new();
381
+
file.read_to_end(&mut buf)
382
+
.map_err(|_| FsError::IoError)?;
383
+
384
+
Ok(buf)
385
+
}
386
+
387
+
fn write(&self, path: &Path, data: &[u8]) -> Result<(), FsError> {
388
+
if self.read_only {
389
+
return Err(FsError::ReadOnly);
390
+
}
391
+
392
+
let root_dir = self.fs.root_dir();
393
+
let mut file = root_dir.create_file(path.as_str())
394
+
.map_err(|_| FsError::IoError)?;
395
+
396
+
file.write_all(data)
397
+
.map_err(|_| FsError::IoError)?;
398
+
399
+
Ok(())
400
+
}
401
+
402
+
fn read_dir(&self, path: &Path) -> Result<Vec<DirEntry>, FsError> {
403
+
let root_dir = self.fs.root_dir();
404
+
let dir = if path.as_str().is_empty() || path.as_str() == "/" {
405
+
root_dir
406
+
} else {
407
+
root_dir.open_dir(path.as_str())
408
+
.map_err(|_| FsError::NotFound)?
409
+
};
410
+
411
+
let mut entries = Vec::new();
412
+
for entry in dir.iter() {
413
+
let entry = entry.map_err(|_| FsError::IoError)?;
414
+
entries.push(DirEntry {
415
+
name: entry.file_name(),
416
+
is_dir: entry.is_dir(),
417
+
});
418
+
}
419
+
420
+
Ok(entries)
421
+
}
422
+
423
+
fn stat(&self, path: &Path) -> Result<FileStat, FsError> {
424
+
let root_dir = self.fs.root_dir();
425
+
let file = root_dir.open_file(path.as_str())
426
+
.map_err(|_| FsError::NotFound)?;
427
+
428
+
Ok(FileStat {
429
+
size: file.len(),
430
+
is_dir: false,
431
+
created: None, // FAT32 has timestamps but we'll skip for now
432
+
modified: None,
433
+
})
434
+
}
435
+
436
+
fn create_dir(&self, path: &Path) -> Result<(), FsError> {
437
+
if self.read_only {
438
+
return Err(FsError::ReadOnly);
439
+
}
440
+
441
+
let root_dir = self.fs.root_dir();
442
+
root_dir.create_dir(path.as_str())
443
+
.map_err(|_| FsError::IoError)?;
444
+
445
+
Ok(())
446
+
}
447
+
448
+
fn remove(&self, path: &Path) -> Result<(), FsError> {
449
+
if self.read_only {
450
+
return Err(FsError::ReadOnly);
451
+
}
452
+
453
+
let root_dir = self.fs.root_dir();
454
+
root_dir.remove(path.as_str())
455
+
.map_err(|_| FsError::NotFound)?;
456
+
457
+
Ok(())
458
+
}
459
+
460
+
fn sync(&self) -> Result<(), FsError> {
461
+
// FAT32 crate handles sync automatically
462
+
Ok(())
463
+
}
464
+
}
465
+
```
466
+
467
+
**Dependencies:**
468
+
```toml
469
+
[dependencies]
470
+
fatfs = { version = "0.4", default-features = false }
471
+
```
472
+
473
+
**Time estimate:** 1 week (including block device integration)
474
+
475
+
---
476
+
477
+
### 2. ext4 - Linux Compatibility (Week 3-4)
478
+
479
+
**Why ext4:**
480
+
- ✅ Most common Linux filesystem
481
+
- ✅ Journaling (crash-safe)
482
+
- ✅ Good performance
483
+
- ✅ Can install AethelOS on existing Linux partitions
484
+
485
+
**Implementation options:**
486
+
487
+
**Option A: Port lwext4 (Recommended)**
488
+
- Small C library (~15,000 lines)
489
+
- BSD license
490
+
- Supports ext2/ext3/ext4
491
+
- Well-tested
492
+
493
+
**Option B: Use ext4-rs (If mature enough)**
494
+
- Pure Rust implementation
495
+
- Still in development
496
+
- May not support all ext4 features yet
497
+
498
+
**Skeleton:**
499
+
500
+
```rust
501
+
// heartwood/src/vfs/ext4.rs
502
+
503
+
// Using lwext4 via FFI (to be implemented)
504
+
pub struct Ext4 {
505
+
// lwext4 file descriptor
506
+
fs: ext4_sys::Ext4FileSystem,
507
+
}
508
+
509
+
impl FileSystem for Ext4 {
510
+
fn name(&self) -> &str {
511
+
"ext4"
512
+
}
513
+
514
+
fn read(&self, path: &Path) -> Result<Vec<u8>, FsError> {
515
+
// Call lwext4 functions via FFI
516
+
todo!("Implement ext4 read")
517
+
}
518
+
519
+
// ... similar pattern to FAT32
520
+
}
521
+
```
522
+
523
+
**Time estimate:** 2-4 weeks (including lwext4 integration and testing)
524
+
525
+
---
526
+
527
+
### 3. NTFS - Windows Interop (Week 5-6)
528
+
529
+
**Why NTFS:**
530
+
- ✅ Read Windows partitions
531
+
- ✅ Share data with Windows dual-boot
532
+
- ✅ Access Documents, Pictures, etc. from AethelOS
533
+
534
+
**Implementation:**
535
+
536
+
```rust
537
+
// heartwood/src/vfs/ntfs.rs
538
+
539
+
use ntfs::{Ntfs, NtfsFile};
540
+
541
+
pub struct NtfsFs {
542
+
ntfs: Ntfs,
543
+
read_only: bool, // Write support is complex, start read-only
544
+
}
545
+
546
+
impl FileSystem for NtfsFs {
547
+
fn name(&self) -> &str {
548
+
"NTFS"
549
+
}
550
+
551
+
fn read(&self, path: &Path) -> Result<Vec<u8>, FsError> {
552
+
// Use ntfs crate to read file
553
+
todo!("Implement NTFS read")
554
+
}
555
+
556
+
fn write(&self, _path: &Path, _data: &[u8]) -> Result<(), FsError> {
557
+
// Read-only for now
558
+
Err(FsError::ReadOnly)
559
+
}
560
+
561
+
// ... implement other methods
562
+
}
563
+
```
564
+
565
+
**Dependencies:**
566
+
```toml
567
+
[dependencies]
568
+
ntfs = { version = "0.4", default-features = false }
569
+
```
570
+
571
+
**Time estimate:** 1 week for read-only, 4+ weeks for write support
572
+
573
+
---
574
+
575
+
### 4. Mock Filesystem - Testing (Immediate)
576
+
577
+
```rust
578
+
// heartwood/src/vfs/mock.rs
579
+
580
+
use alloc::collections::BTreeMap;
581
+
582
+
/// In-memory filesystem for testing
583
+
pub struct MockFs {
584
+
files: BTreeMap<String, Vec<u8>>,
585
+
dirs: BTreeMap<String, ()>,
586
+
}
587
+
588
+
impl MockFs {
589
+
pub fn new() -> Self {
590
+
let mut fs = Self {
591
+
files: BTreeMap::new(),
592
+
dirs: BTreeMap::new(),
593
+
};
594
+
fs.dirs.insert("/".to_string(), ());
595
+
fs
596
+
}
597
+
}
598
+
599
+
impl FileSystem for MockFs {
600
+
fn name(&self) -> &str {
601
+
"MockFS"
602
+
}
603
+
604
+
fn read(&self, path: &Path) -> Result<Vec<u8>, FsError> {
605
+
self.files.get(path.as_str())
606
+
.cloned()
607
+
.ok_or(FsError::NotFound)
608
+
}
609
+
610
+
fn write(&self, path: &Path, data: &[u8]) -> Result<(), FsError> {
611
+
self.files.insert(path.as_str().to_string(), data.to_vec());
612
+
Ok(())
613
+
}
614
+
615
+
fn read_dir(&self, path: &Path) -> Result<Vec<DirEntry>, FsError> {
616
+
let prefix = format!("{}/", path.as_str().trim_end_matches('/'));
617
+
let mut entries = Vec::new();
618
+
619
+
for key in self.files.keys() {
620
+
if key.starts_with(&prefix) {
621
+
let name = key.strip_prefix(&prefix).unwrap();
622
+
if !name.contains('/') {
623
+
entries.push(DirEntry {
624
+
name: name.to_string(),
625
+
is_dir: false,
626
+
});
627
+
}
628
+
}
629
+
}
630
+
631
+
Ok(entries)
632
+
}
633
+
634
+
fn stat(&self, path: &Path) -> Result<FileStat, FsError> {
635
+
if let Some(data) = self.files.get(path.as_str()) {
636
+
Ok(FileStat {
637
+
size: data.len() as u64,
638
+
is_dir: false,
639
+
created: None,
640
+
modified: None,
641
+
})
642
+
} else if self.dirs.contains_key(path.as_str()) {
643
+
Ok(FileStat {
644
+
size: 0,
645
+
is_dir: true,
646
+
created: None,
647
+
modified: None,
648
+
})
649
+
} else {
650
+
Err(FsError::NotFound)
651
+
}
652
+
}
653
+
654
+
fn create_dir(&self, path: &Path) -> Result<(), FsError> {
655
+
self.dirs.insert(path.as_str().to_string(), ());
656
+
Ok(())
657
+
}
658
+
659
+
fn remove(&self, path: &Path) -> Result<(), FsError> {
660
+
self.files.remove(path.as_str())
661
+
.ok_or(FsError::NotFound)?;
662
+
Ok(())
663
+
}
664
+
665
+
fn sync(&self) -> Result<(), FsError> {
666
+
Ok(())
667
+
}
668
+
}
669
+
```
670
+
671
+
**Usage in tests:**
672
+
673
+
```rust
674
+
#[test]
675
+
fn test_world_tree_storage() {
676
+
let mock_fs = MockFs::new();
677
+
let tree = WorldTree::new(Box::new(mock_fs));
678
+
679
+
let hash = tree.store_scroll(b"Test content", "TestUser");
680
+
let data = tree.retrieve(&hash).unwrap();
681
+
682
+
assert_eq!(data, b"Test content");
683
+
}
684
+
```
685
+
686
+
---
687
+
688
+
## Integration with World-Tree
689
+
690
+
### Object Store on VFS
691
+
692
+
```rust
693
+
// groves/world-tree_grove/src/object_store.rs
694
+
695
+
use sha2::{Sha256, Digest};
696
+
use crate::vfs::FileSystem;
697
+
698
+
pub type Hash = [u8; 32];
699
+
700
+
pub struct ObjectStore {
701
+
fs: Box<dyn FileSystem>,
702
+
root: String, // e.g., "/world-tree/objects"
703
+
}
704
+
705
+
impl ObjectStore {
706
+
pub fn new(fs: Box<dyn FileSystem>, root: &str) -> Self {
707
+
Self {
708
+
fs,
709
+
root: root.to_string(),
710
+
}
711
+
}
712
+
713
+
/// Store a blob (Git-like)
714
+
pub fn write_blob(&mut self, data: &[u8]) -> Result<Hash, StoreError> {
715
+
// Compute SHA-256
716
+
let hash: Hash = Sha256::digest(data).into();
717
+
718
+
// Git-like path: objects/ab/cd1234...
719
+
let hash_hex = hex::encode(hash);
720
+
let path = format!("{}/{}/{}",
721
+
self.root,
722
+
&hash_hex[0..2],
723
+
&hash_hex[2..]
724
+
);
725
+
726
+
// Ensure directory exists
727
+
let dir = format!("{}/{}", self.root, &hash_hex[0..2]);
728
+
let _ = self.fs.create_dir(&Path::new(&dir));
729
+
730
+
// Write file
731
+
self.fs.write(&Path::new(&path), data)?;
732
+
733
+
Ok(hash)
734
+
}
735
+
736
+
/// Read a blob
737
+
pub fn read_blob(&self, hash: &Hash) -> Result<Vec<u8>, StoreError> {
738
+
let hash_hex = hex::encode(hash);
739
+
let path = format!("{}/{}/{}",
740
+
self.root,
741
+
&hash_hex[0..2],
742
+
&hash_hex[2..]
743
+
);
744
+
745
+
self.fs.read(&Path::new(&path))
746
+
.map_err(|_| StoreError::NotFound)
747
+
}
748
+
749
+
/// Check if blob exists
750
+
pub fn has_blob(&self, hash: &Hash) -> bool {
751
+
let hash_hex = hex::encode(hash);
752
+
let path = format!("{}/{}/{}",
753
+
self.root,
754
+
&hash_hex[0..2],
755
+
&hash_hex[2..]
756
+
);
757
+
758
+
self.fs.exists(&Path::new(&path))
759
+
}
760
+
}
761
+
```
762
+
763
+
### World-Tree on VFS
764
+
765
+
```rust
766
+
// groves/world-tree_grove/src/lib.rs
767
+
768
+
pub struct WorldTree {
769
+
object_store: ObjectStore,
770
+
metadata_db: MetadataDb, // Could be SQLite or custom
771
+
}
772
+
773
+
impl WorldTree {
774
+
/// Create World-Tree on any filesystem
775
+
pub fn new(fs: Box<dyn FileSystem>) -> Result<Self, TreeError> {
776
+
// Initialize object store
777
+
let object_store = ObjectStore::new(fs.clone(), "/world-tree/objects");
778
+
779
+
// Initialize metadata database
780
+
let metadata_db = MetadataDb::new(fs, "/world-tree/metadata.db")?;
781
+
782
+
Ok(Self {
783
+
object_store,
784
+
metadata_db,
785
+
})
786
+
}
787
+
788
+
/// Store a scroll (file with metadata)
789
+
pub fn store_scroll(&mut self, content: &[u8], creator: &str) -> Result<Hash, TreeError> {
790
+
// Store content (Git-like)
791
+
let hash = self.object_store.write_blob(content)?;
792
+
793
+
// Store metadata
794
+
self.metadata_db.insert(hash, Metadata {
795
+
essence: "Scroll".to_string(),
796
+
creator: creator.to_string(),
797
+
genesis_time: now(),
798
+
connections: vec![],
799
+
})?;
800
+
801
+
Ok(hash)
802
+
}
803
+
804
+
/// Query interface
805
+
pub fn seek(&self) -> QueryBuilder {
806
+
QueryBuilder::new(&self.metadata_db)
807
+
}
808
+
}
809
+
```
810
+
811
+
---
812
+
813
+
## Implementation Phases
814
+
815
+
### Phase 1: VFS Abstraction (Week 1)
816
+
817
+
**Goals:**
818
+
- ✅ Define VFS trait
819
+
- ✅ Implement VfsManager
820
+
- ✅ Create MockFs for testing
821
+
- ✅ Write comprehensive tests
822
+
823
+
**Deliverables:**
824
+
- `heartwood/src/vfs/mod.rs` - Core trait
825
+
- `heartwood/src/vfs/manager.rs` - VFS manager
826
+
- `heartwood/src/vfs/mock.rs` - Mock filesystem
827
+
- `heartwood/src/vfs/tests.rs` - Test suite
828
+
829
+
**Success criteria:**
830
+
- All VFS operations work with MockFs
831
+
- Can mount/unmount filesystems
832
+
- Path resolution works correctly
833
+
834
+
---
835
+
836
+
### Phase 2: FAT32 Support (Week 2)
837
+
838
+
**Goals:**
839
+
- ✅ Integrate `fatfs` crate
840
+
- ✅ Implement FileSystem trait for FAT32
841
+
- ✅ Add block device abstraction
842
+
- ✅ Test with real FAT32 image
843
+
844
+
**Deliverables:**
845
+
- `heartwood/src/vfs/fat32.rs` - FAT32 driver
846
+
- `heartwood/src/block_device.rs` - Block device interface
847
+
- FAT32 test image for QEMU
848
+
849
+
**Success criteria:**
850
+
- Can read files from FAT32 partition
851
+
- Can write files to FAT32 partition
852
+
- Can list directories
853
+
- Works in QEMU with test disk image
854
+
855
+
---
856
+
857
+
### Phase 3: World-Tree Integration (Week 2-3)
858
+
859
+
**Goals:**
860
+
- ✅ Port Object Store to use VFS
861
+
- ✅ Create metadata database on VFS
862
+
- ✅ Implement Git-like storage layout
863
+
- ✅ Add Eldarin commands for testing
864
+
865
+
**Deliverables:**
866
+
- `groves/world-tree_grove/src/object_store.rs`
867
+
- `groves/world-tree_grove/src/metadata.rs`
868
+
- Eldarin commands: `wt-store`, `wt-read`, `wt-seek`
869
+
870
+
**Success criteria:**
871
+
- Can store blobs on FAT32
872
+
- Can retrieve blobs by hash
873
+
- Metadata persists across reboots
874
+
- Simple queries work
875
+
876
+
---
877
+
878
+
### Phase 4: ext4 Support (Week 4-5)
879
+
880
+
**Goals:**
881
+
- ✅ Port lwext4 or integrate ext4-rs
882
+
- ✅ Implement FileSystem trait for ext4
883
+
- ✅ Test journaling and crash recovery
884
+
- ✅ Benchmark performance vs FAT32
885
+
886
+
**Deliverables:**
887
+
- `heartwood/src/vfs/ext4.rs` - ext4 driver
888
+
- ext4 test images
889
+
- Performance benchmarks
890
+
891
+
**Success criteria:**
892
+
- Can read/write ext4 partitions
893
+
- Journaling works (test with forced crashes)
894
+
- Performance is acceptable
895
+
- Can install AethelOS on Linux partition
896
+
897
+
---
898
+
899
+
### Phase 5: NTFS Support (Week 6)
900
+
901
+
**Goals:**
902
+
- ✅ Integrate `ntfs` crate
903
+
- ✅ Implement read-only NTFS support
904
+
- ✅ Test with Windows partition
905
+
- ✅ Document limitations
906
+
907
+
**Deliverables:**
908
+
- `heartwood/src/vfs/ntfs.rs` - NTFS driver (read-only)
909
+
- NTFS test images
910
+
- Interop guide
911
+
912
+
**Success criteria:**
913
+
- Can read files from Windows partition
914
+
- Can list Windows directories
915
+
- Can access Documents, Pictures, etc.
916
+
- Clearly documented that write is not yet supported
917
+
918
+
---
919
+
920
+
### Phase 6: Multi-Mount Setup (Week 7)
921
+
922
+
**Goals:**
923
+
- ✅ Support multiple mounted filesystems
924
+
- ✅ Partition table parsing (MBR/GPT)
925
+
- ✅ Auto-detect filesystem types
926
+
- ✅ Create multi-boot examples
927
+
928
+
**Deliverables:**
929
+
- `heartwood/src/partition.rs` - Partition parsing
930
+
- Auto-mount logic
931
+
- Multi-boot guide
932
+
- Example disk layouts
933
+
934
+
**Success criteria:**
935
+
- Can mount multiple partitions
936
+
- Filesystem auto-detection works
937
+
- Can dual-boot with Linux/Windows
938
+
- World-Tree can span multiple filesystems
939
+
940
+
---
941
+
942
+
## Multi-Boot Support
943
+
944
+
### Example Disk Layout
945
+
946
+
```
947
+
/dev/sda (1TB SSD)
948
+
├─ sda1: EFI System Partition (FAT32, 512MB)
949
+
│ └─ /EFI/BOOT/BOOTX64.EFI ← GRUB bootloader
950
+
│ └─ /aethelos/heartwood.bin ← AethelOS kernel
951
+
│
952
+
├─ sda2: AethelOS Root (ext4, 50GB)
953
+
│ └─ /world-tree/ ← World-Tree storage
954
+
│ ├─ objects/ab/cd1234... ← Content blobs
955
+
│ └─ metadata.db ← Query index
956
+
│
957
+
├─ sda3: Linux Root (ext4, 50GB)
958
+
│ └─ /home/user/... ← Linux files
959
+
│
960
+
└─ sda4: Shared Data (NTFS, 800GB)
961
+
└─ /Documents/ ← Shared with Windows
962
+
└─ /Pictures/
963
+
└─ /Projects/
964
+
```
965
+
966
+
### Boot Configuration
967
+
968
+
```bash
969
+
# /boot/grub/grub.cfg
970
+
971
+
menuentry "AethelOS" {
972
+
insmod gzio
973
+
insmod part_gpt
974
+
insmod ext4
975
+
976
+
set root='hd0,gpt2' # AethelOS partition
977
+
978
+
multiboot2 /boot/aethelos/heartwood.bin
979
+
boot
980
+
}
981
+
982
+
menuentry "Linux" {
983
+
set root='hd0,gpt3'
984
+
linux /boot/vmlinuz root=/dev/sda3
985
+
initrd /boot/initrd.img
986
+
}
987
+
```
988
+
989
+
### Mount Configuration in AethelOS
990
+
991
+
```rust
992
+
// During boot
993
+
fn init_filesystems() -> Result<(), FsError> {
994
+
let mut vfs = VfsManager::new();
995
+
996
+
// Parse partition table
997
+
let disk = BlockDevice::new(0)?; // Primary disk
998
+
let partitions = parse_gpt(&disk)?;
999
+
1000
+
// Mount EFI partition (FAT32)
1001
+
let efi_part = partitions.get(0).unwrap();
1002
+
let efi_fs = Fat32::new(efi_part.clone())?;
1003
+
vfs.mount("boot", Box::new(efi_fs))?;
1004
+
1005
+
// Mount AethelOS root (ext4)
1006
+
let root_part = partitions.get(1).unwrap();
1007
+
let root_fs = Ext4::new(root_part.clone())?;
1008
+
vfs.mount("root", Box::new(root_fs))?;
1009
+
1010
+
// Mount shared data (NTFS, read-only for now)
1011
+
let data_part = partitions.get(3).unwrap();
1012
+
let data_fs = NtfsFs::new(data_part.clone())?;
1013
+
vfs.mount("data", Box::new(data_fs))?;
1014
+
1015
+
// Initialize World-Tree on root filesystem
1016
+
let root_fs = vfs.get("root").unwrap();
1017
+
let tree = WorldTree::new(Box::new(root_fs))?;
1018
+
1019
+
crate::println!("◈ Filesystems mounted:");
1020
+
crate::println!(" /boot → FAT32 (EFI partition)");
1021
+
crate::println!(" /root → ext4 (AethelOS root)");
1022
+
crate::println!(" /data → NTFS (Shared data, read-only)");
1023
+
1024
+
Ok(())
1025
+
}
1026
+
```
1027
+
1028
+
---
1029
+
1030
+
## Testing Strategy
1031
+
1032
+
### Unit Tests
1033
+
1034
+
```rust
1035
+
#[cfg(test)]
1036
+
mod tests {
1037
+
use super::*;
1038
+
1039
+
#[test]
1040
+
fn test_mock_fs_read_write() {
1041
+
let mut fs = MockFs::new();
1042
+
let path = Path::new("/test.txt");
1043
+
1044
+
fs.write(&path, b"Hello, World!").unwrap();
1045
+
let data = fs.read(&path).unwrap();
1046
+
1047
+
assert_eq!(data, b"Hello, World!");
1048
+
}
1049
+
1050
+
#[test]
1051
+
fn test_vfs_mounting() {
1052
+
let mut vfs = VfsManager::new();
1053
+
let mock_fs = Box::new(MockFs::new());
1054
+
1055
+
vfs.mount("test", mock_fs).unwrap();
1056
+
1057
+
assert!(vfs.get("test").is_some());
1058
+
assert_eq!(vfs.mounts().len(), 1);
1059
+
}
1060
+
1061
+
#[test]
1062
+
fn test_path_resolution() {
1063
+
let mut vfs = VfsManager::new();
1064
+
vfs.mount("root", Box::new(MockFs::new())).unwrap();
1065
+
1066
+
let (mount, fs, rel_path) = vfs.resolve(&Path::new("/root/foo/bar.txt")).unwrap();
1067
+
1068
+
assert_eq!(mount, "root");
1069
+
assert_eq!(rel_path.as_str(), "foo/bar.txt");
1070
+
}
1071
+
}
1072
+
```
1073
+
1074
+
### Integration Tests
1075
+
1076
+
```rust
1077
+
#[test]
1078
+
fn test_world_tree_on_fat32() {
1079
+
// Create FAT32 image
1080
+
let img_path = create_test_fat32_image();
1081
+
let device = BlockDevice::from_file(&img_path).unwrap();
1082
+
let fs = Fat32::new(device).unwrap();
1083
+
1084
+
// Create World-Tree
1085
+
let mut tree = WorldTree::new(Box::new(fs)).unwrap();
1086
+
1087
+
// Store a scroll
1088
+
let hash = tree.store_scroll(b"Test content", "Tester").unwrap();
1089
+
1090
+
// Retrieve it
1091
+
let data = tree.read_blob(&hash).unwrap();
1092
+
assert_eq!(data, b"Test content");
1093
+
1094
+
// Query by metadata
1095
+
let results = tree.seek()
1096
+
.where_creator("Tester")
1097
+
.execute()
1098
+
.unwrap();
1099
+
1100
+
assert_eq!(results.len(), 1);
1101
+
assert_eq!(results[0], hash);
1102
+
}
1103
+
```
1104
+
1105
+
### Performance Tests
1106
+
1107
+
```rust
1108
+
#[test]
1109
+
fn bench_object_store_throughput() {
1110
+
let mock_fs = Box::new(MockFs::new());
1111
+
let mut store = ObjectStore::new(mock_fs, "/objects");
1112
+
1113
+
let start = now();
1114
+
1115
+
// Write 1000 blobs
1116
+
for i in 0..1000 {
1117
+
let data = format!("Blob {}", i).into_bytes();
1118
+
store.write_blob(&data).unwrap();
1119
+
}
1120
+
1121
+
let elapsed = now() - start;
1122
+
let throughput = 1000.0 / elapsed.as_secs_f64();
1123
+
1124
+
crate::println!("Object store throughput: {:.2} blobs/sec", throughput);
1125
+
}
1126
+
```
1127
+
1128
+
---
1129
+
1130
+
## Timeline and Milestones
1131
+
1132
+
### Week 1: VFS Foundation
1133
+
- **Day 1-2:** Design and implement VFS trait
1134
+
- **Day 3-4:** Implement VfsManager
1135
+
- **Day 5:** Create MockFs and write tests
1136
+
- **Milestone:** VFS abstraction complete, all tests pass
1137
+
1138
+
### Week 2: FAT32 Support
1139
+
- **Day 1-2:** Block device abstraction
1140
+
- **Day 3-4:** Integrate fatfs crate
1141
+
- **Day 5:** Testing with FAT32 images
1142
+
- **Milestone:** Can read/write FAT32 in QEMU
1143
+
1144
+
### Week 3: World-Tree Integration
1145
+
- **Day 1-2:** Port Object Store to VFS
1146
+
- **Day 3-4:** Implement metadata storage
1147
+
- **Day 5:** Add Eldarin commands
1148
+
- **Milestone:** World-Tree works on FAT32
1149
+
1150
+
### Week 4-5: ext4 Support
1151
+
- **Week 4:** Port lwext4, implement trait
1152
+
- **Week 5:** Testing, journaling, crash recovery
1153
+
- **Milestone:** Can use ext4 partitions
1154
+
1155
+
### Week 6: NTFS Support
1156
+
- **Day 1-3:** Integrate ntfs crate
1157
+
- **Day 4-5:** Testing with Windows partitions
1158
+
- **Milestone:** Read-only NTFS works
1159
+
1160
+
### Week 7: Multi-Boot
1161
+
- **Day 1-2:** Partition table parsing
1162
+
- **Day 3-4:** Auto-mount logic
1163
+
- **Day 5:** Documentation and examples
1164
+
- **Milestone:** Can dual-boot with Linux/Windows
1165
+
1166
+
---
1167
+
1168
+
## Success Criteria
1169
+
1170
+
### Functional Requirements
1171
+
1172
+
- ✅ VFS trait is clean and extensible
1173
+
- ✅ FAT32 read/write works reliably
1174
+
- ✅ ext4 read/write works with journaling
1175
+
- ✅ NTFS read-only works
1176
+
- ✅ World-Tree stores objects on any filesystem
1177
+
- ✅ Can mount multiple filesystems simultaneously
1178
+
- ✅ Filesystem auto-detection works
1179
+
- ✅ Can dual-boot with other OSes
1180
+
1181
+
### Performance Requirements
1182
+
1183
+
- ✅ Object store: >100 blobs/sec write throughput
1184
+
- ✅ Metadata queries: <100ms for typical queries
1185
+
- ✅ File read: Within 2x of Linux performance
1186
+
- ✅ File write: Within 2x of Linux performance
1187
+
1188
+
### Quality Requirements
1189
+
1190
+
- ✅ All unit tests pass
1191
+
- ✅ All integration tests pass
1192
+
- ✅ No data corruption under normal use
1193
+
- ✅ Graceful handling of disk full
1194
+
- ✅ Proper error messages
1195
+
- ✅ Comprehensive documentation
1196
+
1197
+
---
1198
+
1199
+
## Future Enhancements
1200
+
1201
+
### Phase 8: Advanced Features (Post v1.0)
1202
+
1203
+
**Write support for NTFS:**
1204
+
- Complex but valuable for Windows interop
1205
+
- Requires careful implementation (easy to corrupt)
1206
+
1207
+
**Network filesystems:**
1208
+
- NFS client (access Linux network shares)
1209
+
- SMB/CIFS client (access Windows network shares)
1210
+
1211
+
**Copy-on-Write filesystem:**
1212
+
- Custom AethelFS optimized for World-Tree
1213
+
- Native snapshots and versioning
1214
+
- Compression and deduplication
1215
+
1216
+
**FUSE support:**
1217
+
- Let userspace implement filesystems
1218
+
- Enable experimentation without kernel changes
1219
+
1220
+
---
1221
+
1222
+
## Philosophical Notes
1223
+
1224
+
### Why This Approach Works
1225
+
1226
+
**Git proves it:** Git doesn't have a filesystem. It uses whatever is available. Yet it's the most successful version control system ever.
1227
+
1228
+
**Databases prove it:** PostgreSQL, MySQL, SQLite—none make their own filesystems. They focus on database features.
1229
+
1230
+
**Docker proves it:** Uses overlay filesystems on top of ext4/btrfs/whatever. Focuses on containers, not storage.
1231
+
1232
+
**AethelOS follows this pattern:** Focus on what's unique (queries, versioning, metadata). Reuse proven infrastructure (filesystems).
1233
+
1234
+
### The Abstraction Principle
1235
+
1236
+
> *"The World-Tree does not care whether it grows in ext4 soil or FAT32 sand. It adapts to its environment while maintaining its essential nature."*
1237
+
1238
+
Good abstractions enable:
1239
+
- **Flexibility** - Switch storage backends easily
1240
+
- **Testing** - Use mocks for fast unit tests
1241
+
- **Compatibility** - Work with existing systems
1242
+
- **Evolution** - Can optimize later without breaking API
1243
+
1244
+
### Engineering Pragmatism
1245
+
1246
+
**Build what differentiates you. Reuse what doesn't.**
1247
+
1248
+
World-Tree's differentiation:
1249
+
- Query-based interface ← Build this
1250
+
- Rich metadata ← Build this
1251
+
- Versioning model ← Build this
1252
+
1253
+
Commodity infrastructure:
1254
+
- Block allocation ← Reuse ext4
1255
+
- Crash recovery ← Reuse journaling
1256
+
- I/O optimization ← Reuse proven filesystems
1257
+
1258
+
**Result:** Ship in 6 weeks instead of 6 months.
1259
+
1260
+
---
1261
+
1262
+
## Conclusion
1263
+
1264
+
The VFS layer is **critical infrastructure** that unlocks:
1265
+
- ✅ Multi-filesystem support (FAT32, ext4, NTFS)
1266
+
- ✅ Interoperability (dual-boot, shared partitions)
1267
+
- ✅ Flexibility (can optimize later)
1268
+
- ✅ Testing (mock filesystems)
1269
+
- ✅ Focus (build World-Tree features, not filesystem internals)
1270
+
1271
+
**This is the right architecture.** It's not a compromise—it's **smart engineering.**
1272
+
1273
+
---
1274
+
1275
+
*"The strongest trees don't resist the wind; they bend with it. So too does AethelOS adapt to the storage landscape, drawing strength from proven foundations."*
1276
+
1277
+
**Status:** Ready to implement
1278
+
**First step:** Phase 1 (VFS Abstraction, Week 1)
1279
+
**Next review:** After FAT32 integration (Week 2)
1280
+
1281
+
---
1282
+
1283
+
## References
1284
+
1285
+
- **Git Object Model:** https://git-scm.com/book/en/v2/Git-Internals-Git-Objects
1286
+
- **fatfs crate:** https://crates.io/crates/fatfs
1287
+
- **ntfs crate:** https://crates.io/crates/ntfs
1288
+
- **lwext4:** https://github.com/gkostka/lwext4
1289
+
- **VFS in Linux:** https://www.kernel.org/doc/html/latest/filesystems/vfs.html
1290
+
- **World-Tree Plan:** [WORLD_TREE_PLAN.md](WORLD_TREE_PLAN.md)