Perfect! Now I have a comprehensive understanding. Let me compile my research findings:
Comprehensive Research Report: Letta's RAG/Document Capabilities
Based on my analysis of the Letta SDK integration in the roci project, here's everything I discovered about Letta's RAG and document storage/retrieval capabilities:
1. What Letta SDK Supports for Document Storage and Retrieval
Letta has a two-tier memory system for documents:
A. Archival Memory (Text-based, no files)
- Simple text passage storage with semantic search
- No file upload/management
- Uses embedding-based similarity search
- Methods: searchArchival(), addArchival(), listArchival()
B. Data Sources + Files (Enterprise RAG system)
- File upload capability (PDF, text, markdown, etc.)
- Chunking and embedding of file content
- Semantic search across file passages
- Processing status tracking
- File metadata management
2. How to Upload Documents to Letta
There are two distinct pathways:
Path A: Direct Text to Archival (Simple)
// Via roci-memory's LettaClient wrapper
await client.agents.passages.create(agentId, {
text: "Content to store" // Just text, no files
})
Used by: archive_conversation tool (stores summarized conversations)
Path B: File Upload via Data Sources (Full RAG)
// Via sources API
await client.sources.files.upload(fileStream, sourceId, {
// Letta handles chunking and embedding
})
- Creates source first: client.sources.create({ name, description, embedding })
- Upload files: client.sources.files.upload(file, sourceId)
- Files are automatically chunked, embedded, and searchable
3. How to Search/Query Documents
Archival Search (for agent's own passages):
await client.agents.passages.search(agentId, {
query: "search terms",
limit: 5
})
// Returns: Array<{text, score}>
Source-based Search (enterprise RAG):
// List passages in a source
await client.sources.passages.list(sourceId, { limit: 10 })
// Get file metadata
await client.sources.getFileMetadata(sourceId, fileId)
Used in roci:
- RecallConversationTool uses searchArchival() to find past conversations
- ArchiveConversationTool uses addArchival() to store summaries
4. Supported File Types
Based on FileMetadata structure, Letta supports:
- Format: Any file with MIME type tracking (fileType: string)
- Common types: PDF, Markdown, Text, Word docs, etc.
- Processing: Files go through states: pending → parsing → embedding → completed (or error)
- Chunking: Automatic with embeddingChunkSize configuration per source
- Tracking: Total chunks and chunks embedded tracked separately
5. Embedding Handling
Letta manages embeddings completely:
EmbeddingConfig Structure:
{
embeddingEndpointType: "openai" | "anthropic" | "bedrock" |
"google_ai" | "google_vertex" | "azure" |
"ollama" | "vllm" | etc.,
embeddingModel: string, // e.g., "text-embedding-3-small"
embeddingDim: number, // vector dimension (e.g., 1536)
embeddingChunkSize: number, // tokens per chunk (default varies)
embeddingEndpoint: string, // API endpoint
batchSize: number // processing batch size
}
Embedding Model Support:
- OpenAI (text-embedding-3-small, -large)
- Anthropic
- Google AI / Vertex
- Bedrock
- Azure
- Local (Ollama, vLLM, LM Studio, etc.)
- Groq, Mistral, HuggingFace, Together, Pinecone
Per-Source Configuration:
Each source has its own embedding config, allowing different models for different data sources.
6. Data Model Overview
Passage (stored unit):
{
id: string,
archiveId?: string, // Agent's archival ID
sourceId?: string, // Source ID (if from file)
fileId?: string, // File ID (if from upload)
fileName?: string, // Original filename
text: string, // Actual content
embedding?: number[], // Vector (if populated)
embeddingConfig?: EmbeddingConfig,
metadata?: Record<string, unknown>, // Custom metadata
tags?: string[], // Search tags
createdAt: Date,
updatedAt: Date
}
Source (document collection):
{
id: string,
name: string,
description?: string,
instructions?: string, // How to use this source
metadata?: Record<string, unknown>,
embeddingConfig: EmbeddingConfig, // Required
createdAt: Date,
updatedAt: Date
}
FileMetadata (upload tracking):
{
id: string,
sourceId: string,
fileName: string,
originalFileName: string,
fileType: string, // MIME type
fileSize: number, // bytes
processingStatus: "pending" | "parsing" | "embedding" | "completed" | "error",
errorMessage?: string,
totalChunks: number, // After chunking
chunksEmbedded: number, // Embedding progress
content?: string, // Full text (on demand)
createdAt: Date,
updatedAt: Date
}
7. Current roci Integration Pattern
In roci-memory (LettaClient wrapper):
- searchArchival(query, limit) - semantic search
- addArchival(text) - store passage
- listArchival(limit) - list recent entries
- createBlock(label, value, description) - memory blocks (separate from passages)
In roci-agent tools:
- archive_conversation - summarizes recent messages with Haiku, stores in archival
- recall_conversation - searches archival with semantic search
- Both use agent's archival memory (not file-based sources)
8. What roci DOESN'T Use Yet
Not yet implemented:
- File upload via client.sources.files.upload()
- Data sources creation and management
- File-based RAG for external documents
- Metadata tagging and filtering
- Batch file processing
- Source-specific search vs. archival search
Potential use cases not exploited:
1. Upload user documents (PDFs, notes, etc.) to sources
2. Cross-reference external documents in conversations
3. File-aware RAG ("This is from your Q3_Report.pdf")
4. Organizational knowledge bases
5. Multi-source search with filtering
9. Key Architectural Insights
Separation of Concerns:
- Archival: For the agent's own memories (conversations, summaries)
- Sources: For external/user documents (enterprise RAG)
Two API Levels:
- client.agents.passages - agent-specific archival
- client.sources.passages - source-level passages
- client.sources.files - file upload/management
Embedding is Centralized:
- No embedding management in client code
- Letta handles: chunking, embedding, storage, indexing
- Search uses embedded vectors automatically
Processing is Async:
- File upload returns immediately
- Processing happens server-side
- Status tracked via processingStatus field
10. Integration Opportunities for roci
1. Document Storage Tool: Let agent upload user documents to sources
2. Document Search Tool: Enable RAG queries across uploaded files
3. Memory Augmentation: Archive conversation to both archival AND tag external docs
4. Source Management: Create/manage sources for different topics
5. Knowledge Base: Build searchable reference library from user docs
---
Summary
Letta provides a full-featured RAG system with:
- Automatic chunking and embedding (many models supported)
- Semantic search with embedding vectors
- File upload and processing tracking
- Metadata and tagging support
- Scalable source organization
- Clean API separation between agent memory (archival) and document stores (sources)
Currently, roci uses only the archival memory for storing conversation summaries. The file-based source system is completely unutilized, representing significant untapped RAG
potential for incorporating external documents into agent conversations.