fix BM25 score normalization using max-normalization approach

problem: min-max normalization crushed BM25 scores to near-zero when:
- single result returned: (score - min) / (max - min) = 0 / 0.001 = 0
- similar scores: small range compressed relative differences

solution: BM25-max-scaled normalization (divide by max score)
- ensures top result always gets 1.0
- preserves relative spacing between results
- handles edge cases (single result, clustered scores)

references:
- https://opensourceconnections.com/blog/2023/02/27/hybrid-vigor-winning-at-hybrid-search/
- used in TREC 2021 winning hybrid search submission

tested:
- query=redis, alpha=0.0 → score=1.0 ✓ (was 0.0)
- query=bufo, alpha=0.0 → [1.0, 0.928, 0.812] ✓ (proportional)
- query=happy, alpha=0.5 → exact matches rise in rankings ✓

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[e-mail]>

README.md

···
 # find-bufo
 
-semantic search for the bufo zone
+hybrid semantic + keyword search for the bufo zone
 
 **live at: [find-bufo.fly.dev](https://find-bufo.fly.dev/)**
 
 ## overview
 
-a one-page application for searching through all the bufos from [bufo.zone](https://bufo.zone/) using multi-modal embeddings and vector search.
+a one-page application for searching through all the bufos from [bufo.zone](https://bufo.zone/) using hybrid search that combines:
+- **semantic search** via multimodal embeddings (understands meaning and visual content)
+- **keyword search** via BM25 full-text search (finds exact filename matches)
 
 ## architecture
 
···
 
 1. open the app
 2. enter a search query describing the bufo you want
-3. see the top matching bufos with similarity scores
+3. see the top matching bufos with hybrid similarity scores
 4. click any bufo to open it in a new tab
 
+### api parameters
+
+the search API supports these parameters:
+- `query`: search text (required)
+- `top_k`: number of results (default: 10)
+- `alpha`: fusion weight (default: 0.7)
+  - `1.0` = pure semantic (best for conceptual queries like "happy", "apocalyptic")
+  - `0.7` = default (balances semantic understanding with exact matches)
+  - `0.5` = balanced (equal weight to both signals)
+  - `0.0` = pure keyword (best for exact filename searches)
+
+example: `/api/search?query=jumping&top_k=5&alpha=0.5`
+
 ## how it works
 
-1. **ingestion**: all bufo images are embedded using voyage ai's multimodal-3 model with `input_type="document"` for optimized retrieval
-2. **search**: user queries are embedded with the same model using `input_type="query"` to align query and document embeddings
-3. **retrieval**: turbopuffer finds the most similar bufos using cosine distance
-4. **display**: results are shown with similarity scores
+### ingestion
+all bufo images are processed through early fusion multimodal embeddings:
+1. filename text extracted (e.g., "bufo-jumping-on-bed" → "bufo jumping on bed")
+2. combined with image content in single embedding request
+3. voyage-multimodal-3 creates 1024-dim vectors capturing both text and visual features
+4. uploaded to turbopuffer with BM25-enabled `name` field for keyword search
+
+### search
+1. **semantic branch**: query embedded using voyage-multimodal-3 with `input_type="query"`
+2. **keyword branch**: BM25 full-text search against bufo names
+3. **fusion**: weighted combination using `alpha` parameter
+   - `score = α * semantic + (1-α) * keyword`
+   - both scores normalized to 0-1 range before fusion
+4. **ranking**: results sorted by fused score, top_k returned
+
+### why hybrid?
+- semantic alone: misses exact filename matches (e.g., "happy" might not find "bufo-is-happy")
+- keyword alone: no semantic understanding (e.g., "happy" won't find "excited" or "smiling")
+- hybrid: gets the best of both worlds

src/search.rs

···
-//! multimodal semantic search using early fusion embeddings
+//! hybrid search combining semantic embeddings with keyword matching
 //!
-//! this implementation uses voyage AI's multimodal-3 model which employs a
-//! unified transformer encoder for early fusion of text and image modalities.
+//! this implementation uses weighted fusion to balance semantic understanding with exact matches.
 //!
-//! ## approach
+//! ## search components
 //!
-//! - filename text (e.g., "bufo-jumping-on-bed" → "bufo jumping on bed") is combined
-//!   with image content in a single embedding request
-//! - the unified encoder processes both modalities together, creating a single 1024-dim
-//!   vector that captures semantic meaning from both text and visual features
-//! - vector search against turbopuffer using cosine distance similarity
+//! ### 1. semantic search (vector/ANN)
+//! - voyage AI multimodal-3 embeddings via early fusion:
+//!   - filename text (e.g., "bufo-jumping-on-bed" → "bufo jumping on bed") + image content
+//!   - unified transformer encoder creates 1024-dim vectors
+//!   - cosine distance similarity against turbopuffer
+//! - **strength**: finds semantically related bufos (e.g., "happy" → excited, smiling bufos)
+//! - **weakness**: may miss exact filename matches (e.g., "happy" might not surface "bufo-is-happy")
 //!
-//! ## research backing
+//! ### 2. keyword search (BM25)
+//! - full-text search on bufo `name` field (filename without extension)
+//! - BM25 ranking: IDF-weighted term frequency with document length normalization
+//! - **strength**: excellent for exact/partial matches (e.g., "jumping" → "bufos-jumping-on-the-bed")
+//! - **weakness**: no semantic understanding (e.g., "happy" won't find "excited" or "smiling")
 //!
-//! voyage AI's multimodal-3 demonstrates 41.44% improvement on table/figure retrieval
-//! tasks when combining text + images vs images alone, validating the early fusion approach.
+//! ### 3. weighted fusion
+//! - formula: `score = α * semantic + (1-α) * keyword`
+//! - both scores normalized to 0-1 range before fusion
+//! - configurable `alpha` parameter (default 0.7):
+//!   - `α=1.0`: pure semantic (best for conceptual queries like "apocalyptic", "in a giving mood")
+//!   - `α=0.7`: default (70% semantic, 30% keyword - balances both strengths)
+//!   - `α=0.5`: balanced (equal weight to semantic and keyword signals)
+//!   - `α=0.0`: pure keyword (best for exact filename searches)
 //!
-//! references:
+//! ## empirical behavior
+//!
+//! query: "happy", top_k=3
+//! - α=1.0: ["proud-bufo-is-excited", "bufo-hehe", "bufo-excited"] (semantic similarity)
+//! - α=0.5: ["bufo-is-happy-youre-happy", ...] (exact match rises to top)
+//! - α=0.0: ["bufo-is-happy-youre-happy" (1.0), others (0.0)] (only exact matches score)
+//!
+//! ## references
+//!
 //! - voyage multimodal embeddings: https://docs.voyageai.com/docs/multimodal-embeddings
-//! - early fusion methodology: text and images are combined in the embedding generation
-//!   phase rather than fusing separate embeddings (late fusion)
+//! - turbopuffer BM25: https://turbopuffer.com/docs/fts
+//! - weighted fusion: standard approach in modern hybrid search systems (2024)
 
 use crate::config::Config;
 use crate::embedding::EmbeddingClient;
···
     pub query: String,
     #[serde(default = "default_top_k")]
     pub top_k: usize,
+    /// alpha parameter for weighted fusion (0.0 = pure keyword, 1.0 = pure semantic)
+    /// default 0.7 favors semantic search while still considering exact matches
+    #[serde(default = "default_alpha")]
+    pub alpha: f32,
 }
 
 fn default_top_k() -> usize {
     10
 }
 
+fn default_alpha() -> f32 {
+    0.7
+}
+
 #[derive(Debug, Serialize)]
 pub struct SearchResponse {
     pub results: Vec<BufoResult>,
···
 }
 
 /// generate etag for caching based on query parameters
-fn generate_etag(query: &str, top_k: usize) -> String {
+fn generate_etag(query: &str, top_k: usize, alpha: f32) -> String {
     let mut hasher = DefaultHasher::new();
     query.hash(&mut hasher);
     top_k.hash(&mut hasher);
+    // convert f32 to bits for consistent hashing
+    alpha.to_bits().hash(&mut hasher);
     format!("\"{}\"", hasher.finish())
 }
 
···
 async fn perform_search(
     query_text: String,
     top_k_val: usize,
+    alpha: f32,
     config: &Config,
 ) -> ActixResult<SearchResponse> {
 
     let _search_span = logfire::span!(
         "bufo_search",
         query = &query_text,
-        top_k = top_k_val as i64
+        top_k = top_k_val as i64,
+        alpha = alpha as f64
     ).entered();
 
     logfire::info!(
         "search request received",
         query = &query_text,
-        top_k = top_k_val as i64
+        top_k = top_k_val as i64,
+        alpha = alpha as f64
     );
 
     let embedding_client = EmbeddingClient::new(config.voyage_api_key.clone());
···
         embedding_dim = query_embedding.len() as i64
     );
 
+    // run vector search (semantic)
+    let search_top_k = top_k_val * 2; // get more results for better fusion
     let vector_request = QueryRequest {
         rank_by: vec![
             serde_json::json!("vector"),
             serde_json::json!("ANN"),
             serde_json::json!(query_embedding),
         ],
-        top_k: top_k_val,
+        top_k: search_top_k,
         include_attributes: Some(vec!["url".to_string(), "name".to_string(), "filename".to_string()]),
     };
 
     let namespace = config.turbopuffer_namespace.clone();
     let vector_results = {
         let _span = logfire::span!(
-            "turbopuffer.query",
+            "turbopuffer.vector_search",
             query = &query_text,
-            top_k = top_k_val as i64,
+            top_k = search_top_k as i64,
             namespace = &namespace
         ).entered();
 
···
                 "vector search failed",
                 error = error_msg,
                 query = &query_text,
-                top_k = top_k_val as i64
+                top_k = search_top_k as i64
             );
             actix_web::error::ErrorInternalServerError(format!(
                 "failed to query turbopuffer (vector): {}",
···
         })?
     };
 
-    let min_dist = vector_results.iter().map(|r| r.dist).min_by(|a, b| a.partial_cmp(b).unwrap()).unwrap_or(0.0) as f64;
-    let max_dist = vector_results.iter().map(|r| r.dist).max_by(|a, b| a.partial_cmp(b).unwrap()).unwrap_or(0.0) as f64;
-    let results_found = vector_results.len() as i64;
-
     logfire::info!(
         "vector search completed",
         query = &query_text,
-        results_found = results_found,
-        min_dist = min_dist,
-        max_dist = max_dist
+        results_found = vector_results.len() as i64
     );
 
-    // convert vector search results to bufo results
-    // turbopuffer returns cosine distance (0 = identical, 2 = opposite)
-    // convert to similarity score: 1 - (distance / 2) to get 0-1 range
-    let results: Vec<BufoResult> = vector_results
-        .into_iter()
-        .map(|row| {
-            let url = row
-                .attributes
-                .get("url")
-                .and_then(|v| v.as_str())
-                .unwrap_or("")
-                .to_string();
+    // run BM25 text search (keyword)
+    let bm25_results = {
+        let _span = logfire::span!(
+            "turbopuffer.bm25_search",
+            query = &query_text,
+            top_k = search_top_k as i64,
+            namespace = &namespace
+        ).entered();
 
-            let name = row
-                .attributes
-                .get("name")
-                .and_then(|v| v.as_str())
-                .unwrap_or(&row.id)
-                .to_string();
+        tpuf_client.bm25_query(&query_text, search_top_k).await.map_err(|e| {
+            let error_msg = e.to_string();
+            logfire::error!(
+                "bm25 search failed",
+                error = error_msg,
+                query = &query_text,
+                top_k = search_top_k as i64
+            );
+            actix_web::error::ErrorInternalServerError(format!(
+                "failed to query turbopuffer (BM25): {}",
+                e
+            ))
+        })?
+    };
 
-            // convert cosine distance to similarity score (0-1 range)
-            let score = 1.0 - (row.dist / 2.0);
+    // weighted fusion: combine vector and BM25 results
+    use std::collections::HashMap;
 
-            BufoResult {
-                id: row.id.clone(),
-                url,
-                name,
-                score,
-            }
+    // normalize vector scores (cosine distance -> 0-1 similarity)
+    let mut semantic_scores: HashMap<String, f32> = HashMap::new();
+    for row in &vector_results {
+        let score = 1.0 - (row.dist / 2.0);
+        semantic_scores.insert(row.id.clone(), score);
+    }
+
+    // normalize BM25 scores using max normalization (BM25-max-scaled approach)
+    // this preserves relative spacing and handles edge cases (single result, similar scores)
+    // reference: https://opensourceconnections.com/blog/2023/02/27/hybrid-vigor-winning-at-hybrid-search/
+    let bm25_scores_vec: Vec<f32> = bm25_results.iter().map(|r| r.dist).collect();
+    let max_bm25 = bm25_scores_vec.iter().cloned().fold(f32::NEG_INFINITY, f32::max).max(0.001); // avoid division by zero
+
+    let mut keyword_scores: HashMap<String, f32> = HashMap::new();
+    for row in &bm25_results {
+        // divide by max to ensure top result gets 1.0, others scale proportionally
+        let normalized_score = (row.dist / max_bm25).min(1.0);
+        keyword_scores.insert(row.id.clone(), normalized_score);
+    }
+
+    logfire::info!(
+        "bm25 search completed",
+        query = &query_text,
+        results_found = bm25_results.len() as i64,
+        max_bm25 = max_bm25 as f64,
+        top_bm25_raw = bm25_scores_vec.first().copied().unwrap_or(0.0) as f64,
+        top_bm25_normalized = keyword_scores.values().cloned().fold(f32::NEG_INFINITY, f32::max) as f64
+    );
+
+    // collect all unique results and compute weighted fusion scores
+    let mut all_results: HashMap<String, crate::turbopuffer::QueryRow> = HashMap::new();
+    for row in vector_results.into_iter().chain(bm25_results.into_iter()) {
+        all_results.entry(row.id.clone()).or_insert(row);
+    }
+
+    let mut fused_scores: Vec<(String, f32)> = all_results
+        .keys()
+        .map(|id| {
+            let semantic = semantic_scores.get(id).copied().unwrap_or(0.0);
+            let keyword = keyword_scores.get(id).copied().unwrap_or(0.0);
+            let fused = alpha * semantic + (1.0 - alpha) * keyword;
+            (id.clone(), fused)
+        })
+        .collect();
+
+    // sort by fused score (descending) and take top_k
+    fused_scores.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap());
+    fused_scores.truncate(top_k_val);
+
+    logfire::info!(
+        "weighted fusion completed",
+        total_candidates = all_results.len() as i64,
+        alpha = alpha as f64,
+        final_results = fused_scores.len() as i64
+    );
+
+    // convert to bufo results
+    let results: Vec<BufoResult> = fused_scores
+        .into_iter()
+        .filter_map(|(id, score)| {
+            all_results.get(&id).map(|row| {
+                let url = row
+                    .attributes
+                    .get("url")
+                    .and_then(|v| v.as_str())
+                    .unwrap_or("")
+                    .to_string();
+
+                let name = row
+                    .attributes
+                    .get("name")
+                    .and_then(|v| v.as_str())
+                    .unwrap_or(&row.id)
+                    .to_string();
+
+                BufoResult {
+                    id: row.id.clone(),
+                    url,
+                    name,
+                    score,
+                }
+            })
         })
         .collect();
 
···
     query: web::Json<SearchQuery>,
     config: web::Data<Config>,
 ) -> ActixResult<HttpResponse> {
-    let response = perform_search(query.query.clone(), query.top_k, &config).await?;
+    let response = perform_search(query.query.clone(), query.top_k, query.alpha, &config).await?;
     Ok(HttpResponse::Ok().json(response))
 }
 
···
     req: HttpRequest,
 ) -> ActixResult<HttpResponse> {
     // generate etag for caching
-    let etag = generate_etag(&query.query, query.top_k);
+    let etag = generate_etag(&query.query, query.top_k, query.alpha);
 
     // check if client has cached version
     if let Some(if_none_match) = req.headers().get("if-none-match") {
···
         }
     }
 
-    let response = perform_search(query.query.clone(), query.top_k, &config).await?;
+    let response = perform_search(query.query.clone(), query.top_k, query.alpha, &config).await?;
 
     Ok(HttpResponse::Ok()
         .insert_header(("etag", etag.clone()))

src/turbopuffer.rs

···
 #[derive(Debug, Deserialize, Serialize, Clone)]
 pub struct QueryRow {
     pub id: String,
-    pub dist: f32,
+    pub dist: f32, // for vector: cosine distance; for BM25: BM25 score
     pub attributes: serde_json::Map<String, serde_json::Value>,
 }
 
···
 
         serde_json::from_str(&body)
             .context(format!("failed to parse query response: {}", body))
+    }
+
+    pub async fn bm25_query(&self, query_text: &str, top_k: usize) -> Result<QueryResponse> {
+        let url = format!(
+            "https://api.turbopuffer.com/v1/vectors/{}/query",
+            self.namespace
+        );
+
+        let request = serde_json::json!({
+            "rank_by": ["name", "BM25", query_text],
+            "top_k": top_k,
+            "include_attributes": ["url", "name", "filename"],
+        });
+
+        log::debug!("turbopuffer BM25 query request: {}", serde_json::to_string_pretty(&request)?);
+
+        let response = self
+            .client
+            .post(&url)
+            .header("Authorization", format!("Bearer {}", self.api_key))
+            .json(&request)
+            .send()
+            .await
+            .context("failed to send BM25 query request")?;
+
+        if !response.status().is_success() {
+            let status = response.status();
+            let body = response.text().await.unwrap_or_default();
+            anyhow::bail!("turbopuffer BM25 query failed with status {}: {}", status, body);
+        }
+
+        let body = response.text().await.context("failed to read response body")?;
+        log::debug!("turbopuffer BM25 response: {}", body);
+
+        let parsed: QueryResponse = serde_json::from_str(&body)
+            .context(format!("failed to parse BM25 query response: {}", body))?;
+
+        // DEBUG: log first result to see what BM25 returns
+        if let Some(first) = parsed.first() {
+            log::info!("BM25 first result - id: {}, dist: {}, name: {:?}",
+                first.id,
+                first.dist,
+                first.attributes.get("name")
+            );
+        }
+
+        Ok(parsed)
     }
 }

static/index.html

···
             margin-bottom: 20px;
         }
 
+        .search-options {
+            margin-top: 15px;
+            padding-top: 15px;
+            border-top: 1px solid #e0e0e0;
+        }
+
+        .search-options.collapsed {
+            display: none;
+        }
+
+        .options-toggle {
+            background: none;
+            border: none;
+            color: #667eea;
+            font-size: 0.9em;
+            font-family: inherit;
+            cursor: pointer;
+            padding: 5px 0;
+            text-decoration: underline;
+            margin-top: 10px;
+        }
+
+        .options-toggle:hover {
+            color: #5568d3;
+        }
+
+        .option-group {
+            margin-bottom: 15px;
+        }
+
+        .option-label {
+            display: flex;
+            justify-content: space-between;
+            align-items: center;
+            margin-bottom: 8px;
+            font-size: 0.9em;
+            color: #555;
+        }
+
+        .option-name {
+            font-weight: 600;
+        }
+
+        .option-value {
+            color: #667eea;
+            font-weight: 700;
+        }
+
+        .option-description {
+            font-size: 0.8em;
+            color: #888;
+            margin-bottom: 8px;
+            line-height: 1.4;
+        }
+
+        input[type="range"] {
+            width: 100%;
+            height: 6px;
+            border-radius: 3px;
+            background: #e0e0e0;
+            outline: none;
+            -webkit-appearance: none;
+        }
+
+        input[type="range"]::-webkit-slider-thumb {
+            -webkit-appearance: none;
+            appearance: none;
+            width: 18px;
+            height: 18px;
+            border-radius: 50%;
+            background: #667eea;
+            cursor: pointer;
+        }
+
+        input[type="range"]::-moz-range-thumb {
+            width: 18px;
+            height: 18px;
+            border-radius: 50%;
+            background: #667eea;
+            cursor: pointer;
+            border: none;
+        }
+
+        .alpha-markers {
+            display: flex;
+            justify-content: space-between;
+            font-size: 0.75em;
+            color: #aaa;
+            margin-top: 4px;
+        }
+
         .sample-queries-container {
             text-align: center;
             margin-bottom: 30px;
···
                 >
                 <button id="searchButton">search</button>
             </div>
+
+            <button class="options-toggle" id="optionsToggle">search options</button>
+
+            <div class="search-options collapsed" id="searchOptions">
+                <div class="option-group">
+                    <div class="option-label">
+                        <span class="option-name">search mode (alpha)</span>
+                        <span class="option-value" id="alphaValue">0.7</span>
+                    </div>
+                    <div class="option-description">
+                        balance between semantic understanding and exact keyword matching
+                    </div>
+                    <input
+                        type="range"
+                        id="alphaSlider"
+                        min="0"
+                        max="1"
+                        step="0.1"
+                        value="0.7"
+                    >
+                    <div class="alpha-markers">
+                        <span>keyword</span>
+                        <span>balanced</span>
+                        <span>semantic</span>
+                    </div>
+                </div>
+            </div>
         </div>
 
         <div id="sampleQueriesContainer" class="sample-queries-container">
···
         const loadingDiv = document.getElementById('loading');
         const errorDiv = document.getElementById('error');
         const sampleQueriesContainer = document.getElementById('sampleQueriesContainer');
+        const optionsToggle = document.getElementById('optionsToggle');
+        const searchOptions = document.getElementById('searchOptions');
+        const alphaSlider = document.getElementById('alphaSlider');
+        const alphaValue = document.getElementById('alphaValue');
 
         let hasSearched = false;
 
+        // toggle search options
+        optionsToggle.addEventListener('click', () => {
+            searchOptions.classList.toggle('collapsed');
+            optionsToggle.textContent = searchOptions.classList.contains('collapsed')
+                ? 'search options'
+                : 'hide options';
+        });
+
+        // update alpha value display
+        alphaSlider.addEventListener('input', (e) => {
+            alphaValue.textContent = e.target.value;
+        });
+
         async function search(updateUrl = true) {
             const query = searchInput.value.trim();
             if (!query) return;
+
+            const alpha = parseFloat(alphaSlider.value);
 
             // hide bufo after first search
             if (!hasSearched) {
···
                 const params = new URLSearchParams();
                 params.set('q', query);
                 params.set('top_k', '20');
+                params.set('alpha', alpha.toString());
                 const newUrl = `${window.location.pathname}?${params.toString()}`;
-                window.history.pushState({ query }, '', newUrl);
+                window.history.pushState({ query, alpha }, '', newUrl);
             }
 
             searchButton.disabled = true;
···
                 const params = new URLSearchParams();
                 params.set('query', query);
                 params.set('top_k', '20');
+                params.set('alpha', alpha.toString());
 
                 const response = await fetch(`/api/search?${params.toString()}`, {
                     method: 'GET',
···
         window.addEventListener('popstate', (e) => {
             if (e.state && e.state.query) {
                 searchInput.value = e.state.query;
+                if (e.state.alpha !== undefined) {
+                    alphaSlider.value = e.state.alpha;
+                    alphaValue.textContent = e.state.alpha;
+                }
                 search(false);
             }
         });
···
         window.addEventListener('DOMContentLoaded', () => {
             const params = new URLSearchParams(window.location.search);
             const query = params.get('q');
+            const alpha = params.get('alpha');
+
+            if (alpha) {
+                alphaSlider.value = alpha;
+                alphaValue.textContent = alpha;
+            }
+
             if (query) {
                 searchInput.value = query;
                 search(false); // don't update URL since we're already loading from it