chore: drop semantic search (for now!)

docs/roadmap.md

···
 
 ## API: Search Stabilization
 
-Highest priority. This work blocks further investment in semantic search, hybrid ranking, and broader discovery features.
+Highest priority. This work blocks further investment in search quality and broader discovery features.
 
 - [x] Stabilize local development and experimentation around a local `file:` database
 - [x] Document backup, restore, and disk-growth procedures for the experimental local DB
···
 
 Completed on [2026-03-25](../CHANGELOG.md#2026-03-25)
 
-## API: Semantic Search Pipeline
-
-Nomic Embed Text v1.5 via Railway template, async embedding pipeline.
-
-**Blocked on:** API: Search Stabilization
-
-- [ ] Deploy nomic-embed Railway template (`POST /api/embeddings` with Bearer auth)
-- [ ] Embedding client in Go API (`internal/embedding/`) calling the Nomic service
-- [ ] Embed-worker: consume `embedding_jobs` queue, generate 768-dim vectors, store in `document_embeddings`
-- [ ] `GET /search/semantic` endpoint using DiskANN vector_top_k
-- [ ] Reembed command for bulk re-generation
-
-## API: Hybrid Search
-
-Combine keyword and semantic results.
+## API: FTS5 Search Quality
 
-**Blocked on:** API: Search Stabilization, API: Semantic Search Pipeline
+Improve keyword search quality without external dependencies.
 
-- [ ] Score normalization (keyword BM25 → [0,1], semantic cosine → [0,1])
-- [ ] Weighted merge (0.65 keyword + 0.35 semantic, configurable)
-- [ ] Deduplication by document ID
-- [ ] `matched_by` metadata in results
+**Depends on:** API: Search Stabilization
 
-## API: Search Quality
-
-**Blocked on:** API: Search Stabilization
-
-- [ ] Field weight tuning based on real queries
+- [ ] Synonym expansion at query time (e.g. "repo" matches "repository")
+- [ ] Stemming tokenizer (porter or unicode61+porter)
+- [ ] Prefix search support for autocomplete
+- [ ] Field weight tuning based on real query patterns
 - [ ] Recency boost for recently updated content
 - [ ] Star count ranking signal (via Constellation)
 - [ ] State filtering defaults (exclude closed issues)

docs/specs/search.md

···
 Search now has two phases:
 
 1. Stabilize indexing and activity caching so search is cheap and reliable.
-2. Resume semantic and hybrid work only after the base pipeline is stable.
+2. Enhance keyword search quality with FTS5 features once the base pipeline is stable.
 
 ## Immediate Priority
 
···
 
 Production storage is Turso cloud. The reasoning is recorded in `docs/adr/storage.md`, with the comparison inputs in `docs/adr/pg.md` and `docs/adr/turso.md`.
 
-These tasks block further work on semantic and hybrid search.
+These tasks block further work on search quality improvements.
 
 ## Planning Decisions
 
···
 2. Direct API reads enqueue background indexing for misses.
 3. JetStream fills only the recent-activity cache.
 4. Smoke tests guard the critical paths.
-5. Semantic and hybrid search remain blocked until the base pipeline is stable.
+5. FTS5 quality improvements (synonyms, stemming, prefix search) follow once the base pipeline is stable.
 
 ## Backfill Strategy
 
···
 | Param        | Required | Default | Description                           |
 | ------------ | -------- | ------- | ------------------------------------- |
 | `q`          | Yes      | —       | Query string                          |
-| `mode`       | No       | keyword | keyword, semantic, or hybrid          |
+| `mode`       | No       | keyword | keyword                               |
 | `limit`      | No       | 20      | Results per page (1–100)              |
 | `offset`     | No       | 0       | Pagination offset                     |
 | `collection` | No       | —       | Filter by collection NSID             |
···
 
 4. **JetStream is for recent activity, not authoritative indexing.** Use it to power the cached feed, not to replace Tap or repo re-sync.
 
-5. **Semantic search is additive.** It improves discovery for vague queries but is not required for the app to be useful.
+5. **FTS5 enhancements are the next quality step.** Synonym expansion, stemming, and prefix search improve discovery without external dependencies.
 
 6. **Graceful degradation.** The mobile app treats the search API as optional. If Twister is unavailable, handle-based direct browsing still works. Search results link into the same browsing screens.
 
 ## Quality Improvements (Planned)
 
+- Synonym expansion at query time (e.g. "repo" matches "repository")
+- Stemming tokenizer (porter or unicode61+porter)
+- Prefix search support for autocomplete
 - Field weight tuning based on real query patterns
 - Recency boost for recently updated content
 - Collection-aware ranking

packages/api/.env.example

···
 SEARCH_MAX_LIMIT=100
 SEARCH_DEFAULT_MODE=keyword
 
-HYBRID_KEYWORD_WEIGHT=0.65
-HYBRID_SEMANTIC_WEIGHT=0.35
-
-# EMBEDDING_PROVIDER=openai
-# EMBEDDING_MODEL=text-embedding-3-small
-# EMBEDDING_API_KEY=sk-...
-# EMBEDDING_API_URL=
-# EMBEDDING_DIM=768
-# EMBEDDING_BATCH_SIZE=32
-
 HTTP_BIND_ADDR=:8080
 LOG_LEVEL=info
 LOG_FORMAT=json

packages/api/internal/api/api.go

···
 	mux.HandleFunc("GET /oauth/client-metadata.json", s.handleOAuthClientMetadata)
 	mux.HandleFunc("GET /search", s.handleSearch)
 	mux.HandleFunc("GET /search/keyword", s.handleSearchKeyword)
-	mux.HandleFunc("GET /search/semantic", s.handleNotImplemented)
-	mux.HandleFunc("GET /search/hybrid", s.handleNotImplemented)
 
 	mux.HandleFunc("GET /documents/{id}", s.handleGetDocument)
 	mux.HandleFunc("GET /profiles/{did}/summary", s.handleProfileSummary)
···
 
 	if s.cfg.EnableAdminEndpoints {
 		mux.HandleFunc("POST /admin/reindex", s.handleAdminReindex)
-		mux.HandleFunc("POST /admin/reembed", s.handleNotImplemented)
 	}
 
 	site := view.Handler()
···
 	switch mode {
 	case "keyword":
 		s.handleSearchKeyword(w, r)
-	case "semantic", "hybrid":
-		s.handleNotImplemented(w, r)
 	default:
-		writeJSON(w, http.StatusBadRequest, errorBody("invalid_parameter", "mode must be keyword, semantic, or hybrid"))
+		writeJSON(w, http.StatusBadRequest, errorBody("invalid_parameter", "mode must be keyword"))
 	}
 }
 
···
 	})
 }
 
-func (s *Server) handleNotImplemented(w http.ResponseWriter, _ *http.Request) {
-	writeJSON(w, http.StatusNotImplemented, errorBody("not_implemented", "this endpoint is not yet available"))
-}

packages/api/internal/api/readthrough.go

···
 		return fmt.Errorf("upsert document: %w", err)
 	}
 
-	if adapter.Searchable(record) {
-		if err := s.store.EnqueueEmbeddingJob(ctx, doc.ID); err != nil {
-			s.log.Warn("read-through enqueue embedding failed",
-				slog.String("document_id", doc.ID),
-				slog.String("error", err.Error()),
-			)
-		}
-	}
 	return nil
 }
 

packages/api/internal/config/config.go

···
 	SearchDefaultLimit     int
 	SearchMaxLimit         int
 	SearchDefaultMode      string
-	EmbeddingProvider      string
-	EmbeddingModel         string
-	EmbeddingAPIKey        string
-	EmbeddingAPIURL        string
-	EmbeddingDim           int
-	EmbeddingBatchSize     int
-	HybridKeywordWeight    float64
-	HybridSemanticWeight   float64
 	HTTPBindAddr           string
 	IndexerHealthAddr      string
 	LogLevel               string
···
 		TapAuthPassword:        os.Getenv("TAP_AUTH_PASSWORD"),
 		IndexedCollections:     os.Getenv("INDEXED_COLLECTIONS"),
 		SearchDefaultMode:      envOrDefault("SEARCH_DEFAULT_MODE", "keyword"),
-		EmbeddingProvider:      os.Getenv("EMBEDDING_PROVIDER"),
-		EmbeddingModel:         os.Getenv("EMBEDDING_MODEL"),
-		EmbeddingAPIKey:        os.Getenv("EMBEDDING_API_KEY"),
-		EmbeddingAPIURL:        os.Getenv("EMBEDDING_API_URL"),
 		HTTPBindAddr:           envOrDefault("HTTP_BIND_ADDR", ":8080"),
 		IndexerHealthAddr:      envOrDefault("INDEXER_HEALTH_ADDR", ":9090"),
 		LogLevel:               envOrDefault("LOG_LEVEL", "info"),
···
 		AdminAuthToken:         os.Getenv("ADMIN_AUTH_TOKEN"),
 		SearchDefaultLimit:     envInt("SEARCH_DEFAULT_LIMIT", 20),
 		SearchMaxLimit:         envInt("SEARCH_MAX_LIMIT", 100),
-		EmbeddingDim:           envInt("EMBEDDING_DIM", 768),
-		EmbeddingBatchSize:     envInt("EMBEDDING_BATCH_SIZE", 32),
-		HybridKeywordWeight:    envFloat("HYBRID_KEYWORD_WEIGHT", 0.65),
-		HybridSemanticWeight:   envFloat("HYBRID_SEMANTIC_WEIGHT", 0.35),
 		EnableAdminEndpoints:   envBool("ENABLE_ADMIN_ENDPOINTS", false),
 		EnableIngestEnrichment: envBool("ENABLE_INGEST_ENRICHMENT", true),
 		PLCDirectoryURL:        envOrDefault("PLC_DIRECTORY_URL", "https://plc.directory"),
···
 		return def
 	}
 	return n
-}
-
-func envFloat(key string, def float64) float64 {
-	v := os.Getenv(key)
-	if v == "" {
-		return def
-	}
-	f, err := strconv.ParseFloat(v, 64)
-	if err != nil {
-		return def
-	}
-	return f
 }
 
 func envDuration(key string, def time.Duration) time.Duration {

packages/api/internal/ingest/ingest.go

···
 		return err
 	}
 
-	if adapter.Searchable(record.Record) {
-		if err := r.store.EnqueueEmbeddingJob(ctx, doc.ID); err != nil {
-			r.log.Warn("embedding enqueue failed",
-				slog.Int64("event_id", event.ID),
-				slog.String("document_id", doc.ID),
-				slog.String("error", err.Error()),
-			)
-		}
-	}
-
 	return r.advanceCursorAndAck(ctx, event.ID)
 }
 

packages/api/internal/ingest/ingest_test.go

···
 	initialSync  *store.SyncState
 	recordStates map[string]string
 	handles      map[string]string
-	enqueued     map[string]bool
-	onSetSync    func()
+	onSetSync func()
 }
 
 func newFakeStore() *fakeStore {
···
 		docs:         make(map[string]*store.Document),
 		deleted:      make(map[string]bool),
 		recordStates: make(map[string]string),
-		handles:      make(map[string]string),
-		enqueued:     make(map[string]bool),
+		handles: make(map[string]string),
 	}
 }
 
···
 
 func (f *fakeStore) GetIdentityHandle(_ context.Context, did string) (string, error) {
 	return f.handles[did], nil
-}
-
-func (f *fakeStore) EnqueueEmbeddingJob(_ context.Context, documentID string) error {
-	f.enqueued[documentID] = true
-	return nil
 }
 
 func (f *fakeStore) EnqueueIndexingJob(_ context.Context, _ store.IndexingJobInput) error {
···
 	if doc.AuthorHandle != "author.tangled.org" {
 		t.Fatalf("author handle: got %q", doc.AuthorHandle)
 	}
-	if !st.enqueued[docID] {
-		t.Fatalf("embedding job not enqueued for %q", docID)
-	}
-
 	deleteEvent := normalize.TapRecordEvent{
 		ID:   202,
 		Type: "record",

packages/api/internal/store/db.go

···
 //go:embed migrations/*.sql
 var migrationsFS embed.FS
 
-var extensionMigrationNoticeLogged bool
-
 type migrationMode struct {
 	allowTursoExtensionSkip bool
 	targetDescription       string
···
 		if _, err := db.Exec(stmt); err != nil {
 			upper := strings.ToUpper(stmt)
 			if strings.Contains(upper, "LIBSQL_VECTOR_IDX") {
-				if !extensionMigrationNoticeLogged {
-					extensionMigrationNoticeLogged = true
-					slog.Info("migration: skipping unsupported extension index",
-						"migration", name,
-						"reason", "database engine does not support vector index DDL in this environment",
-					)
-				}
+				slog.Debug("migration: skipping unsupported vector index DDL",
+					"migration", name,
+				)
 				continue
 			}
 			if strings.Contains(upper, "CREATE VIRTUAL TABLE") && strings.Contains(upper, "USING FTS5") {

packages/api/internal/store/migrations/007_drop_embeddings.sql

···
+DROP TABLE IF EXISTS document_embeddings;
+DROP TABLE IF EXISTS embedding_jobs;

packages/api/internal/store/sql_store.go

···
 	return handle.String, nil
 }
 
-func (s *SQLStore) EnqueueEmbeddingJob(ctx context.Context, documentID string) error {
-	now := time.Now().UTC().Format(time.RFC3339)
-	_, err := s.db.ExecContext(ctx, `
-		INSERT INTO embedding_jobs (document_id, status, attempts, last_error, scheduled_at, updated_at)
-		VALUES (?, 'pending', 0, NULL, ?, ?)
-		ON CONFLICT(document_id) DO UPDATE SET
-			status = 'pending',
-			last_error = NULL,
-			scheduled_at = excluded.scheduled_at,
-			updated_at = excluded.updated_at`,
-		documentID, now, now,
-	)
-	if err != nil {
-		return fmt.Errorf("enqueue embedding job: %w", err)
-	}
-	return nil
-}
-
 func (s *SQLStore) EnqueueIndexingJob(ctx context.Context, input IndexingJobInput) error {
 	now := time.Now().UTC().Format(time.RFC3339)
 	_, err := s.db.ExecContext(ctx, `

packages/api/internal/store/store.go

···
 	UpdateRecordState(ctx context.Context, subjectURI string, state string) error
 	UpsertIdentityHandle(ctx context.Context, did, handle string, isActive bool, status string) error
 	GetIdentityHandle(ctx context.Context, did string) (string, error)
-	EnqueueEmbeddingJob(ctx context.Context, documentID string) error
 	EnqueueIndexingJob(ctx context.Context, input IndexingJobInput) error
 	ClaimIndexingJob(ctx context.Context) (*IndexingJob, error)
 	CompleteIndexingJob(ctx context.Context, documentID string) error

packages/api/internal/store/store_test.go

···
 
 import (
 	"context"
-	"database/sql"
 	"os"
 	"path/filepath"
 	"testing"
···
 		}
 		if handle != "alice2.tangled.org" {
 			t.Fatalf("handle after update: got %q, want %q", handle, "alice2.tangled.org")
-		}
-	})
-
-	t.Run("enqueue embedding job is idempotent", func(t *testing.T) {
-		doc := &store.Document{
-			ID:         "did:plc:embed|sh.tangled.string|abc",
-			DID:        "did:plc:embed",
-			Collection: "sh.tangled.string",
-			RKey:       "abc",
-			ATURI:      "at://did:plc:embed/sh.tangled.string/abc",
-			CID:        "bafyreienqueue",
-			RecordType: "string",
-			Title:      "foo.go",
-			Body:       "package main",
-		}
-		if err := st.UpsertDocument(ctx, doc); err != nil {
-			t.Fatalf("upsert doc for embedding queue: %v", err)
-		}
-
-		if err := st.EnqueueEmbeddingJob(ctx, doc.ID); err != nil {
-			t.Fatalf("enqueue embedding job: %v", err)
-		}
-		if err := st.EnqueueEmbeddingJob(ctx, doc.ID); err != nil {
-			t.Fatalf("enqueue embedding job second call: %v", err)
-		}
-
-		row := db.QueryRowContext(ctx, `SELECT status, attempts, last_error FROM embedding_jobs WHERE document_id = ?`, doc.ID)
-		var (
-			status    string
-			attempts  int
-			lastError sql.NullString
-		)
-		if err := row.Scan(&status, &attempts, &lastError); err != nil {
-			t.Fatalf("query embedding job: %v", err)
-		}
-		if status != "pending" {
-			t.Fatalf("status: got %q, want pending", status)
-		}
-		if attempts != 0 {
-			t.Fatalf("attempts: got %d, want 0", attempts)
-		}
-		if lastError.Valid {
-			t.Fatalf("last_error: got %q, want NULL", lastError.String)
 		}
 	})
 

packages/api/main.go

···
 		newAPICmd(&local),
 		newIndexerCmd(&local),
 		newBackfillCmd(&local),
-		newEmbedWorkerCmd(&local),
 		newReindexCmd(&local),
-		newReembedCmd(&local),
 		newEnrichCmd(&local),
 		newHealthcheckCmd(&local),
 	)
···
 	}
 }
 
-func newEmbedWorkerCmd(local *bool) *cobra.Command {
-	return &cobra.Command{
-		Use:   "embed-worker",
-		Short: "Start the async embedding worker",
-		RunE: func(cmd *cobra.Command, args []string) error {
-			cfg, err := config.Load(config.LoadOptions{Local: *local})
-			if err != nil {
-				return fmt.Errorf("config: %w", err)
-			}
-			log := observability.NewLogger(cfg)
-			log.Info("starting embed-worker", slog.String("service", "embed-worker"), slog.String("version", version))
-			ctx, cancel := baseContext()
-			defer cancel()
-			<-ctx.Done()
-			log.Info("shutting down embed-worker")
-			return nil
-		},
-	}
-}
-
 func newBackfillCmd(local *bool) *cobra.Command {
 	var opts backfill.Options
 
···
 	cmd.Flags().BoolVar(&opts.DryRun, "dry-run", false, "Show intended work without writing")
 
 	return cmd
-}
-
-func newReembedCmd(local *bool) *cobra.Command {
-	return &cobra.Command{
-		Use:   "reembed",
-		Short: "Re-generate all embeddings",
-		RunE: func(cmd *cobra.Command, args []string) error {
-			cfg, err := config.Load(config.LoadOptions{Local: *local})
-			if err != nil {
-				return fmt.Errorf("config: %w", err)
-			}
-			log := observability.NewLogger(cfg)
-			log.Info("reembed: not yet implemented")
-			return nil
-		},
-	}
 }
 
 func newEnrichCmd(local *bool) *cobra.Command {