# Backlog

Tracked issues, hardening items, and technical debt. Check items off as they are resolved.

---

## Hardening (branch: `hardening`)

Security and robustness improvements identified during code review (2026-04).

### ~~H1~~ — Replace `eval()` with `ast.literal_eval()` for file-pattern parsing (done)

- **Where:** `owilix/core/tasks/remote.py:177`, `owilix/core/tasks/query.py:65`
- **What:** Both files use `eval(files)` to parse a string like `"['**/*.parquet']"` into a list. `eval` can execute arbitrary Python and raises confusing errors on invalid input.
- **Fix:** Created `parse_files_pattern()` in `query_utils.py` using `ast.literal_eval()` with fallback. Replaced all 13 `eval(files)` instances across `query.py`, `remote.py`, `query_warc.py`.
- **Tests:** 16 tests in `tests/owilix/core/tasks/test_query_utils.py` including code injection rejection.

### ~~H2~~ — Track and surface download failures in `remote_pull()` (done)

- **Where:** `owilix/core/tasks/remote.py:214–247`
- **What:** Per-file exceptions are caught and printed but not accumulated. The function returns `CommandResult(success=True)` (line 247) even when individual files failed.
- **Fix:** Collect `failed_files: list[tuple[str, str]]` (path, error message). After all downloads, if non-empty: print a summary table and return `success=False` with `ExitCode.DATA_ERROR`.
- **Also fix:** In the sequential path, progress advance (line 245) is outside the `try` block — it runs even after an exception.
- **Why it matters:** Users and automation see exit code 0 on partial failure; silent data loss.

### ~~H3~~ — Extend `CommandExecutor` exception map (done)

- **Where:** `owilix/cli/_common/executor.py:73–86`
- **Fix:** Added `TimeoutError` → `NETWORK_ERROR` and `OSError` → `DATA_ERROR` handlers. Ordered after `ConnectionError`/`PermissionError`/`FileNotFoundError` (all OSError subclasses) to preserve specificity.
- **Tests:** 6 tests in `tests/owilix/cli/test_executor.py::TestExtendedExceptionMap` covering OSError hierarchy.

### ~~H4~~ — Guard private method call in `get_irods_client()` (done)

- **Where:** `owilix/core/manager/session.py:109`
- **Fix:** Wrapped `_iRODS__check_access_token()` in `try/except AttributeError` with warning log; client still returned when method is missing.
- **Tests:** 7 tests in `tests/owilix/core/manager/test_session.py` covering normal op, missing method, exception propagation, and properties.

### H5 — Add HTTP timeout defaults

- **Where:** `owilix/core/warc/presign_client.py` (has retry but no explicit `timeout`), `owilix/core/fsspec/http2irods.py` (IrodsHttpClient construction)
- **What:** No global HTTP timeout is set. A hung connection blocks indefinitely.
- **Fix:** Pass `timeout=(10, 300)` (connect, read) wherever HTTP sessions are created. For `PresignClient`, add `timeout` to the session adapter or to individual `requests.get/post` calls.
- **Why it matters:** Any network partition or unresponsive server hangs the CLI forever.

### H6 — OWILRZZONE iRODS zone unreachable / empty JSON response from server

- **Where:** `owilix/core/fsspec/http2irods.py` (`find` / listing path)
- **Observed error:**
  ```
  ERROR - find error on /OWILRZZONE/LEXIS/.../450feb34-...: Expecting value: line 1 column 1 (char 0)
  ERROR - Error listing /OWILRZZONE/LEXIS/...: /OWILRZZONE/LEXIS/...
  Found 0 remote files. Syncing with local files
  Dataset ... is already up to date. Missing files are 0.
  ```
- **What:** The iRODS HTTP endpoint for the `OWILRZZONE` zone returned an empty body (not valid JSON). The client silently treats this as "0 files found" and reports success. Root cause candidates:
  1. The LRZ zone endpoint is not reachable or not yet provisioned server-side (server is in dev mode)
  2. Zone URL or token routing to `OWILRZZONE` is misconfigured in the client
  3. The server returns an empty 200 body for the list call instead of a proper error status
- **Fix (client-side, within our control):**
  - Distinguish "empty JSON body" (server bug / connectivity gap) from "empty list" (legitimately no files)
  - When `json.JSONDecodeError` is raised on a listing response, log a clear error with the raw response body and surface it to the user as a connectivity failure, not "0 files"
  - Return `CommandResult(success=False)` with a message like "Zone OWILRZZONE returned no data — server may be unreachable or unprovisionened"
- **Why it matters:** Silent data loss — user believes dataset is synced when it isn't. Also blocks any LRZ zone usage.

### ~~H7 — `owi remote doctor` does not verify per-zone HTTP connectivity~~ (done)

- **Where:** `owilix/core/tasks/remote.py` (or wherever `doctor` is implemented), `owilix/core/repository/lexis.py:status()`
- **Observed output:**
  ```
  lexis: connected (lexis-http:LEXIS HTTP Repository (py4lexis 5.x))
          user=True   project=True    public=True
  py4lexis version:  5.0.1        ← stale after upgrade
  ```
- **What:** `doctor` only checks that the DDI metadata API is reachable (dataset listing). It does not:
  1. Probe the iRODS HTTP endpoint for each configured zone (e.g., `OWILRZZONE`, `LRZZONE`)
  2. Attempt a lightweight file listing on a known path to confirm the zone is actually accessible
  3. Report the correct `py4lexis` version (was showing `5.0.1` after upgrade to `5.4.0` — likely cached or hardcoded)
- **Fix:**
  - In `doctor`, after confirming DDI connectivity, attempt a `fs.ls("/OWILRZZONE/")` (or equivalent zone root ping) for each configured zone; report per-zone status as `connected` / `unreachable` / `empty-response`
  - Pull `py4lexis` version dynamically via `importlib.metadata.version("py4lexis")` instead of from config or hardcode
- **Why it matters:** Doctor is the first debugging tool users reach for. If it shows "connected" when a zone is broken, it actively misleads diagnosis (as seen with the OWILRZZONE issue above).

### ~~H7b~~ — Doctor should probe `irods_api/info` endpoint per zone (done)

- **Where:** `owilix/core/repository/lexis.py:_probe_http_endpoint()`, `owilix/core/tasks/remote.py:remote_doctor()`
- **What:** The iRODS HTTP API exposes a `/info` endpoint that returns server version, zone name, and API capabilities. Doctor currently only does a bare `GET` on the API root URL. Using `/info` gives richer diagnostics (correct zone name, server version, supported operations).
- **Fix:** In `_probe_http_endpoint()`, hit `{api_url}/info` instead of (or in addition to) the bare URL. Parse the JSON response and include the zone/version info in the doctor output.
- **Why it matters:** `/info` is the canonical health-check endpoint; it provides authoritative zone and version data that helps diagnose misconfiguration vs. server downtime.

### ~~H8 — Improve login user journey: B2ACCESS guidance + clearer token prompts~~ (done)

- **Where:** `Readme.md`, `docs/source/install.md`, `owilix/core/manager/manager.py` (`_get_lexis_session()`)
- **What:** When the CLI needs a token (first run, expired token, refresh failure), the user gets a bare URL with no context. New users don't know they should use **federated login via B2ACCESS**.
- **Fix (docs):** Update `Readme.md` and install docs to tell users to log in via B2ACCESS using federated institutional login (not a local LEXIS account).
- **Fix (CLI):** When `_get_lexis_session()` falls through to the URL-based login, print a clear message:
  ```
  No valid token found. Please log in via the link below.
  Use your institutional account (federated login via B2ACCESS).
  ```
  Similarly, when a refresh token is used, print a brief note: `Refreshing session token...`  
  When the refresh fails and a re-login is needed, print the B2ACCESS guidance instead of just the URL.
- **Why it matters:** First-time users and users whose tokens expire get dropped into a login flow with no guidance, leading to confusion and failed setups.

### H9 — Integration tests must pass; installation must be validated

- **What:** Integration tests (`@pytest.mark.integration`) and the install script (`scripts/install.sh`) must be verified to work with py4lexis 5.4.0 and `python-http-irods-client` 1.1.4.
- **Scope:**
  - Run `uv run pytest tests/ -m integration` and fix any failures caused by the py4lexis upgrade
  - Run `scripts/install.sh` in a clean venv and confirm it produces a working `owi` binary
  - Check that `owi remote doctor` reports the correct py4lexis version (5.4.0) after install
- **Why it matters:** The upgrade changed a transitive dependency (`python-http-irods-client` 1.1.2 → 1.1.4) which may alter HTTP client behaviour; untested breakage could reach users silently.

---

## Mirror Operability (2026-08)

Open items from the OpenWebSearch.eu `ourrs` deployment review campaign. The
released fixes are in v5.6.0–v5.8.0 and recorded in
[`docs/review/remote_pull_and_mirror_operability_2026-08-04.md`](review/remote_pull_and_mirror_operability_2026-08-04.md);
what follows is what is *not* fixed.

All three share one shape, which is worth stating once because it is the reason
they belong together: **an answer that is wrong in the direction that looks
fine.** The released review finding F2 found nothing and reported success; M1
fails a read and reports an empty store; M2 finds some and reports all; M3 lists
a sidecar and implies data. Each is invisible to the consumer that would act on
it.

> **Numbering:** `F1`/`F2`/`F5` inside the review documents are *findings* and
> are unrelated to `F1`/`F2`/`F3` under **Features** below, which are backlog
> items. References to findings are written as "review F2" here.

See also **H6**, which is this same failure shape on the iRODS side: an empty
JSON body from a zone endpoint is silently treated as "0 files found" and
reported as success. Whatever convention M1 settles on for "a read failed, do
not call this empty" should cover H6 too.

### ~~M1 — A read failure must not become an empty success~~ (fixed in v5.9.0)

- **Report:** [`docs/review/silent_empty_listing_stale_etag_2026-08-07.md`](review/silent_empty_listing_stale_etag_2026-08-07.md) — deterministic reproduction, root cause verified step by step.
- **Status:** **Fixed and released in v5.9.0** — all five asks below. `DatasetListing` (`owilix/core/listing.py`) carries the verdict; `list()` tolerates unreadable sidecars; `load_metadata` passes an explicit `size`; `status()` reads one sidecar; collections are pruned before loading. Was live in production when filed: `GET /remote/ls?specifier=owi-up` returns `{"count":0,"total":0,"has_more":false,"datasets":[]}` with HTTP 200 against a bucket holding 591 datasets, while `remote doctor` simultaneously reports the repository healthy. Affects 5.6.0 and 5.8.0 alike — **not a regression**, the reporter checked `git diff v5.6.0..v5.8.0` over `core/repository/` and cleared 5.8.0 explicitly.
- **Chain:** with `collections_in_path: true`, `FileBasedRepository.list()` reads *every* `*.json` in *every* collection before `Dataset.filter_datasets` applies `collectionName` — so asking for one collection reads all of them. `check_path_for_uuid_filename("latest")` returns `True`, so an unrelated `host-lifecycle/latest.json` passes the UUID guard. s3fs sets `IfMatch: <ETag>` on ranged reads from fsspec's `dircache`, which does not expire by default; once another service rewrites that object the cached ETag is stale and the store answers **412 Precondition Failed**. `list()` has no per-file tolerance, so one 424-byte file in a collection nobody asked for loses all 591 datasets.
- **Fixes, in the reporter's priority order:**
  1. **Carry the outcome.** One repository failed, it was the only one asked for, and the caller was told `total: 0`. Minimally `{"total": 0, "partial": true, "failed": ["owi-up"]}`. Independent of everything below, and the one that matters. Overlaps M2 — the same field probably serves both.
  2. **Tolerate one bad sidecar.** Collect per-file errors in `load_metadata`, return the rest, report the skipped count. On its own this turns the outage into a warning.
  3. **Stop the stale ETag causing a 412.** Cheapest: pass an explicit `size=` to `fs.open()` in `load_metadata` — the s3fs guard is `size is None`, and `list()` already has the size from the listing it just did. Alternatives: `listings_expiry_time` in `get_fsspec_from_config`, or `invalidate_cache()` and retry once on `Precondition Failed`.
  4. **Make `status()` read something.** It calls `fs.exists()` on three paths and nothing else, so it reports `connected` for a repository whose every listing fails. One `load_metadata` of one sidecar would catch it, for one small GET. `remote doctor` is a reachability check being read as a health check — see also H7.
  5. **Do not read collections the caller did not ask for.** Filtering after loading is a large amount of avoidable I/O on a big mirror, and it is what let an unrelated collection take out `main`.
- **Why it matters:** every consumer reads `total: 0, has_more: false` as "the mirror is empty". A shared bucket is a normal deployment, and "somebody rewrote a file" should not be able to empty a listing silently.

### M2 — A listing must be able to say it is incomplete (**listing half done in v5.9.0**)

- **Proposal:** [`docs/review/partial_listing_contract_2026-08-06.md`](review/partial_listing_contract_2026-08-06.md) — scoping the item left open under F3.
- **Delivered in v5.9.0**, as a side effect of M1 needing the same contract: `DatasetListing` carries `complete` and `incomplete` entries of `{source, reason, message}` with `reason` from the closed set the proposal asked for, and `/remote/ls` exposes `partial` / `failed` / `incomplete`. Field naming follows the bug report's `partial`/`failed`, with the proposal's `incomplete` array alongside, so both documents' consumers are served. `complete` defaults to true and `partial` is always present, so a caller that ignores them is unaffected.
- **Still open — the `remote pull` half**, which is the part the proposal flagged as most worth arguing about: `remote pull` does not yet fail on a *partial* discovery, and there is no `--allow-partial`. A partial listing is a display problem; a partial pull silently mirrors a subset. This is a behaviour change for existing callers and was deliberately not folded into the M1 fix release.
- **What:** when a specifier spans several repositories or LEXIS zones and one rejects the credential while another answers, the caller gets a shorter list and **no indication anything was skipped**. `AggregatedRepository.list()` re-raises only when the auth failure left nothing to return; a partial success is returned as though complete.
- **Asked for:** one bit and a reason — `"complete": false` plus `"incomplete": [{"source": "lexis:OWILRZZONE", "reason": "auth", "message": "..."}]`, with `reason` a small closed set (`auth | unreachable | timeout | error`). `complete` defaults to `true` so existing callers keep working and absent reads as complete. For `list()`, the same information however is idiomatic — **not** an exception, because the useful case is the one where there *is* data to return.
- **`remote pull` deserves a stronger default than `remote ls`,** and this is the part worth arguing about: a partial listing is a display problem, a partial pull is a correctness problem. Proposed — `remote pull` fails on partial discovery with review F2's `ErrorType.DATA` treatment, and `--allow-partial` opts into transferring what is reachable. The reporter would run without it.
- **Why it matters to the consumer:** their catalog marks anything absent from the latest listing as `missing_since` and hides it. A credential lapse in one zone therefore retires that zone's collections labelled *"no longer listed upstream"* — a statement about the data when the truth was about the credential. They cannot defend against it because nothing in the response distinguishes the two.
- **Cost:** changes the `/remote/ls` response shape and the `list()` signature, so it should not ride in a bugfix release. Additive fields make it a minor bump; the `remote pull` default change is the genuinely behaviour-altering part and could ship separately or behind a flag first.

### ~~M3 — Verify a dataset, not just list it (`observed` vs `declared`)~~ (delivered v5.10.1–v5.10.5)

- **Status:** delivered across v5.10.1–v5.10.5, superseded by the summarize/integrity plan. All three tiers exist (`owi remote verify --tier presence|readable|checksum`), results are stored beside the data as `checksums.json`, iRODS checksums are read where available, and the collection index makes incremental re-verification possible. **Four acceptance gaps remain** — `remote ls --observed`, `summarize --write-back`, `verify --since`, and a per-collection verification endpoint — recorded in Part III of [`review/summarize_and_integrity_feature_plan_2026-08-07.md`](review/summarize_and_integrity_feature_plan_2026-08-07.md).
- **Proposal:** [`docs/review/dataset_integrity_verification_2026-08-07.md`](review/dataset_integrity_verification_2026-08-07.md)
- **What:** the mirror writes a ~4 KB sidecar per dataset *and* the parquet. `remote ls` counts a dataset as present when the sidecar exists — defensible, but it means **a dataset whose transfer failed after the sidecar looks identical to one that completed**. `fileCount` does not disambiguate: it comes from the sidecar and describes the *upstream* dataset, so one reported `fileCount 567, 14.7 GB` against a single 4 KB object. Their catalog told visitors 586 collections were mirrored when 4 held data.
- **Three tiers, in cost order:**
  1. **Presence** — `declared` (from the sidecar) alongside `observed` (from the store) plus a `materialised` flag. This is a listing, not a read; it costs what `ls` already costs and would have caught the 586-vs-4. **If only one tier ships, this is the one.**
  2. **Readability** — read one row group of one part per dataset, weekly. Presence does not prove usability; a truncated parquet is present.
  3. **Checksums** — full hashing of ~51 TiB is about a day of pure I/O, so not nightly. Cheaper than it looks: iRODS already exposes a per-object `checksum` which `irodsfsspec.py` surfaces, so LEXIS-backed data may be *readable* rather than recomputable, and only changed objects need rehashing on an incremental pass. A dataset-level digest over the sorted per-file checksums is what a consumer actually wants.
- **Builds on what exists:** `remote summarize` already walks datasets, computes statistics and writes `README.md`/`stats.json` back; `remote diff` already compares. The missing dimension is integrity — there is no checksum of a data object anywhere in the codebase today.
- **Explicitly out of scope:** writing fixity results back into the upstream LEXIS dataset. That needs write access to someone else's authoritative copy — a different trust boundary. Discrepancies get *reported*; a human decides. Results belong in the mirror as a sibling of `stats.json`, readable over HTTP so a batch job can consume them without scraping console output (the review F5 lesson).

---

## Features

### F3 — Workflow execution and workflow statistics

Planned direction: drive workflow **execution** and report workflow
**statistics** through owilix. Epic entry in
[`docs/epics.md`](epics.md). **Not yet scoped — the notes below are what the
repository already contains, so that scoping starts from the actual state.**

**Execution — partly exists.** `owi batch run <specifier> <taskfile.yaml>`
already executes YAML-defined tasks against datasets matching a specifier, with
`--workers`, `--keep-going`, `--resume`, `--job`, `--reset` and `--dry-run`
(`owilix/cli/batch.py`). It is a thin Typer shim delegating to
`owilix.compat.batch.BatchCommands`. It is **not** exposed over HTTP — there is
no `/batch/*` endpoint — so the HTTP service cannot currently run one.

**Statistics — exists, but stranded.** `owilix/compat/workflows.py` is a
963-line Airflow 2.8 REST client (`AirflowMonitor`, `WorkflowCommands`) with
`dags`, `dag_runs` and `summary`, summarising executions by operator and DAG
with state counts and duration stats (avg, p95). It is **not registered in the
modern Typer CLI, not exposed over HTTP, and has no tests** — `owi workflow`
does not exist as a command. It sits in `compat/`, the legacy layer that T1
renamed from `cmd/`.

**First decision, before any design:** revive-and-port `compat/workflows.py`, or
delete it and start from the current architecture. It encodes a real data model
for workflow statistics and a set of assumptions about Airflow; both are worth
reading before choosing. Whichever way, it should not stay unregistered and
untested in `compat/`.

**Open questions that change the shape of the work:**
- Is "workflow" here Airflow DAGs (what `compat/workflows.py` assumes), owilix's
  own `batch run` task files, or both behind one surface?
- Do statistics describe *owilix's* executions (which `JobManager` already has
  the raw material for: `operation`, `status`, `created_at`/`started_at`/
  `finished_at`, `logs`) or an external orchestrator's?
- Execution over HTTP would need durable jobs — currently in-process only and
  lost on restart, already tracked as the first item of F2.

**Dependency worth stating plainly:** workflow statistics computed over listings
that can silently be partial (M2), silently empty (M1), or count a sidecar as a
materialised dataset (M3) will be **confidently wrong in exactly the way this
review campaign has spent three releases removing**. A statistics surface built
on top of them inherits every one of those failure modes and adds a
dashboard-shaped veneer of authority. M1 at minimum should land first.

### F2 — HTTP API production hardening backlog

Items intentionally deferred from `feature/http-server` while the first HTTP command surface is built:

- **Durable jobs:** Persist job state across server restarts, support multiple worker processes, add cancellation, and expose structured progress counters beyond captured logs.
- **Security model:** Add an explicit HTTP API authentication/authorization model for OWILIX endpoints, separate from local refresh-token presence and the optional `/s3` SigV4 facade.
- **Remote list performance:** Avoid full backend fetches for paginated `remote/ls`; options include repository-level pagination, DDI offset support through py4lexis, or HTTP-side stale-while-refresh caching.
- **HTTP-side listing cache:** Return cached `remote/ls` data immediately, refresh stale entries in the background, and support request flags for forced live refresh or cached-only responses.
- **Integration tests:** Add authenticated end-to-end tests for HTTP pull, remote search, `/s3` with SigV4 clients, and selected query endpoints.

### F1 — Remote full-text search via DuckLake / Remote Index

**Detailed plan:** [`docs/plans/remote-search.md`](plans/remote-search.md)

The OWI remote index (~550M documents) is stored as partitioned Parquet on S3 (Allas, `s3://2006391-owi-remote-index`). Using DuckLake (DuckDB >= 1.5), it can be queried as two virtual tables (`remote_index`, `documents`) with partition pruning and ranking via `sum(log(tf))`.

**Phases:**

- [x] **Phase 1 — Test script**: `tests/scripts/remote_index_test.py` — validate DuckLake connection from Python, run a simple ranked search, measure latency. Also serves as integration test. DuckDB bumped to >= 1.5.0.
- [x] **Phase 2 — `owi remote search`**: New CLI command for full-text search against the remote index. `owi remote search "open web search" --language deu --limit 10`. Supports `--all` (conjunctive), `--repr`, `--explain`, table/json/jsonl output.
- [x] **Phase 3 — `owi query less --search`**: `--search` option on `less` and `slice` commands. Search narrows files + adds `id IN (...)` filter. Includes id-only file selection to minimize parquet scans.
- [x] **Phase 4 — Fetch/export matching records**: `owi remote search --fetch` retrieves full document records via the query pipeline. Supports `--select`, `--where`, `--output` (parquet/json/jsonl/csv).

**Prerequisites:** DuckDB >= 1.5 (currently 1.4.3), DuckLake extension, S3 secret for Allas endpoint (`a3s.fi`).

---

## Technical Debt

Lower-priority items; not blocking current work.

### ~~T1~~ — Remove or migrate legacy `cmd/` (Click-based CLI) (done)

- **What:** Renamed `owilix/cmd/` → `owilix/compat/`. Migrated `render_table_dynamic()` to `core/manager/ui.py`. Fixed all core/tasks imports to remove cmd/ references. Updated plugins and appkernel. Deleted legacy Click tests. Net -1,505 lines.

### T2 — Structured logging (JSON) for automation

- **Where:** `owilix/core/manager/logging.py`
- **What:** Logs are human-readable text. Automation pipelines and log aggregators benefit from JSON-structured logs.
- **Action:** Add an optional `--log-format json` flag that switches the root logger's formatter.

### T3 — Periodic token refresh during long operations

- **Where:** `owilix/core/manager/session.py`, `owilix/core/tasks/remote.py`
- **What:** Access tokens expire; large downloads can outlive them. Token is only refreshed at `get_irods_client()` call time.
- **Action:** Add a background thread or periodic check (e.g., every 5 min) that calls the refresh method during active download operations.

### T4 — Resume / checkpoint for partial downloads

- **Where:** `owilix/core/tasks/remote.py`
- **What:** If a download is interrupted, it restarts from scratch. The missing-files calculation already computes a delta, but there is no state file for in-progress batches.
- **Action:** Persist the list of remaining files to a `.owi-pull-checkpoint` file; re-read it on retry.

### T5 — Document plugin system

- **Where:** `owilix/plugins/`, `owilix/cli/plugin.py`
- **What:** The plugin system is implemented but not documented for users or contributors.
- **Action:** Add `docs/source/plugins.md` describing the plugin API and the `ngram` / `push` examples.