Backlog #

Tracked issues, hardening items, and technical debt. Check items off as they are resolved.

---

Hardening (branch: hardening) #

Security and robustness improvements identified during code review (2026-04).

H1 — Replace eval() with ast.literal_eval() for file-pattern parsing (done) #

• Where: owilix/core/tasks/remote.py:177, owilix/core/tasks/query.py:65
• What: Both files use eval(files) to parse a string like "['**/*.parquet']" into a list. eval can execute arbitrary Python and raises confusing errors on invalid input.
• Fix: Created parse_files_pattern() in query_utils.py using ast.literal_eval() with fallback. Replaced all 13 eval(files) instances across query.py, remote.py, query_warc.py.
• Tests: 16 tests in tests/owilix/core/tasks/test_query_utils.py including code injection rejection.

H2 — Track and surface download failures in remote_pull() (done) #

• Where: owilix/core/tasks/remote.py:214–247
• What: Per-file exceptions are caught and printed but not accumulated. The function returns CommandResult(success=True) (line 247) even when individual files failed.
• Fix: Collect failed_files: list[tuple[str, str]] (path, error message). After all downloads, if non-empty: print a summary table and return success=False with ExitCode.DATA_ERROR.
• Also fix: In the sequential path, progress advance (line 245) is outside the try block — it runs even after an exception.
• Why it matters: Users and automation see exit code 0 on partial failure; silent data loss.

H3 — Extend CommandExecutor exception map (done) #

• Where: owilix/cli/_common/executor.py:73–86
• Fix: Added TimeoutError → NETWORK_ERROR and OSError → DATA_ERROR handlers. Ordered after ConnectionError/PermissionError/FileNotFoundError (all OSError subclasses) to preserve specificity.
• Tests: 6 tests in tests/owilix/cli/test_executor.py::TestExtendedExceptionMap covering OSError hierarchy.

H4 — Guard private method call in get_irods_client() (done) #

• Where: owilix/core/manager/session.py:109
• Fix: Wrapped _iRODS__check_access_token() in try/except AttributeError with warning log; client still returned when method is missing.
• Tests: 7 tests in tests/owilix/core/manager/test_session.py covering normal op, missing method, exception propagation, and properties.

H5 — Add HTTP timeout defaults #

• Where: owilix/core/warc/presign_client.py (has retry but no explicit timeout), owilix/core/fsspec/http2irods.py (IrodsHttpClient construction)
• What: No global HTTP timeout is set. A hung connection blocks indefinitely.
• Fix: Pass timeout=(10, 300) (connect, read) wherever HTTP sessions are created. For PresignClient, add timeout to the session adapter or to individual requests.get/post calls.
• Why it matters: Any network partition or unresponsive server hangs the CLI forever.

H6 — OWILRZZONE iRODS zone unreachable / empty JSON response from server #

• Where: owilix/core/fsspec/http2irods.py (find / listing path)
• Observed error:

  ERROR - find error on /OWILRZZONE/LEXIS/.../450feb34-...: Expecting value: line 1 column 1 (char 0)
  ERROR - Error listing /OWILRZZONE/LEXIS/...: /OWILRZZONE/LEXIS/...
  Found 0 remote files. Syncing with local files
  Dataset ... is already up to date. Missing files are 0.
  

• What: The iRODS HTTP endpoint for the OWILRZZONE zone returned an empty body (not valid JSON). The client silently treats this as "0 files found" and reports success. Root cause candidates:
  1. The LRZ zone endpoint is not reachable or not yet provisioned server-side (server is in dev mode)
  2. Zone URL or token routing to OWILRZZONE is misconfigured in the client
  3. The server returns an empty 200 body for the list call instead of a proper error status
• Fix (client-side, within our control):
  • Distinguish "empty JSON body" (server bug / connectivity gap) from "empty list" (legitimately no files)
  • When json.JSONDecodeError is raised on a listing response, log a clear error with the raw response body and surface it to the user as a connectivity failure, not "0 files"
  • Return CommandResult(success=False) with a message like "Zone OWILRZZONE returned no data — server may be unreachable or unprovisionened"
• Why it matters: Silent data loss — user believes dataset is synced when it isn't. Also blocks any LRZ zone usage.

H7 — owi remote doctor does not verify per-zone HTTP connectivity (done) #

• Where: owilix/core/tasks/remote.py (or wherever doctor is implemented), owilix/core/repository/lexis.py:status()
• Observed output:

  lexis: connected (lexis-http:LEXIS HTTP Repository (py4lexis 5.x))
          user=True   project=True    public=True
  py4lexis version:  5.0.1        ← stale after upgrade
  

• What: doctor only checks that the DDI metadata API is reachable (dataset listing). It does not:
  1. Probe the iRODS HTTP endpoint for each configured zone (e.g., OWILRZZONE, LRZZONE)
  2. Attempt a lightweight file listing on a known path to confirm the zone is actually accessible
  3. Report the correct py4lexis version (was showing 5.0.1 after upgrade to 5.4.0 — likely cached or hardcoded)
• Fix:
  • In doctor, after confirming DDI connectivity, attempt a fs.ls("/OWILRZZONE/") (or equivalent zone root ping) for each configured zone; report per-zone status as connected / unreachable / empty-response
  • Pull py4lexis version dynamically via importlib.metadata.version("py4lexis") instead of from config or hardcode
• Why it matters: Doctor is the first debugging tool users reach for. If it shows "connected" when a zone is broken, it actively misleads diagnosis (as seen with the OWILRZZONE issue above).

H7b — Doctor should probe irods_api/info endpoint per zone (done) #

• Where: owilix/core/repository/lexis.py:_probe_http_endpoint(), owilix/core/tasks/remote.py:remote_doctor()
• What: The iRODS HTTP API exposes a /info endpoint that returns server version, zone name, and API capabilities. Doctor currently only does a bare GET on the API root URL. Using /info gives richer diagnostics (correct zone name, server version, supported operations).
• Fix: In _probe_http_endpoint(), hit {api_url}/info instead of (or in addition to) the bare URL. Parse the JSON response and include the zone/version info in the doctor output.
• Why it matters: /info is the canonical health-check endpoint; it provides authoritative zone and version data that helps diagnose misconfiguration vs. server downtime.

H8 — Improve login user journey: B2ACCESS guidance + clearer token prompts (done) #

• Where: Readme.md, docs/source/install.md, owilix/core/manager/manager.py (_get_lexis_session())
• What: When the CLI needs a token (first run, expired token, refresh failure), the user gets a bare URL with no context. New users don't know they should use federated login via B2ACCESS.
• Fix (docs): Update Readme.md and install docs to tell users to log in via B2ACCESS using federated institutional login (not a local LEXIS account).
• Fix (CLI): When _get_lexis_session() falls through to the URL-based login, print a clear message:

  No valid token found. Please log in via the link below.
  Use your institutional account (federated login via B2ACCESS).
  

  Similarly, when a refresh token is used, print a brief note: Refreshing session token...
  When the refresh fails and a re-login is needed, print the B2ACCESS guidance instead of just the URL.
• Why it matters: First-time users and users whose tokens expire get dropped into a login flow with no guidance, leading to confusion and failed setups.

H9 — Integration tests must pass; installation must be validated #

• What: Integration tests (@pytest.mark.integration) and the install script (scripts/install.sh) must be verified to work with py4lexis 5.4.0 and python-http-irods-client 1.1.4.
• Scope:
  • Run uv run pytest tests/ -m integration and fix any failures caused by the py4lexis upgrade
  • Run scripts/install.sh in a clean venv and confirm it produces a working owi binary
  • Check that owi remote doctor reports the correct py4lexis version (5.4.0) after install
• Why it matters: The upgrade changed a transitive dependency (python-http-irods-client 1.1.2 → 1.1.4) which may alter HTTP client behaviour; untested breakage could reach users silently.

---

Mirror Operability (2026-08) #

Open items from the OpenWebSearch.eu ourrs deployment review campaign. The released fixes are in v5.6.0–v5.8.0 and recorded in docs/review/remote_pull_and_mirror_operability_2026-08-04.md; what follows is what is not fixed.

All three share one shape, which is worth stating once because it is the reason they belong together: an answer that is wrong in the direction that looks fine. The released review finding F2 found nothing and reported success; M1 fails a read and reports an empty store; M2 finds some and reports all; M3 lists a sidecar and implies data. Each is invisible to the consumer that would act on it.

Numbering: F1/F2/F5 inside the review documents are findings and are unrelated to F1/F2/F3 under Features below, which are backlog items. References to findings are written as "review F2" here.

See also H6, which is this same failure shape on the iRODS side: an empty JSON body from a zone endpoint is silently treated as "0 files found" and reported as success. Whatever convention M1 settles on for "a read failed, do not call this empty" should cover H6 too.

M1 — A read failure must not become an empty success (fixed in v5.9.0) #

• Report: docs/review/silent_empty_listing_stale_etag_2026-08-07.md — deterministic reproduction, root cause verified step by step.
• Status: Fixed and released in v5.9.0 — all five asks below. DatasetListing (owilix/core/listing.py) carries the verdict; list() tolerates unreadable sidecars; load_metadata passes an explicit size; status() reads one sidecar; collections are pruned before loading. Was live in production when filed: GET /remote/ls?specifier=owi-up returns {"count":0,"total":0,"has_more":false,"datasets":[]} with HTTP 200 against a bucket holding 591 datasets, while remote doctor simultaneously reports the repository healthy. Affects 5.6.0 and 5.8.0 alike — not a regression, the reporter checked git diff v5.6.0..v5.8.0 over core/repository/ and cleared 5.8.0 explicitly.
• Chain: with collections_in_path: true, FileBasedRepository.list() reads every *.json in every collection before Dataset.filter_datasets applies collectionName — so asking for one collection reads all of them. check_path_for_uuid_filename("latest") returns True, so an unrelated host-lifecycle/latest.json passes the UUID guard. s3fs sets IfMatch: <ETag> on ranged reads from fsspec's dircache, which does not expire by default; once another service rewrites that object the cached ETag is stale and the store answers 412 Precondition Failed. list() has no per-file tolerance, so one 424-byte file in a collection nobody asked for loses all 591 datasets.
• Fixes, in the reporter's priority order:
  1. Carry the outcome. One repository failed, it was the only one asked for, and the caller was told total: 0. Minimally {"total": 0, "partial": true, "failed": ["owi-up"]}. Independent of everything below, and the one that matters. Overlaps M2 — the same field probably serves both.
  2. Tolerate one bad sidecar. Collect per-file errors in load_metadata, return the rest, report the skipped count. On its own this turns the outage into a warning.
  3. Stop the stale ETag causing a 412. Cheapest: pass an explicit size= to fs.open() in load_metadata — the s3fs guard is size is None, and list() already has the size from the listing it just did. Alternatives: listings_expiry_time in get_fsspec_from_config, or invalidate_cache() and retry once on Precondition Failed.
  4. Make status() read something. It calls fs.exists() on three paths and nothing else, so it reports connected for a repository whose every listing fails. One load_metadata of one sidecar would catch it, for one small GET. remote doctor is a reachability check being read as a health check — see also H7.
  5. Do not read collections the caller did not ask for. Filtering after loading is a large amount of avoidable I/O on a big mirror, and it is what let an unrelated collection take out main.
• Why it matters: every consumer reads total: 0, has_more: false as "the mirror is empty". A shared bucket is a normal deployment, and "somebody rewrote a file" should not be able to empty a listing silently.

M2 — A listing must be able to say it is incomplete (listing half done in v5.9.0) #

• Proposal: docs/review/partial_listing_contract_2026-08-06.md — scoping the item left open under F3.
• Delivered in v5.9.0, as a side effect of M1 needing the same contract: DatasetListing carries complete and incomplete entries of {source, reason, message} with reason from the closed set the proposal asked for, and /remote/ls exposes partial / failed / incomplete. Field naming follows the bug report's partial/failed, with the proposal's incomplete array alongside, so both documents' consumers are served. complete defaults to true and partial is always present, so a caller that ignores them is unaffected.
• Still open — the remote pull half, which is the part the proposal flagged as most worth arguing about: remote pull does not yet fail on a partial discovery, and there is no --allow-partial. A partial listing is a display problem; a partial pull silently mirrors a subset. This is a behaviour change for existing callers and was deliberately not folded into the M1 fix release.
• What: when a specifier spans several repositories or LEXIS zones and one rejects the credential while another answers, the caller gets a shorter list and no indication anything was skipped. AggregatedRepository.list() re-raises only when the auth failure left nothing to return; a partial success is returned as though complete.
• Asked for: one bit and a reason — "complete": false plus "incomplete": [{"source": "lexis:OWILRZZONE", "reason": "auth", "message": "..."}], with reason a small closed set (auth | unreachable | timeout | error). complete defaults to true so existing callers keep working and absent reads as complete. For list(), the same information however is idiomatic — not an exception, because the useful case is the one where there is data to return.
• remote pull deserves a stronger default than remote ls, and this is the part worth arguing about: a partial listing is a display problem, a partial pull is a correctness problem. Proposed — remote pull fails on partial discovery with review F2's ErrorType.DATA treatment, and --allow-partial opts into transferring what is reachable. The reporter would run without it.
• Why it matters to the consumer: their catalog marks anything absent from the latest listing as missing_since and hides it. A credential lapse in one zone therefore retires that zone's collections labelled "no longer listed upstream" — a statement about the data when the truth was about the credential. They cannot defend against it because nothing in the response distinguishes the two.
• Cost: changes the /remote/ls response shape and the list() signature, so it should not ride in a bugfix release. Additive fields make it a minor bump; the remote pull default change is the genuinely behaviour-altering part and could ship separately or behind a flag first.

M3 — Verify a dataset, not just list it (observed vs declared) (delivered v5.10.1–v5.10.5) #

• Status: delivered across v5.10.1–v5.10.5, superseded by the summarize/integrity plan. All three tiers exist (owi remote verify --tier presence|readable|checksum), results are stored beside the data as checksums.json, iRODS checksums are read where available, and the collection index makes incremental re-verification possible. Four acceptance gaps remain — remote ls --observed, summarize --write-back, verify --since, and a per-collection verification endpoint — recorded in Part III of review/summarize_and_integrity_feature_plan_2026-08-07.md.
• Proposal: docs/review/dataset_integrity_verification_2026-08-07.md
• What: the mirror writes a ~4 KB sidecar per dataset and the parquet. remote ls counts a dataset as present when the sidecar exists — defensible, but it means a dataset whose transfer failed after the sidecar looks identical to one that completed. fileCount does not disambiguate: it comes from the sidecar and describes the upstream dataset, so one reported fileCount 567, 14.7 GB against a single 4 KB object. Their catalog told visitors 586 collections were mirrored when 4 held data.
• Three tiers, in cost order:
  1. Presence — declared (from the sidecar) alongside observed (from the store) plus a materialised flag. This is a listing, not a read; it costs what ls already costs and would have caught the 586-vs-4. If only one tier ships, this is the one.
  2. Readability — read one row group of one part per dataset, weekly. Presence does not prove usability; a truncated parquet is present.
  3. Checksums — full hashing of ~51 TiB is about a day of pure I/O, so not nightly. Cheaper than it looks: iRODS already exposes a per-object checksum which irodsfsspec.py surfaces, so LEXIS-backed data may be readable rather than recomputable, and only changed objects need rehashing on an incremental pass. A dataset-level digest over the sorted per-file checksums is what a consumer actually wants.
• Builds on what exists: remote summarize already walks datasets, computes statistics and writes README.md/stats.json back; remote diff already compares. The missing dimension is integrity — there is no checksum of a data object anywhere in the codebase today.
• Explicitly out of scope: writing fixity results back into the upstream LEXIS dataset. That needs write access to someone else's authoritative copy — a different trust boundary. Discrepancies get reported; a human decides. Results belong in the mirror as a sibling of stats.json, readable over HTTP so a batch job can consume them without scraping console output (the review F5 lesson).

---

Features #

F3 — Workflow execution and workflow statistics #

Planned direction: drive workflow execution and report workflow statistics through owilix. Epic entry in docs/epics.md. Not yet scoped — the notes below are what the repository already contains, so that scoping starts from the actual state.

Execution — partly exists. owi batch run <specifier> <taskfile.yaml> already executes YAML-defined tasks against datasets matching a specifier, with --workers, --keep-going, --resume, --job, --reset and --dry-run (owilix/cli/batch.py). It is a thin Typer shim delegating to owilix.compat.batch.BatchCommands. It is not exposed over HTTP — there is no /batch/* endpoint — so the HTTP service cannot currently run one.

Statistics — exists, but stranded. owilix/compat/workflows.py is a 963-line Airflow 2.8 REST client (AirflowMonitor, WorkflowCommands) with dags, dag_runs and summary, summarising executions by operator and DAG with state counts and duration stats (avg, p95). It is not registered in the modern Typer CLI, not exposed over HTTP, and has no tests — owi workflow does not exist as a command. It sits in compat/, the legacy layer that T1 renamed from cmd/.

First decision, before any design: revive-and-port compat/workflows.py, or delete it and start from the current architecture. It encodes a real data model for workflow statistics and a set of assumptions about Airflow; both are worth reading before choosing. Whichever way, it should not stay unregistered and untested in compat/.

Open questions that change the shape of the work:

• Is "workflow" here Airflow DAGs (what compat/workflows.py assumes), owilix's own batch run task files, or both behind one surface?
• Do statistics describe owilix's executions (which JobManager already has the raw material for: operation, status, created_at/started_at/ finished_at, logs) or an external orchestrator's?
• Execution over HTTP would need durable jobs — currently in-process only and lost on restart, already tracked as the first item of F2.

Dependency worth stating plainly: workflow statistics computed over listings that can silently be partial (M2), silently empty (M1), or count a sidecar as a materialised dataset (M3) will be confidently wrong in exactly the way this review campaign has spent three releases removing. A statistics surface built on top of them inherits every one of those failure modes and adds a dashboard-shaped veneer of authority. M1 at minimum should land first.

F2 — HTTP API production hardening backlog #

Items intentionally deferred from feature/http-server while the first HTTP command surface is built:

• Durable jobs: Persist job state across server restarts, support multiple worker processes, add cancellation, and expose structured progress counters beyond captured logs.
• Security model: Add an explicit HTTP API authentication/authorization model for OWILIX endpoints, separate from local refresh-token presence and the optional /s3 SigV4 facade.
• Remote list performance: Avoid full backend fetches for paginated remote/ls; options include repository-level pagination, DDI offset support through py4lexis, or HTTP-side stale-while-refresh caching.
• HTTP-side listing cache: Return cached remote/ls data immediately, refresh stale entries in the background, and support request flags for forced live refresh or cached-only responses.
• Integration tests: Add authenticated end-to-end tests for HTTP pull, remote search, /s3 with SigV4 clients, and selected query endpoints.

F1 — Remote full-text search via DuckLake / Remote Index #

Detailed plan: docs/plans/remote-search.md

The OWI remote index (~550M documents) is stored as partitioned Parquet on S3 (Allas, s3://2006391-owi-remote-index). Using DuckLake (DuckDB >= 1.5), it can be queried as two virtual tables (remote_index, documents) with partition pruning and ranking via sum(log(tf)).

Phases:

• Phase 1 — Test script: tests/scripts/remote_index_test.py — validate DuckLake connection from Python, run a simple ranked search, measure latency. Also serves as integration test. DuckDB bumped to >= 1.5.0.
• Phase 2 — owi remote search: New CLI command for full-text search against the remote index. owi remote search "open web search" --language deu --limit 10. Supports --all (conjunctive), --repr, --explain, table/json/jsonl output.
• Phase 3 — owi query less --search: --search option on less and slice commands. Search narrows files + adds id IN (...) filter. Includes id-only file selection to minimize parquet scans.
• Phase 4 — Fetch/export matching records: owi remote search --fetch retrieves full document records via the query pipeline. Supports --select, --where, --output (parquet/json/jsonl/csv).

Prerequisites: DuckDB >= 1.5 (currently 1.4.3), DuckLake extension, S3 secret for Allas endpoint (a3s.fi).

---

Technical Debt #

Lower-priority items; not blocking current work.

T1 — Remove or migrate legacy cmd/ (Click-based CLI) (done) #

• What: Renamed owilix/cmd/ → owilix/compat/. Migrated render_table_dynamic() to core/manager/ui.py. Fixed all core/tasks imports to remove cmd/ references. Updated plugins and appkernel. Deleted legacy Click tests. Net -1,505 lines.

T2 — Structured logging (JSON) for automation #

• Where: owilix/core/manager/logging.py
• What: Logs are human-readable text. Automation pipelines and log aggregators benefit from JSON-structured logs.
• Action: Add an optional --log-format json flag that switches the root logger's formatter.

T3 — Periodic token refresh during long operations #

• Where: owilix/core/manager/session.py, owilix/core/tasks/remote.py
• What: Access tokens expire; large downloads can outlive them. Token is only refreshed at get_irods_client() call time.
• Action: Add a background thread or periodic check (e.g., every 5 min) that calls the refresh method during active download operations.

T4 — Resume / checkpoint for partial downloads #

• Where: owilix/core/tasks/remote.py
• What: If a download is interrupted, it restarts from scratch. The missing-files calculation already computes a delta, but there is no state file for in-progress batches.
• Action: Persist the list of remaining files to a .owi-pull-checkpoint file; re-read it on retry.

T5 — Document plugin system #

• Where: owilix/plugins/, owilix/cli/plugin.py
• What: The plugin system is implemented but not documented for users or contributors.
• Action: Add docs/source/plugins.md describing the plugin API and the ngram / push examples.