# Owilix Changes Human-curated changelog tracking significant changes across branches. --- ## Format Each entry follows: ``` ## [Branch] @ [commit-hash] - YYYY-MM-DD ### Summary Brief description of what was accomplished. ### Changes - List of specific changes ### Breaking Changes (if any) - List of breaking changes ``` --- ## Unreleased ### main @ v5.10.9 - 2026-08-08 #### The descriptive record, and two accessors that lied about it **Summary**: Asks 1, 2 and 3a/3b from the openwebindex.eu dashboard. Each was reproduced against live data before being changed. **Changes**: - `?include=metadata` on `/remote/ls` and `/local/ls`, sourced from `as_json_dict()`. Verified against the real `German Commons` record: 8 creators with ORCIDs where `get()` said `Lukas Gienapp+7`, a 1,385-character description where `get()` truncated, and a licence where `get()` returned `None`. Swept `as_json_dict()` across all **1,864 LEXIS records: zero failures**, across all six shape variants. - `DatasetMetadata.__getitem__` now agrees with `.get()` for advertised-but-absent keys, so `dict(md)` works. Six keys raised before. - `RightsListMetadata.short_repr()` no longer assumes `rightsIdentifier`. This is why `rightsList` was missing from the flat projection: the `KeyError` was swallowed by `.get()`. - Response models declared for the listing endpoints, with coercing validators so typing cannot turn one odd sidecar into a 500. - `require_auth` renamed to `require_upstream_session`, alias kept, wire error code kept, docs made explicit about the absence of caller authentication. **Two mistakes caught during the work, both worth recording**: - Declaring the response model initially emitted `metadata: null` and `metadata_error: null` on every row — precisely the null-filling the report asked us to avoid. Fixed with `response_model_exclude_unset=True`, verified by diffing the default response. - Two workflow tests written yesterday pinned a fixture date and queried a relative `days=7` window. They passed on the day written and failed the next morning, looking like a regression in unrelated code. Fixtures are now relative to now. **Breaking Changes**: None. `include` is opt-in, the default listing is byte-identical, `require_auth` still resolves, and the 401 code is unchanged. **Not addressed** (reported, deferred by the reporter): `/health` cannot fail and there is no readiness probe; `/remote/ls` pagination re-walks the backend per page with no cache. ### main @ v5.10.8 - 2026-08-07 #### The last two plan items: --since, and reading verification back **Summary**: `verify --since` and `GET /remote/collections/{c}/verify`. All four gaps recorded in Part III are now closed, with one deliberate non-delivery documented. **Changes**: - `--since` skips datasets verified more recently than the cutoff, making a weekly pass over a large mirror practical. Tier order is respected — a recorded `checksum` satisfies a `readable` request, not the reverse — and a record whose `computed_at` is missing or unparseable is treated as stale, so incomplete information re-verifies rather than skips. - `GET /remote/collections/{c}/verify` exposes the last outcome per dataset with `ok` / `failing` / `unverified` counted apart. The distinction is the point: a dashboard that reads "no record" as "failing" reports unverified data as broken. - `GET /collections/{c}/index` deliberately not built — the index is an internal optimisation over the sidecars, and the catalog summary already serves consumers. Recorded rather than left implied. **Verified on the fixture**: `--since 2026-08-06` skipped all 3 as fresh (0 verified); `--since 2026-08-09` re-verified all 3; a malformed date is rejected with a validation error rather than silently ignored. **Breaking Changes**: None — new flag and new endpoint. **Plan status**: M0–M6 complete, all Part III gaps closed. ### main @ v5.10.7 - 2026-08-07 #### `ls --observed`: the claim and the measurement, side by side **Summary**: M1's acceptance criterion, the last operationally meaningful gap in the summarize/integrity plan. **Changes**: - `observed_totals()` in `core/tasks/remote.py` aggregates `stats.json` across a listing, returning `measured`, `unmeasured`, `observed`, `declared` and the ids of datasets holding no data — counts rather than one total, because summing across unmeasured datasets produces a number that looks like a measurement and is not. - `owi remote ls --observed` prints it beneath the normal table. **Two design choices worth recording**: - **Additional block, not a changed summary line.** Mutating the default alters numbers every consumer reads, and the value here is the *comparison*: declared 570 files against measured 3 says something neither figure says alone. - **Unmeasured is its own count.** `0 files measured` and `3 datasets never summarized` are different statements, and collapsing them is precisely the 586-vs-4 error this closes. **Verified on the fixture** both before and after `summarize`: unmeasured 3 / measured 0 first, then measured 3 files against declared 570, with the empty dataset named. **Breaking Changes**: None — opt-in flag, default output unchanged. **Remaining from the plan**: `verify --since` (F7's last cost control) and a per-collection verification endpoint (M5 shipped in a different shape). Neither is operationally urgent. ### main @ v5.10.6 - 2026-08-07 #### Write-back, and the phantom dataset it would have created **Summary**: F5 delivered, plus documentation housekeeping recording which acceptance criteria are still unmet. **Changes**: - `summarize --write-back/--no-write-back`, and `--summary --write-back` placing `_collection_summary.json` beside each collection. - **The guard that had to come first**: `list()` treats any `*.json` in a collection directory as a dataset sidecar, so the aggregate would have appeared in every listing as a phantom — the `latest.json` failure shape, self-inflicted. `COLLECTION_ARTIFACTS` excludes owilix's own collection-level artifacts from listings, the index, and the fingerprint. Verified on the fixture: 3 datasets before writing the aggregate, 3 after. - Part III of the plan now records shipped-vs-open per acceptance criterion, after "essentially complete" was claimed here once and was wrong. **Tested against the local file:// fixture only.** `--write-back` writes into the repository, and the LEXIS credential in this environment is currently valid, so running it against a real specifier would have written to production storage. **Breaking Changes**: None. `--write-back` defaults to true, which is the previous behaviour. **Still open**: `remote ls --observed` (the number a catalogue reads is still declared-only), `verify --since`, and a per-collection verification endpoint. ### main @ v5.10.5 - 2026-08-07 #### Derived artifacts over HTTP, and checksums that can actually be re-run **Summary**: M5 plus the incremental mode M4 unblocked. Two bugs found by running the code rather than reading it. **Changes**: - `GET /remote/catalog/summary` and `GET /remote/datasets/{id}/summary` expose what M0–M4 write beside the data. Unmeasured is kept distinct from measured-as-empty, and `verification: null` distinct from verified-and-clean — the same 0-vs-NULL rule, now at the API boundary. - Incremental checksums: reuse a recorded digest when size and mtime both match, labelled `cached` and counted. `--recheck-all` and `--only-missing` bound a run. - **Bug found while testing**: `checksums.json` was verifying itself. Self-referential by construction — hashing it changes it — so it could never converge and would rewrite on every run. Derived artifacts are now excluded, which also stops a regenerated README changing `dataset_digest`. - **Bug found while testing**: `fill_file_details` fetched only `size`, so the incremental reuse and the iRODS-checksum preference — both written, both tested against injected data — could never fire against a real repository. Switched to `fs.info()`, same round trip, and the tests that passed for the wrong reason now exercise the real path. **Also corrected**: the new HTTP tests initially passed a `NameError` off as "no stats", because `_read_stats_json` swallows exceptions and `json` was not imported in that test module. Worth noting as a pattern — a helper that catches broadly turns a test bug into a plausible-looking result. **Breaking Changes**: None. New endpoints; verification behaviour is additive and `--recheck-all` restores the previous full-pass semantics. **Still open from the plan**: nothing in M0–M6 except cost controls beyond `--max-files`/`--max-datasets`/`--only-missing` (F7's `--since`), and the CLI/HTTP surfaces for `reindex` scheduling. ### main @ v5.10.4 - 2026-08-07 #### A collection index that refuses to be trusted when it should not be **Summary**: M4. `_index.parquet` per collection plus `remote reindex`, shipped together because an index without a rebuild path is a liability rather than a cache. **Changes**: - New `owilix/core/index.py`: fingerprinting, build, read, and an `IndexUnusable` exception whose only correct handling is to walk — raising rather than returning `None` makes that impossible to forget at the call site. - `FileBasedRepository.list()` consults the index per collection and falls through to the existing walk on any failure. - `remote reindex` walks the sidecars once and writes the index; `owi remote ls` needs no flag. - Staleness is a fingerprint over each sidecar's name, size and mtime, recomputed from one `ls`. Chosen over a bare object count because it also catches a resized or re-uploaded sidecar at identical cost, since the listing already returns the details. **Verified end to end on the fixture**: walk and index return identical results; adding a dataset without reindexing yields 4 not 3; a corrupted index yields 4; a deleted index yields 4. All four paths agree with the source. **Breaking Changes**: None. No index means the previous behaviour exactly. **Second customer**: M6 tier 3 currently re-reads every byte each pass because it has no way to know what changed. The index is what an incremental checksum mode would build on — that is now the obvious next step for it. **Still open from the plan**: M5 (HTTP exposure of the derived artifacts), and incremental checksums on top of this index. ### main @ v5.10.3 - 2026-08-07 #### Verification: presence, readability, checksums — and what a hash cannot tell you **Summary**: M6 tiers 2–3. `owi remote verify` with three cost tiers, format-aware corruption detection, and results stored beside the data. **Changes**: - New `owilix/core/tasks/verify.py` and `owi remote verify`. Tiers `presence` / `readable` / `checksum`. - **Format validation belongs in the checksum tier**, per the user requirement: a truncated Parquet hashes perfectly, so hashing alone certifies corruption rather than detecting it. `checksum` and `format_valid` are independent facts; `format_valid: null` for uncheckable formats keeps "unknown" distinct from "invalid" — the same 0-vs-NULL rule as everywhere else in this plan. - **Hashes stored alongside the file** (`checksums.json`) so non-iRODS clients can verify independently; `md5` kept beside `sha256` for MD5-only tooling. - **iRODS checksums read where available**, labelled `checksum_source` so a reader can distinguish verified from asserted. - Multipart ETag `-N` detected and refused as MD5; a genuine single-part disagreement is recorded, never repaired. - `dataset_digest` over sorted `(path, size, sha256)`, with unhashed files contributing `?` so partial can never masquerade as complete. **Verified**: on the local fixture, a deliberately truncated parquet produced a valid SHA-256 **and** `format_valid: false` with the parser's reason ("Parquet magic bytes not found in footer"), while the healthy dataset reported rows, row groups and columns. `checksums.json` written for all three datasets. **Breaking Changes**: None — new command and a new derived artifact. **Still open from the plan**: M4 (`_index.parquet` + `remote reindex`) and M5 (HTTP exposure). M6 tier 3 has no incremental mode yet — it re-reads everything each pass, which is what M4's index would fix. ### main @ v5.10.2 - 2026-08-07 #### README gets a machine-readable head, without restoring the churn **Summary**: M3 of the summarize/integrity plan. YAML front matter on `README.md`, prose untouched. **Changes**: - `_readme_front_matter` emits the index-row fields as YAML; `split_front_matter` parses them back and is tolerant of a malformed or absent head. - **The interaction that mattered**: `computed_at` now appears in the README too, so the M2 unchanged-write check would never have matched and every README would have been rewritten on every run — reinstating the exact churn that could invalidate a dircache ETag and, before v5.9.0, empty a listing. `_stats_computed_at_only_diff` now strips volatile keys from front matter as well as from `stats.json`, and compares the prose literally. Caught by running the fixture, not by reading the diff. - `_yaml_safe` coerces unrepresentable metadata values rather than raising. **Verified**: front matter round-trips; a second `--force` pass writes nothing; the human section is byte-identical when only the timestamp moves. **Breaking Changes**: None. Additive head; existing readers of the prose are unaffected. **Still open from the plan**: M4 (`_index.parquet` + `remote reindex` — where the risk concentrates, and its own release), M5 (HTTP exposure), M6 (verify tiers, checksums, cost controls). ### main @ v5.10.1 - 2026-08-07 #### Observed vs declared, provenance, and an end to write churn **Summary**: M0–M2 of the summarize/integrity plan, in the order the plan set. All additive. **Changes**: - **M0 provenance**: `stats.json` records `computed_at` and `owilix_version`. It had none — the fields the plan credited it with are `remote doctor` output from a different command. Provenance was pulled ahead of the index because index rows that cannot say when they were computed reintroduce the ambiguity below at a larger scale. - **M0 honest empties**: `parquet_files_found` plus `materialised` separate *measured, empty* from *nothing measured*. `_generate_stats_json` returned the same literal for both. - **M1 observed vs declared**: `declared` from the sidecar, `observed` from the store. `observed` reports `0` rather than `null` when a listing succeeded and found nothing, because zero is a measurement; `declared` reports `null` for fields the sidecar lacks. - **M2 content-addressed writes**: `_write_if_changed` skips a write when bytes would not change. `stats.json` is compared ignoring `computed_at` alone — otherwise the timestamp would defeat the fix — and an unreadable existing object is written rather than assumed unchanged. **Verified** on the same three-dataset fixture used to schedule the plan: the dataset declaring 567 files and 250k objects over an empty directory now records `observed: {files: 0, bytes: 0}`, `materialised: false`; a second `--force` pass writes nothing where it previously rewrote every object. **Breaking Changes**: None. `statistics[]` keeps its shape and every new key is additive. **Still open from the plan**: M3 (README front matter), M4 (`_index.parquet` + `remote reindex`, where the risk concentrates), M5 (HTTP exposure), M6 (verify tiers, checksums, cost controls). ### main @ v5.10.0 - 2026-08-07 #### Workflow monitoring, and the reason it had never worked **Summary**: `owi workflow` (dags, runs, executions, summary) plus four HTTP endpoints, filterable by date and metadata, with JSON everywhere. The blocker turned out to be a one-line auth bug, not missing code. **Changes**: - **Root cause first**: `AirflowMonitor` sent `X-Api-Token`, which the LEXIS gateway answers with 401 for a valid token. Probed all three variants against the live instance — `Authorization: Bearer` returns 200, `X-Api-Token` returns 401 — so the 963-line module in `compat/` had never been able to work. That explains why it was unregistered and untested rather than merely neglected. - **New shared layer** `owilix/core/tasks/workflow.py`: window resolution, filtering, summarisation and the error taxonomy, called by both the CLI and the HTTP API so the filters cannot drift. The Airflow client itself is reused from `compat/workflows.py` rather than rewritten. - **CLI** `owi workflow` registered in the modern Typer app: `dags`, `runs`, `executions`, `summary`, each with `--json`. - **Filters**: `--dag`, `--task`, `--operator`, `--state`, `--pool`, `--queue`, `--hostname`, `--min-duration`, `--since`/`--until`, `--days`/`--hours`, `--limit`/`--offset`. OR within a field, AND across fields. Field names taken from real task instances (`dag_id`, `task_id`, `operator`, `state`, `duration`, `pool`, `queue`, `hostname`, …), not guessed. - **HTTP**: `GET /workflow/{dags,runs,executions,summary}` with the same filters as query parameters, synchronous because they answer in seconds. `400` bad date / `401` no credential / `502` Airflow unreachable; empty stays `200` with `total: 0`. - `summarize_executions` is reimplemented in the shared layer rather than reusing `AirflowMonitor.summarize`, because the latter re-fetches and cannot see a filter — a summary must describe exactly the set that was listed. - Durations report `None` rather than `0` when unmeasured, following the rule from the summarize/integrity plan. **Verified live**: against the LEXIS Airflow with a fresh token — 9 DAGs, 435 executions in 7 days, 6 failed. The failures are `heappe_keep` and `ref_token_keeper` pairs recurring on 2026-08-03, 08-05 and 08-07, i.e. a token-keeper failing roughly every other day. Both CLI and HTTP paths exercised, including date-range filtering and the 400 on a malformed date. **Breaking Changes**: None. New command group and new endpoints; nothing existing changed. ### main @ v5.9.1 - 2026-08-07 #### Verified the summarize/integrity plan and gave it an execution schedule **Summary**: Docs only. Part II appended to `docs/review/summarize_and_integrity_feature_plan_2026-08-07.md`: verification against the code, three corrections, milestones M0–M6 with acceptance criteria, and standing rules. Part I left as submitted. **Changes**: - Built a three-dataset collection against a `FileBasedRepository` (the class backing s3a remotes) including the 586-vs-4 shape — a sidecar declaring 567 files over an empty directory — and ran the real commands rather than reasoning about them. - **Correction 1**: `stats.json` carries no provenance. The plan credited it with `owilix.version`/`py4lexis.version`; those are `remote doctor` output. Provenance becomes a prerequisite (M0), not a preserved property. - **Correction 2**: the 0-vs-NULL rule the plan asks us to adopt is already violated — `_generate_stats_json` returns the same `{"values": {}}` for "measured, empty" and for "no parquet found". The rule now has a concrete first target. - **Correction 3**: host stats select a `url` column and silently skip when it is absent. - **Measurements kept**: `ls` reports 570 files / 250,200 objects where 3 / 200 exist; `summarize` fully describes a dataset holding nothing; `--force` rewrites byte-identical objects, confirming F4.1 is churn removal rather than an optimisation. - Sequencing: M0–M3 are small and safe (patch-sized); M4 (index + reindex) is flagged as where risk concentrates, because it puts a derived artifact in the read path — the one place this codebase has repeatedly turned a failure into a confident wrong answer — so it gets fallback-and-warn, a staleness check and its own release. **Breaking Changes**: None — documentation only. ### main @ v5.9.0 - 2026-08-07 #### One rewritten file no longer empties a mirror listing, successfully **Summary**: M1 from the backlog — the live production bug. `GET /remote/ls?specifier=owi-up` returned `{"total":0,"has_more":false}` with HTTP 200 against a bucket holding 591 datasets, while `remote doctor` called the same repository healthy. All five asks from `docs/review/owilix_improvement_brief_2026-08-07.md`, in the order given. **Changes**: - **Carry the outcome** (`owilix/core/listing.py`): `DatasetListing` is a `list` subclass with `complete` and `incomplete` entries of `{source, reason, message}`; `/remote/ls` and `/local/ls` expose `partial`, `failed` and `incomplete`. `reason` is a closed set because the reason is what a consumer branches on. This also delivers the listing half of M2 — field naming follows the bug report's `partial`/`failed` with the M2 proposal's `incomplete` array alongside, so both documents' consumers are served. - **Per-file tolerance**: `list()` collects read errors, returns the survivors and reports the skipped count. On its own this turns 591 → 0 into 591 → 590 plus a warning. - **Explicit `size` to `fs.open`**: s3fs sets `IfMatch` only when `size is None`, from a `dircache` that never expires, so a long-lived server 412s forever on any object rewritten after it was first listed. `list()` already has the size from the listing it just did. Falls back for filesystems that reject `size=`. - **`status()` reads one sidecar** and reports `readable`, failing a repository whose metadata cannot be read. An empty repository stays healthy — `readable` is `None`, not `False`, when there is nothing to read. - **Prune before loading**: an exact `collectionName=` filter restricts the directories walked. Regex and negation forms still walk everything, and a pruned walk that finds nothing re-walks — `collectionName` is metadata, the directory is convention, and a confidently empty answer is the bug being fixed. **Verification**: the reproduction was shrunk and run against both tags. At v5.8.0 it raises `OSError [Errno 22] Precondition Failed`, which the aggregate turns into an empty listing; at HEAD the same input returns the 5 survivors with `complete=False`. Suite 621 → 647. **Breaking Changes**: None. No existing field or signature changed; `DatasetListing` is a `list`, `partial` defaults to `false`. Minor rather than patch because the response gained fields and `doctor` can now report unhealthy where it previously said connected — deployments that alert on doctor output may see a repository flip that was already broken. **Left open**: M2's `remote pull` half (fail on partial discovery, `--allow-partial`) is a behaviour change for existing callers and was not folded into a fix release. M3 (`observed` vs `declared`) untouched. ### main @ v5.8.0 - 2026-08-06 #### Auth failures named, listing cost documented, and the small stuff **Summary**: Closes F3, F4, F6 and a new F8 from the mirror-operability review. With v5.7.0 the review is fully released except one item knowingly left open. **Changes**: - **F3**: New `owilix/core/exceptions.py` (`OwilixError`, `AuthenticationError`, `looks_like_authentication_failure`). `LexisRepository.list` re-raises auth-shaped failures instead of returning `[]`; `AggregatedRepository.list` re-raises when an auth failure left nothing to return; `remote_pull` reports `ErrorType.AUTH` / `ExitCode.AUTH_ERROR`. Two findings worth keeping: the failure was swallowed at *two* layers, so fixing either alone would have changed nothing; and detection cannot go by exception type, because py4lexis reports a missing credential as `Py4LexisAPIException` rather than its own `Py4LexisAuthException` — the DDI API answers with an `errorString` it does not recognise. Phrase matching is kept specific, with a test that "Failed to tokenize response body" does not read as a login problem. - **F4**: Documented on `/remote/ls` and in `remote ls --help` with both measurements. The cheap existence check turned out to already exist — `FileBasedRepository.status` does one `fs.exists` per access level and never enumerates, and F7 in v5.7.0 is what stopped it raising for those backends — so the docs point at `POST /remote/doctor/status` rather than adding an endpoint. A cheap *count* is declined with the measurement as the reason: the 84 s call returned zero datasets, so `load_metadata` was never reached and all 84 s was `listdir`/`ls`; a shallow count would cost the same. - **F6**: `ask_yes_no` detects a non-interactive stdin and declines rather than raising from `termios.tcgetattr`. Declining rather than accepting is deliberate — `default='y'` is what Enter means for a human, not consent from a caller that was never asked. The two bare `except:` in `core/tasks/remote.py` are now `except Exception` with a debug log, so Ctrl-C is no longer swallowed. Needed a module logger in that file; it had none. - **F8** (new, found while editing the docstring for F4): `owi remote ls --help` crashed with `MarkupError`, because the specifier syntax `[/]` in the docstring parses as a Rich closing tag and Typer renders docstrings as markup. Reproduced at v5.7.0, so it long predates this work. Nothing had ever invoked `--help` in a test; a sweep over all 57 commands now exists and fails at v5.7.0 on exactly `remote ls`. **Breaking Changes**: None to the HTTP API shape. `remote pull` gains exit code `3` for an unusable credential, where it previously reported the v5.7.0 data error (`4`) or, before that, success. A non-interactive caller that omitted `--yes` previously crashed and now receives a declined answer — a behaviour change only in the sense that the previous behaviour was a traceback. **Left open, deliberately**: a caller still cannot tell that a listing was *partial* when one repository failed and another answered. Fixing it means changing the `list()` contract and with it the `/remote/ls` response shape, so it is recorded under F3 rather than folded in. ### main @ v5.7.0 - 2026-08-06 #### Discovery failures are visible, and doctor works on non-LEXIS backends **Summary**: Released F2 and F7 from the mirror-operability review, both found by operating 5.6.0 against a live ~51 TiB mirror rather than by reading the code. Minor rather than patch: both change what an existing call reports. **Changes**: - **F2** (`1a11291`): `remote_pull` returns `success=False` / `ErrorType.DATA` / `ExitCode.DATA_ERROR` when discovery matched no dataset, naming the specifier. Deliberately narrow — datasets discovered whose files are already present stays a success and stays exit 0, because conflating the two would make an idle nightly mirror look broken. The HTTP job layer maps `success=False` to a `failed` job (`http_api/jobs.py:121`), so API consumers inherit the distinction without an API shape change. - **F7** (`dbf2cca`): `progress_cb=None` accepted and ignored on `AbstractRepository.status` and `FileBasedRepository.status`, forwarded by `AggregatedRepository.status`. `remote doctor` calls it unconditionally at `core/tasks/remote.py:504`; only `LexisRepository` had it, so every file-based backend answered with a `TypeError` instead of a status. - Reviewed both before release. Audited all four `status()` implementations and both call sites (`cli/admin.py:56` passes nothing and is fine on the default); confirmed `ddi.py` and `irods.py` define no repository `status()`; confirmed the F2 job mapping actually exists rather than taking the commit message's word for it. - Added the regression test F7 shipped without (`TestStatusProgressCbContract`). Written against dynamically discovered backends rather than the three that existed, with a guard against vacuous passes, because F7 *was* a contract satisfied by one implementation and unenforced everywhere else. Verified to fail at `dbf2cca~1` with the production error. - Fixed test pollution introduced by F2's own counter-case test: it requested `tmp_path` and never used it, so `_write_pull_sidecars` → `core/sync.py` resolved a bare `MagicMock` through `__fspath__` and mkdir'd `MagicMock/mock.local.create().path//` into the repo root on every run. Fixed rather than gitignored, and the absence is now asserted. **Breaking Changes**: None to the HTTP API shape. `remote pull`'s exit code and `success` flag change for the zero-discovered case — intended, and the reason for a minor bump. Automation that alerts on non-zero exit will now surface a permanently mis-scoped specifier instead of passing silently. ### main @ v5.6.0 - 2026-08-04 #### Remote pull `--push-to` source selection, and HTTP logout / doctor status **Summary**: Fixed a self-cancelling query that made `remote pull --push-to` report "0 datasets found" and exit 0 for nearly every specifier — the reason the OWI mirror could not be filled. Landed the two HTTP endpoints the accompanying operability review called for, and recorded the review itself. **Changes**: - `remote_pull` now passes `spec.get("data_center")` as the listing source instead of falling back to `push_to_remote`. The fallback combined with `ignore_data_centers=[push_to_remote]` into "list from X while excluding X", which is empty by construction. It hit every specifier whose data center is `None` — `all`, and every `it4i:`/`lrz:` zone alias, since the aliases reset the data center before setting a zone query — so only direct repository specifiers (`lexis:…`) ever worked. `remote_ls` was never affected because it already passed the data center alone, which is why `ls` and `pull` disagreed on identical input. `ignore_data_centers` is deliberately kept. - `remote_doctor` returns its `results` dict as `CommandResult.object` when `as_json` is set (previously always `None`), so the per-repository status, backend, DDI counts and per-zone breakdown are reachable without scraping rendered console output. - Added `POST /auth/logout` (`DeviceAuthManager.logout()` / `clear_device_states()`): deletes the refresh token, drops pending device logins, and returns `409 env_token_auth_active` when `PY4LEXIS_TOKEN` is set, since the server cannot revoke a credential handed to it by its environment. `_poll_until_complete` now returns rather than raising `KeyError` when its state was cleared underneath it — reachable once states can be cleared. - Added `POST /remote/doctor/status` for a synchronous status document; `POST /remote/doctor` stays the async job form. - Recorded `docs/review/remote_pull_and_mirror_operability_2026-08-04.md`: findings reproduced against the live clusters rather than inferred. F1 (the `--push-to` defect) and F5 (doctor unreadable over HTTP) ship here. **F2 remains open and is the most valuable of the rest** — zero *discovered* datasets still takes the success path and is indistinguishable from "already up to date", which is what hid F1 for weeks. F3 (auth failure surfacing as a `TypeError` during error formatting), F4 (an 84 s `remote ls` against object-store remotes) and F6 (two bare `except:`, `ask_yes_no` assuming a TTY) are also open. - Verified during release that the review's F6 note about `eval(files)` does not apply to any build carrying the HTTP API: the bare form existed through v5.2.0, `ast.literal_eval` landed in v5.3.0 (`fb24445`), and the HTTP API first shipped in v5.5.0. Recorded as an upgrade note in `CHANGELOG.md` rather than a fix. - Bumped package metadata to v5.6.0 (minor, not patch: two new endpoints) and refreshed the `Readme.md` version badge, stale at 5.5.0 since that release. **Breaking Changes**: None. Both endpoints are additive and no existing response shape changed. ### main @ v5.5.2 - 2026-07-15 #### Query slice CIFF index fix **Summary**: Fixed `query slice` writing a header-only (empty) `index.ciff.gz` for the sliced dataset, and restored compatibility with ciff-toolkit 0.2.x. **Changes**: - Filter each source dataset's CIFF index into its own temp file and merge the non-empty results per partition (previously every source was filtered onto the same target path, so the last source — often without matching documents — clobbered the index). - Warn on the console when a sliced index contains 0 documents or a partition has no matching source index. - Ported `owilix/core/ciff.py` to the ciff-toolkit 0.2.x `CiffWriter` API and bumped the dependency pin to `>=0.2.2`; `drop_parallel` returns `(kept_docs, postings_lists)`. - Fixed `UnboundLocalError` when `--search` is combined with CIFF slicing; normalized parquet `id` values (incl. binary columns) before matching. - Added regression tests (clobbering, merging, partitioned layouts, binary ids, zero matches, `keep=False`) and a network-free end-to-end CLI test that asserts sliced index document counts. - Bumped package metadata for the `v5.5.2` patch release. **Breaking Changes**: None (requires `ciff-toolkit>=0.2.2` at install time). ### main @ v5.5.1 - 2026-06-02 #### Remote pull push-to payload routing fix **Summary**: Fixed LEXIS-backed `remote pull --push-to ` so dataset payload files are streamed into the configured destination filesystem instead of being written under the process working directory. **Changes**: - Made `LexisRepository.get(..., filesystem=...)` copy from the LEXIS filesystem into the provided destination fsspec filesystem. - Preserved existing local-download behavior when no destination filesystem is supplied. - Added regression coverage using in-memory fsspec filesystems. - Bumped package metadata for the `v5.5.1` patch release. **Breaking Changes**: None. ### feature/http-server @ v5.5.0 - 2026-05-30 #### Optional HTTP API preview **Summary**: Added an optional FastAPI-based HTTP API for local OWILIX automation and controlled demos, released as a preview local service with explicit deployment limits. **Changes**: - Added `owilix http serve` through the optional `http` extra. - Added health, device-login auth, jobs, local, remote, query, config, and plugin endpoints. - Added in-process background jobs for long-running operations and pollable `/jobs/{job_id}` status. - Added optional local S3-compatible facade mounted at `/s3`. - Added local dataset quota status, quota configuration, and dry-run-by-default cleanup endpoints. - Added Docker HTTP compose support with localhost-only published port by default. - Documented the HTTP API preview, local-only release posture, S3 facade, quota controls, and known limits. **Breaking Changes**: None. ### feature/remote-sync-sidecar @ 1b1ebafd9ad408abbc4d8784e62f27f74272a5b8 - 2026-04-30 #### Remote sync sidecar workflow **Summary**: Added a sidecar-based remote sync workflow for pull/status/push operations so users can pull a dataset, edit local files, inspect dirty state, and push changed files without requiring remote-side checksums. **Changes**: - Added `.owi-sync.json` and `.owi-files.json.gz` sidecars capturing local and remote file baselines from `remote pull` - Added `owi remote status` with local dirty-state detection, optional remote refresh, details output, generated-file inclusion, and explicit `not-found` rows - Made `owi remote push` upload sidecar-detected local additions and modifications while excluding sync sidecars and editor swap files - Refreshed sidecar baselines after successful sidecar-aware push - Fixed dataset ID handling in `local free` and `remote push` - Fixed metadata update regressions during remote push - Documented the implemented workflow under `docs/features/implemented/remote-sync-sidecar-inventory-workflow.md` **Breaking Changes**: None. ### main - 2026-02-25 #### Remote upload S3-source support and release preparation **Summary**: Added direct S3 URI source support to `owi remote upload`, aligned repository upload plumbing for external filesystems, and prepared release metadata for `v5.2.0`. **Changes**: - Extended `remote_upload` source handling to support local paths plus `s3://`/`s3a://` URIs - Passed external source filesystems through aggregate/file/lexis repository `put()` methods - Added unit coverage for unsupported source schemes and S3-backed upload flows - Updated remote command documentation and examples - Updated installer dependency handling for `python-http-irods-client` to satisfy `uv` direct URL requirement during install - Pinned installer-side `py4lexis==5.0.1` to avoid transitive URL dependency drift conflicts in `uv` - Bumped package version and finalized release changelog entries for `v5.2.0` ### multifs - 2026-02-17 #### Multi-filesystem and CLI consistency updates **Summary**: Improved multi-filesystem behavior across CLI and manager flows, with corresponding tests and docs updates. **Changes**: - Extended HTTP-to-iRODS fsspec integration in `owilix/core/fsspec/http2irods.py` - Updated CLI and context/output handling in `owilix/cli/` modules - Updated manager/query flow integration in `owilix/core/manager/manager.py` and `owilix/core/tasks/query.py` - Expanded unit coverage in `tests/owilix/core/fsspec/test_core_fsspec_unit.py` - Updated dependency lock and configuration docs (`uv.lock`, `pyproject.toml`, `docs/source/config.md`) ### py4lexis4 - 2026-01-02 #### Py4Lexis 5.x Migration Complete **Summary**: Full migration to py4lexis 5.x HTTP APIs, replacing python-irodsclient with HTTP-based iRODS access. **Changes**: - Added `Http2IrodsFileSystem` fsspec backend for HTTP-based iRODS access - Created `owilix/core/repository/` modular package (replaces monolithic repository.py) - Added `OWILIXSession` wrapper for unified session management - Implemented configuration V3 with migration support (`owi config migrate`) - Added new display formats: `--format=short|wide|markdown` for `owi remote ls` / `owi local ls` - Added config commands: `config version`, `config migrate`, `config delete`, `config reset` - Improved test infrastructure with progress indicators **Breaking Changes**: - Deprecated repositories `it4i`, `lrz` removed - replaced by unified `lexis` - Config auto-migrated from V2 to V3 via `owi config migrate` ### main - 2026-01-01 #### Documentation Alignment - Added `docs/source/contributing/` section with symlinks to dev docs - Updated `index.rst` toctree to include contributing section - Added AI agent onboarding section to `docs/epics.md` - Fixed external links in dev docs for Sphinx compatibility - Created branch documentation at `docs/branch/feature-docs-alignment.md` --- ## History ### main @ initial - 2026-01-01 #### Summary Initial AI infrastructure setup for owilix. #### Changes - Added `.agent/workflows/` with development, testing, documentation, and release workflows - Created `docs/architecture.md` with project overview - Created `docs/CODING_GUIDELINES.md` with coding standards - Created `docs/epics.md` for epic tracking - Created `docs/branch/` for per-branch documentation - Created `docs/external/py4lexis.md` with best practices - Fixed pyproject.toml merge conflict from poetry→uv migration