# HTTP API Preview OWILIX includes an optional FastAPI server for local automation and controlled service demos. It is a preview API in `v5.5.0`: the endpoint surface is useful, but the server should be treated as a local service unless an external access-control layer is placed in front of it. **There is no caller authentication.** The dependency guarding 25 routes — `require_upstream_session`, formerly named `require_auth` — checks only whether *the server* holds a usable LEXIS credential. It never inspects the request. There are no `securitySchemes` in the OpenAPI document and no operation declares `security`. Two consequences worth stating plainly: - A `401` means **the server is logged out**, not *you are not allowed*. - **Anything that can reach the port can drive every route**, including `POST /remote/remove`, `POST /remote/push` and `POST /local/rm`, under the server's single shared identity. Do not expose `owilix http serve` on a public or shared network as-is. Run it on localhost, or behind a reverse proxy, tunnel or gateway that authenticates callers. ## Install The normal CLI install remains lightweight. Install the optional dependencies when you need the HTTP server: ```bash uv pip install "owilix[http]" ``` Docker builds install the `http` extra by default. ## Run Locally ```bash owilix http serve ``` The command binds to `127.0.0.1:8080` by default. Keep that default for local use. If you bind to `0.0.0.0`, protect the service with a reverse proxy, firewall, tunnel, or another access-control mechanism. Useful checks: ```bash curl http://127.0.0.1:8080/health curl http://127.0.0.1:8080/auth/status ``` ## Authentication Flow The server can start an OAuth device login and store the resulting refresh token in the same OWILIX token location used by the CLI: ```bash curl -X POST http://127.0.0.1:8080/auth/device/start curl http://127.0.0.1:8080/auth/device/ ``` Remote and query endpoints require the server process to have a refresh token or `PY4LEXIS_TOKEN`. This is a process-credential check, not caller-level API authorization. `POST /auth/logout` reverses a device login: it deletes the stored refresh token and discards any pending device-login attempts, so `GET /auth/status` reports `authenticated: false` afterwards. ```bash curl -X POST http://127.0.0.1:8080/auth/logout ``` When the server was started with `PY4LEXIS_TOKEN` in its environment, logout cannot revoke that credential — the endpoint returns `409` with `{"error": "env_token_auth_active"}` and leaves the token in place. Restart the process without the variable instead. ## Endpoint Surface Core endpoints: - `GET /health` - `GET /auth/status` - `POST /auth/logout` - `POST /auth/device/start` - `GET /auth/device/{state_id}` - `GET /jobs` - `GET /jobs/{job_id}` Local endpoints: - `GET /local/ls` - `POST /local/free` and `POST /local/rm` - `POST /local/insert` - `POST /local/export` - `POST /local/analyze` - `GET /local/quota` - `POST /local/quota` - `POST /local/cleanup` Remote endpoints: - `GET /remote/ls` (**can take minutes on object-store remotes** — see [Listing cost](#listing-cost)) - `POST /remote/search` - `POST /remote/pull` - `POST /remote/push` - `POST /remote/upload` - `POST /remote/remove` - `POST /remote/doctor` (asynchronous: returns a job record) - `POST /remote/doctor/status` (synchronous: returns the status document inline; reports `readable` per repository, and fails one whose metadata cannot be read) - `POST /remote/catalog` - `POST /remote/status` - `POST /remote/diff` Query endpoints: - `POST /query/less` - `POST /query/sites` - `POST /query/analyze` - `POST /query/aggregate` - `POST /query/stats` - `POST /query/slice` - `POST /query/stream` `GET /remote/ls` and `GET /local/ls` accept `?include=metadata` — see [Full descriptive metadata](#full-descriptive-metadata). Derived artifact endpoints: - `GET /remote/catalog/summary` — declared vs observed across a scope - `GET /remote/datasets/{id}/summary` — one dataset, with its verification record - `GET /remote/collections/{c}/verify` — last verification outcome per dataset Workflow monitoring endpoints (Airflow): - `GET /workflow/dags` - `GET /workflow/runs` - `GET /workflow/executions` - `GET /workflow/summary` These are synchronous — they are read-only queries that answer in seconds, and a monitoring caller wants the answer rather than a job handle. See [Workflow monitoring](#workflow-monitoring). Configuration and plugin endpoints: - `GET /config` - `POST /config/get` - `POST /config/set` - `GET /plugin/list` - `POST /plugin/run` Long-running operations return a job record. Poll `GET /jobs/{job_id}` until the job reaches `succeeded` or `failed`. Job state is in-process and is lost when the server restarts. ## Local Dataset Quota The HTTP server can report and enforce a local dataset cache budget based on the filesystem that backs the local repository root. Configure the quota: ```bash curl -X POST http://127.0.0.1:8080/local/quota \ -H 'content-type: application/json' \ -d '{"max_usage_percent": 40}' ``` Check status: ```bash curl http://127.0.0.1:8080/local/quota ``` Dry-run cleanup: ```bash curl -X POST http://127.0.0.1:8080/local/cleanup \ -H 'content-type: application/json' \ -d '{"dry_run": true}' ``` Actual cleanup removes oldest local dataset copies until the quota is met: ```bash curl -X POST http://127.0.0.1:8080/local/cleanup \ -H 'content-type: application/json' \ -d '{"dry_run": false}' ``` You can also set `OWILIX_HTTP_LOCAL_MAX_USAGE_PERCENT`. ## Optional S3 Facade Set `OWILIX_HTTP_S3_ENABLED=true` to mount the local repository through a local S3-compatible facade at `/s3`. Credentials are required unless local test mode is enabled: ```bash export OWILIX_HTTP_S3_ENABLED=true export OWILIX_HTTP_S3_ACCESS_KEY=local-access-key export OWILIX_HTTP_S3_SECRET_KEY=local-secret-key owilix http serve ``` For local-only testing, `OWILIX_HTTP_S3_NO_AUTH=true` disables S3 facade credential checks. ## Full descriptive metadata `GET /remote/ls` returns eleven flat keys per dataset. Those come from `metadata.get()`, which returns a **display rendering, not the stored value**: | field | flat key gives | actual value | | --- | --- | --- | | `creators` | `"Lukas Gienapp+7"` | 8 author objects, with ORCIDs and affiliations | | `descriptions` | ellipsis-truncated | full text | | `publisher` | `{"name": "CORAL Project"}` | same | | `rightsList` | *(absent)* | licence with `rights` and `rightsUri` | The `+7` means "and seven more" — a page cannot recover eight authors from it. For anything that needs the record rather than a table row, ask for it: ```bash curl "http://127.0.0.1:8080/remote/ls?specifier=lexis/collectionName=corpora&include=metadata" ``` **Shape.** `metadata` carries the record as stored, with DataCite fields nested under a `datacite` key: ```jsonc { "id": "…", "title": "German Commons", "collectionName": "corpora", "metadata": { "datacite": { "creators": [{"name": "Lukas Gienapp", "givenName": "Lukas", "nameIdentifiers": [{"nameIdentifier": "0000-0001-…", "nameIdentifierScheme": "ORCID"}], "affiliation": [{"name": "University of Kassel"}]}, …], "publisher": {"name": "CORAL Project"}, "publicationYear": 2025, "rightsList": [{"rights": "Open Data Commons Attribution License v1.0", "rightsUri": "https://opendatacommons.org/licenses/by/1-0/"}], "relatedIdentifiers": [{"relatedIdentifier": "https://huggingface.co/…", "relatedIdentifierType": "URL"}] }, "schema": "http://datacite.org/schema/kernel-4" } } ``` **Why opt-in.** An unfiltered listing is ~1,900 records and most callers want the flat keys. `include` takes a comma-separated list; unknown values are ignored rather than rejected. **Absent fields are omitted, never null-filled.** Mirrors carry several metadata shapes — one deployment has six distinct key sets between 33 and 43 keys, with `relatedIdentifiers` on only 65% of datasets. Nothing keys on a fixed key set or count, `schema` is read as the version marker where present and tolerated where absent, and **a record whose metadata cannot be rendered reports `metadata_error` while the rest of the page still answers** — one malformed sidecar does not fail the listing. No re-pull or migration is required: old sidecars work as they are. ## Derived Artifacts What `summarize`, `verify` and `reindex` write beside the data, readable without scraping console output or holding object-store credentials. ```bash curl "http://127.0.0.1:8080/remote/catalog/summary?specifier=owi-up" curl "http://127.0.0.1:8080/remote/datasets//summary?specifier=owi-up" ``` `GET /remote/catalog/summary` reports **declared against observed** for every dataset in a scope, plus two counts that must not be confused: | field | meaning | | --- | --- | | `materialised_count` | datasets measured and found to hold data | | `unmeasured_count` | datasets **never summarized** — `materialised` is `null` | **`unmeasured_count` is not `0` materialised.** A dataset that has never been summarized has no observation at all, and reporting it as empty is the mistake that told visitors 586 collections were mirrored when 4 held data. `observed` is `null` for those, never `{files: 0}`. `GET /remote/datasets/{id}/summary` adds the verification record when one exists: ```jsonc { "id": "0027fdbc-…", "declared": { "files": 567, "objects": 250000, "bytes": 14748809896 }, "observed": { "files": 0, "bytes": 0 }, "materialised": false, "stats_computed_at": "2026-08-07T18:26:48+00:00", "verification": { "tier": "checksum", "dataset_digest": "01001c23…", "checked": 12, "invalid": 1, "unreadable": 0, "ok": false, "computed_at": "2026-08-07T19:02:11+00:00" }, "partial": false, "failed": [] } ``` **`verification: null` means never verified**, which is not the same as verified and clean. Branch on its presence before branching on `ok`. **Status codes carry the same care as everything else here:** a dataset that is absent from a *complete* listing is `404`; a dataset absent from a **partial** listing is `503` with `partial: true`, because we cannot say it is missing when we could not see everything. Both list endpoints carry `partial` / `failed` as described under [Partial Listings](#partial-listings). ## Workflow Monitoring Four read-only endpoints over the LEXIS Airflow instance, sharing their filter logic with `owi workflow` so the two cannot drift. An **execution** is one task instance: one task, in one DAG run, with a state, a duration and the operator that ran it. That is the unit worth counting. ```bash curl "http://127.0.0.1:8080/workflow/summary?days=7" curl "http://127.0.0.1:8080/workflow/executions?state=failed&days=30&limit=20" curl "http://127.0.0.1:8080/workflow/executions?since=2026-08-01&until=2026-08-05" curl "http://127.0.0.1:8080/workflow/runs?dag_id=my_dag&state=failed" ``` **Filters** (all optional, all accepting comma-separated values): | parameter | meaning | | --- | --- | | `dag_id`, `task_id`, `operator` | identity | | `state` | `success`, `failed`, `skipped`, `queued`, `up_for_reschedule` | | `pool`, `queue`, `hostname` | where it ran | | `min_duration` | seconds, floor | | `since`, `until` | `YYYY-MM-DD` or ISO-8601; naive values are read as UTC | | `days`, `hours` | relative window; **defaults to the last 7 days** | | `offset`, `limit` | paging (`executions` only) | Values are OR within a field and AND across fields, so `?state=failed,up_for_reschedule&dag_id=X` means "either of those states, in DAG X". The response echoes back `window` and `filters`, so a caller does not have to remember what it asked for. `GET /workflow/summary` groups the *same filtered set* by state, operator and DAG, with `avg_duration_sec` and `p95_duration_sec`. Those are **`null` rather than `0`** when nothing reported a duration — a task that never recorded one is not a task that took no time. `GET /workflow/runs` carries the partial contract: if one DAG cannot be read the others are still returned, with `partial: true` and `failed: [...]`. **Status codes:** `400` for an unparseable date, `401` when there is no LEXIS credential, `502` when Airflow is unreachable. An empty result is `200` with `total: 0` — distinguishable from all three. ## Partial Listings `GET /remote/ls` and `GET /local/ls` say whether the answer is the whole answer: ```jsonc { "count": 584, "total": 584, "offset": 0, "limit": null, "has_more": false, "datasets": [ /* ... */ ], "partial": false, // always present; false when nothing was skipped "failed": [] // sources that contributed less than they hold } ``` When something *was* skipped, `partial` is `true`, `failed` names the sources, and an `incomplete` array carries the reason: ```jsonc { "total": 0, "has_more": false, "datasets": [], "partial": true, "failed": ["owi-up"], "incomplete": [ { "source": "owi-up", "reason": "error", "message": "[Errno 22] Precondition Failed" } ] } ``` `reason` is a small closed set — `auth`, `unreachable`, `timeout`, `error`, `partial_read` — because the reason is what a consumer should branch on: `auth` is operator-actionable and probably persistent, `timeout` is probably transient and worth retrying. Matching on `message` text is fragile and is how this class of failure goes unnoticed. **Why this exists.** A listing that fails every read used to return `total: 0` with HTTP 200, which is indistinguishable from a store that genuinely holds nothing — and consumers that retire anything absent from the latest listing would quietly retire the lot. `partial` is the bit that tells them not to. **Backward compatible.** `partial` is `false` and `failed` is `[]` for a complete answer, `incomplete` is present only when there is something to report, and no existing field changed. A caller that ignores all three behaves exactly as before. ## Listing Cost `GET /remote/ls` enumerates. For a LEXIS repository the catalogue is queried and the answer comes back quickly; for an object-store remote (`s3a`/`s3`) the store itself is walked, so **the cost tracks the size of the store rather than the size of the answer** — and it is slowest when it has least to say. Measured against a full OWI cluster: | specifier | time | result | | --- | --- | --- | | `owi-up/collectionName=main` (s3a) | **84.1 s** | 0 datasets | | `lexis/collectionName=main` | 7.1 s | 584 datasets | Consequences for a synchronous caller: - **Budget minutes, not seconds.** A 30 s client timeout against an object-store remote will fail every time, and it fails in the way least likely to be diagnosed correctly — as a connection error rather than as a slow answer. - **Cache the result** if you are rendering it repeatedly (a dashboard panel, a health view). The answer changes far more slowly than the cost of asking. - **Do not use `ls` as a liveness probe.** Use `POST /remote/doctor/status`, which reports per-access-level existence with a handful of prefix checks and does not enumerate. It answers "is this remote reachable, and does it hold anything at each access level" — which is what a health light needs. There is deliberately **no cheap dataset *count*** endpoint. A count over an object store requires listing the keys under the prefix; there is no cheaper primitive to expose. The measurement above is the evidence: that 84 s call returned *zero* datasets, so no per-dataset metadata was ever fetched and the time went entirely into walking the store. A "count without enumerating" would have to be served from a maintained index or manifest rather than from the store, which is a different feature and not something `ls` can be made to do. ## Known Limits - HTTP endpoint authorization is preview-level; use only on localhost or behind external access control. - Jobs are in-process only and disappear after restart. - Job cancellation and durable progress records are not implemented. - `remote/ls` pagination currently slices after fetching backend results — so a page request costs the same as the full listing (see [Listing cost](#listing-cost)). - The S3 facade has been smoke-tested; broader compatibility with `aws s3`, `boto3`, and `rclone` remains follow-up work.