# Owilix Epics

Epic and backlog tracking for owilix development.

> **For AI Agents**: This is your starting point. Check the **In Progress** section for current work, or the **Backlog** for planned work. Key resources:
> - `docs/AI_DEVELOPMENT.md` - AI development guide
> - `docs/testing.md` - Testing guide for AI agents
> - `.agent/workflows/` - Workflows (invoke with `/development`, `/testing`, etc.)
> - `docs/architecture.md` - Project structure
> - `docs/CODING_GUIDELINES.md` - Coding standards
> - `docs/branch/` - Per-branch documentation

## Status Legend

| Status | Meaning |
|--------|---------|
| 🟢 Done | Epic completed and merged |
| 🟡 In Progress | Epic has active branch |
| ⚪ Planned | Epic in backlog |

---

## Backlog

Planned epics not yet started:

- **Workflow Execution & Statistics**: Run workflows and report execution statistics through owilix — see the epic below and `docs/backlog.md` F3
- **Mirror Operability (M1–M3)**: Open items from the 2026-08 review campaign — a read failure must not become an empty success (**live bug**), partial listings must be able to say so, and datasets must be verifiable rather than merely listed. See `docs/backlog.md`.
- **Query Federation**: Add support for federated queries across datacenters
- **Tag-based Specifiers**: Allow tag-based query federation
- **Async Query Optimization**: True async I/O implementation for query commands to improve performance (currently uses thread pools)
- **Dataset Cleaning**: Tools for cleaning/validating datasets
- **Overall Clean-up**: Code restructuring and cleanup
- **Security Hardening**: Defer and track hardening tasks captured in `docs/hardening.md`

### ⚪ Workflow Execution & Statistics

- **Branch**: feature/workflows (not started)
- **Description**: Drive workflow execution and report workflow statistics through owilix.
- **Detail**: [`docs/backlog.md`](backlog.md) § Features → F3
- **Start here — this is not greenfield:**
  - Execution partly exists: `owi batch run <specifier> <taskfile.yaml>` (`owilix/cli/batch.py` → `compat.batch.BatchCommands`), with workers, keep-going, resume and dry-run. Not exposed over HTTP.
  - Statistics exist but are stranded: `owilix/compat/workflows.py` is a 963-line Airflow 2.8 monitor (`dags`, `dag_runs`, `summary`) that is unregistered in the modern CLI, absent from the HTTP API, and untested. `owi workflow` is not a command.
- **Cycles** (provisional — the first one decides the rest):
  - Cycle 1: Decide the scope of "workflow" (Airflow DAGs, owilix `batch run` task files, or both), and whether to port or delete `compat/workflows.py`
  - Cycle 2: Expose execution — `batch run` over HTTP, which needs durable jobs (F2's first item; jobs are in-process and lost on restart)
  - Cycle 3: Statistics surface, reusing what `JobManager` already records (`operation`, `status`, the three timestamps, `logs`)
- **Blocked-by, and worth taking seriously**: M1–M3 in `docs/backlog.md`. Statistics computed over listings that can be silently empty, silently partial, or count a sidecar as a materialised dataset will be confidently wrong in exactly the way the 2026-08 campaign spent three releases fixing. M1 is a live bug and should land first.

### ⚪ Deployment & Release Prep

- **Description**: Prepare version 4.0.0 release and deployment infrastructure
- **Tasks**:
  - [ ] Update installation methods (uv, mamba, micromamba, conda, docker)
  - [ ] Prepare deployment: Bump version to 4.0.0 (align with py4lexis 4.0+)
  - [ ] Document all proper changes and update README
  - [ ] Add migration hints (migrate local datasets or delete them)
  - [ ] Update GitLab runner if necessary and deploy version

---

## In Progress

(No active epics)

---

## Completed

### � DuckDB Query Integration

- **Branch**: py4lexis4-duckdb (merged to py4lexis4)
- **Started**: 2026-01-02
- **Completed**: 2026-01-05
- **Description**: Integrate DuckDB querying with new fsspec backend for direct iRODS queries
- **Details**: [docs/branch/py4lexis4-duckdb.md](branch/py4lexis4-duckdb.md)
- **Deliverables**:
  - `owilix/core/db/` package with `OWIDuckDBSelectExecutor`, `OWIDuckDBAsyncExecutor`
  - Query commands with `Http2IrodsFileSystem` integration
  - `--async` and `--prefetch` options for all query commands
  - Automatic session token refresh via callable

### 🟢 Py4Lexis 5.x Migration

- **Branch**: py4lexis4
- **Started**: 2026-01-01
- **Completed**: 2026-01-02
- **Description**: Migrate to py4lexis 5.x with HTTP iRODS client
- **Details**: [docs/branch/py4lexis4.md](branch/py4lexis4.md)
- **Cycles**:
  - Cycle 1: iRODS HTTP fsspec implementation (`Http2IrodsFileSystem`)
  - Cycle 2: Session management (`OWILIXSession` wrapper)
  - Cycle 3: Repository package refactor (`owilix/core/repository/`)
  - Cycle 4: Configuration management (V3 migration, config commands)
  - Cycle 5: UI improvements (short/wide/markdown display formats)
- **Deliverables**:
  - `owilix/core/fsspec/http2irods.py` - HTTP-based iRODS filesystem
  - `owilix/core/repository/` - Modular repository package
  - `owilix/core/manager/session.py` - Unified session wrapper
  - Config V3 migration with `owi config migrate`
  - New display formats: `owi remote ls --format=short|wide|markdown`
  - `docs/source/config.md` - Configuration documentation

### 🟢 Documentation Alignment

- **Branch**: main (direct)
- **Completed**: 2026-01-01
- **Description**: Aligned documentation structure - added contributing section to Sphinx with symlinks to dev docs
- **Deliverables**:
  - `docs/source/contributing/` section
  - AI agent onboarding in `docs/epics.md`
  - Branch documentation template

---

### 🟢 AI Infrastructure Setup

- **Branch**: main (direct)
- **Completed**: 2026-01-01
- **Description**: Set up AI coding infrastructure for development with Antigravity
- **Deliverables**:
  - `.agent/workflows/` directory
  - Documentation structure
  - Coding guidelines

---

### 🟢 Web-graph Support

- **Branch**: (integrated)
- **Completed**: Prior to v0.23.0
- **Description**: Added graph analysis and reporting features

---

### 🟢 Docker Install

- **Branch**: (integrated)
- **Completed**: Prior to v0.8.0
- **Description**: Docker containerization support

---

## Epic Template

When creating a new epic, use this template:

```markdown
### ⚪ [Epic Name]

- **Branch**: feature/<epic-name>
- **Started**: YYYY-MM-DD
- **Description**: Brief description
- **Cycles**:
  - Cycle 1: Goal
  - Cycle 2: Goal
```