# OWIlix Commands and Usage

The OWIlix CLI provides a suite of commands for managing datasets, interacting with remote repositories, and performing data analysis.

## Command Groups

| Group | Description | Documentation |
|-------|-------------|---------------|
| **[local](details/local.md)** | Manage datasets in your local repository. | [Read More](details/local.md) |
| **[remote](details/remote.md)** | Interact with remote data centers (list, search, pull, upload, diagnose). | [Read More](details/remote.md) |
| **[query](details/query.md)** | SQL-based analysis and filtering (slice, stats, less). | [Read More](details/query.md) |
| **[config](details/config.md)** | View and modify CLI configuration. | [Read More](details/config.md) |
| **[admin](details/admin.md)** | Administrative tasks, repository/path diagnostics, and filesystem inspection. | [Read More](details/admin.md) |
| **[http](http.md)** | Optional local HTTP API preview. | [Read More](http.md) |

## Detailed Documentation

- **[Local Commands](details/local.md)**: `ls`, `rm`, `analyze`, `init`
- **[Remote Commands](details/remote.md)**: `ls`, `search`, `pull`, `push`, `upload`, `diff`, `summarize`, `summarize-hosts`, `verify`, `doctor`, `logout`
    - **[Verifying mirrored data](details/verify.md)**: presence, readability and checksums — and why a hash alone does not prove the data is usable.
- **[Admin Commands](details/admin.md)**: `repos`, `path`, `fs`, `logs`, `stats`, `check`
- **[HTTP API Preview](http.md)**: `serve`
- **[Query Commands](details/query.md)**:
    - **[Slice](details/query_slice.md)**: Create new datasets from queries.
    - **[WARC](details/warc.md)**: WARC file extraction.
    - **[Graphs](details/graphs.md)**: Web graph analysis.
    - `less` (interactive data browser), `stats`, `sites`, `aggregate`
    - Supports `--search` remote-index prefiltering and `--async` mode for concurrent query execution
- **[Configuration](details/config.md)**: `list`, `get`, `set`

## Global Options

The following flags can be used with almost any command:

| Flag | Short | Description |
|------|-------|-------------|
| `--verbose` | `-v` | Enable verbose output (debug logs). |
| `--loglevel` | `-l` | Set log level: DEBUG, INFO, WARNING, ERROR. |
| `--yes` | `-y` | Auto-confirm interactive prompts. |
| `--config` | `-c` | Specify a custom configuration file path. |
| `--format` | `-f` | Output format: `table`, `json`, `jsonl`. |
| `--output` | `-o` | Write output to a file. |
| `--no-display` | | Suppress console output (useful for scripting). |
| `--target` | `-t` | Override the target directory for data. |
| `--no-progress` | `-N` | Suppress progress bars. |
| `--sql-log` |  | Write executed SQL statements to `~/.owi/.logs/sql.log` for debugging. |

## Defaults & Specifiers

### Paths
- Default Path: `~/.owi`
- Can be modified via environment variable `OWS_OWI_PATH` or `owi config`.

### Specifier Format
Access datasets using the following specifier format:
`{scope|all}[:{YYYY-MM-DD|latest|YYYY-MM-DD#days|YYYY-MM-DD..YYYY-MM-DD}][/{key=value;key=value}]`

Rules:
- `all` searches all configured sources.
- `it4i` and `lrz` remain valid legacy scope aliases and now translate to zone filters (`IT4ILexisV2` and `OWILRZZONE`).
- `;` combines metadata filters with AND.
- `value1,value2` means OR within one field.
- `key*=` performs a regex match.
- `key-=` excludes exact values.

Examples:
- `all:latest`: Latest datasets from all configured sources.
- `lexis:2025-01-01`: Datasets from LEXIS on a specific date.
- `all:2025-01-01..2025-01-10`: Explicit inclusive date range.
- `it4i:latest`: Latest datasets in zone `IT4ILexisV2`.
- `all/collectionName=main`: Filter by collection name.
- `all/zone=IT4ILexisV2,OWILRZZONE`: Match either zone.
- `all/title*=.*Open.*;zone-=OWILRZZONE`: Regex title match while excluding LRZ.

## Admin Commands

Admin commands are used for troubleshooting and system monitoring.

- `owi admin logs [selector]`: View logs by selector (`latest`, `events`, `sql`, filename) or print paths with `location`.
- `owi admin stats`: View system statistics.
- `owi admin repos`: List configured repositories and their status.
- `owi admin path <path> [--repository <name>]`: Check whether a path exists in one/all repositories.
- `owi admin fs <operation> <path> [options]`: Run low-level filesystem commands (`ls`, `exists`, `info`, `find`, `glob`, `cat`, `mkdir`, `cp`, `mv`, `rm`).

## Publishing to OpenSearch

OWILIX can stream query results into OpenSearch using the built-in plugin and helper script.

### Environment Variables

Set these in `.env-rc` (the script sources it) or your shell:

```bash
export OWILIX_OPENSEARCH_HOST="https://opensearch.example.org:9200"
export OWILIX_OPENSEARCH_USERNAME="your-user"
export OWILIX_OPENSEARCH_PASSWORD="your-password"
export OWILIX_OPENSEARCH_VERIFY_CERTS="true"  # optional
export OWILIX_OPENSEARCH_CA_CERTS="/path/to/ca.pem"  # optional
```

### Using `scripts/ours_index.sh`

This script runs a `query less` against a dataset specifier and pipes JSONL into the
OpenSearch indexer using `OURSConverter`.

```bash
./scripts/ours_index.sh "all/id=<DATASET_ID>" "ows-title-index"
```

Arguments:
- `$1`: dataset specifier (e.g. `all/id=<DATASET_ID>`)
- `$2`: OpenSearch index name

### Direct CLI (no script)

```bash
owilix --format jsonl query less --remote "all/id=<DATASET_ID>" \
  --select "url,id,title,warc_date,day,month,year,curlielabels_en as ows_curlielabel" \
  | owilix plugin owilix.plugins.push.opensearch.OpenSearchIndexer index "ows-title-index" \
      convert_fn=owilix.plugins.push.convert.OURSConverter
```

Notes:
- The OpenSearch index is created automatically if missing.
- Use `delete_first=true` to recreate the index on each run.