OWIlix Commands and Usage #

The OWIlix CLI provides a suite of commands for managing datasets, interacting with remote repositories, and performing data analysis.

Command Groups #

Group	Description	Documentation
local	Manage datasets in your local repository.	Read More
remote	Interact with remote data centers (list, search, pull, upload, diagnose).	Read More
query	SQL-based analysis and filtering (slice, stats, less).	Read More
config	View and modify CLI configuration.	Read More
admin	Administrative tasks, repository/path diagnostics, and filesystem inspection.	Read More
http	Optional local HTTP API preview.	Read More

Detailed Documentation #

• Local Commands: ls, rm, analyze, init
• Remote Commands: ls, search, pull, push, upload, diff, summarize, summarize-hosts, verify, doctor, logout
  • Verifying mirrored data: presence, readability and checksums — and why a hash alone does not prove the data is usable.
• Admin Commands: repos, path, fs, logs, stats, check
• HTTP API Preview: serve
• Query Commands:
  • Slice: Create new datasets from queries.
  • WARC: WARC file extraction.
  • Graphs: Web graph analysis.
  • less (interactive data browser), stats, sites, aggregate
  • Supports --search remote-index prefiltering and --async mode for concurrent query execution
• Configuration: list, get, set

Global Options #

The following flags can be used with almost any command:

Flag	Short	Description
--verbose	-v	Enable verbose output (debug logs).
--loglevel	-l	Set log level: DEBUG, INFO, WARNING, ERROR.
--yes	-y	Auto-confirm interactive prompts.
--config	-c	Specify a custom configuration file path.
--format	-f	Output format: table, json, jsonl.
--output	-o	Write output to a file.
--no-display		Suppress console output (useful for scripting).
--target	-t	Override the target directory for data.
--no-progress	-N	Suppress progress bars.
--sql-log		Write executed SQL statements to ~/.owi/.logs/sql.log for debugging.

Defaults & Specifiers #

Paths #

• Default Path: ~/.owi
• Can be modified via environment variable OWS_OWI_PATH or owi config.

Specifier Format #

Access datasets using the following specifier format: {scope|all}[:{YYYY-MM-DD|latest|YYYY-MM-DD#days|YYYY-MM-DD..YYYY-MM-DD}][/{key=value;key=value}]

Rules:

• all searches all configured sources.
• it4i and lrz remain valid legacy scope aliases and now translate to zone filters (IT4ILexisV2 and OWILRZZONE).
• ; combines metadata filters with AND.
• value1,value2 means OR within one field.
• key*= performs a regex match.
• key-= excludes exact values.

Examples:

• all:latest: Latest datasets from all configured sources.
• lexis:2025-01-01: Datasets from LEXIS on a specific date.
• all:2025-01-01..2025-01-10: Explicit inclusive date range.
• it4i:latest: Latest datasets in zone IT4ILexisV2.
• all/collectionName=main: Filter by collection name.
• all/zone=IT4ILexisV2,OWILRZZONE: Match either zone.
• all/title*=.*Open.*;zone-=OWILRZZONE: Regex title match while excluding LRZ.

Admin Commands #

Admin commands are used for troubleshooting and system monitoring.

• owi admin logs [selector]: View logs by selector (latest, events, sql, filename) or print paths with location.
• owi admin stats: View system statistics.
• owi admin repos: List configured repositories and their status.
• owi admin path <path> [--repository <name>]: Check whether a path exists in one/all repositories.
• owi admin fs <operation> <path> [options]: Run low-level filesystem commands (ls, exists, info, find, glob, cat, mkdir, cp, mv, rm).

Publishing to OpenSearch #

OWILIX can stream query results into OpenSearch using the built-in plugin and helper script.

Environment Variables #

Set these in .env-rc (the script sources it) or your shell:

export OWILIX_OPENSEARCH_HOST="https://opensearch.example.org:9200"
export OWILIX_OPENSEARCH_USERNAME="your-user"
export OWILIX_OPENSEARCH_PASSWORD="your-password"
export OWILIX_OPENSEARCH_VERIFY_CERTS="true"  # optional
export OWILIX_OPENSEARCH_CA_CERTS="/path/to/ca.pem"  # optional

Using scripts/ours_index.sh #

This script runs a query less against a dataset specifier and pipes JSONL into the OpenSearch indexer using OURSConverter.

./scripts/ours_index.sh "all/id=<DATASET_ID>" "ows-title-index"

Arguments:

• $1: dataset specifier (e.g. all/id=<DATASET_ID>)
• $2: OpenSearch index name

Direct CLI (no script) #

owilix --format jsonl query less --remote "all/id=<DATASET_ID>" \
  --select "url,id,title,warc_date,day,month,year,curlielabels_en as ows_curlielabel" \
  | owilix plugin owilix.plugins.push.opensearch.OpenSearchIndexer index "ows-title-index" \
      convert_fn=owilix.plugins.push.convert.OURSConverter

Notes:

• The OpenSearch index is created automatically if missing.
• Use delete_first=true to recreate the index on each run.