Command line interface for accessing the Open Web Index shards and slices.
owi-cli docs source config.md
7.0 kB · 303 lines
Markdown

Configuration Guide #

OWILIX uses YAML-based configuration stored at ~/.owi/owilix.cfg. This guide covers configuration structure, migration, and CLI commands.

Configuration Structure #

general:
  version: "4.0.0"
  server-update-url: "https://dashboard.ows.eu/api/owilix/default-cfg-4-0"
  license: "1.0"

server-config:
  version: "4.0"  # Config schema version
  last_update: "2026-01-05"

repositories:
  config:
    lexis:
      repository: lexis
      options:
        project_id: "proj862c5962623246664c1fda27b7afb108"
    local:
      repository: file
      options:
        path: ~/.owi/{access}
        collections_in_path: true
  selected_remote: [lexis]
  selected_local: [local]

profiles:
  default:
    showfields: "internalID,title,dataCenter,startDate,endDate,..."
    nodisplay: ".*"
    sort: "+startDate"

theme:
  info: "dim cyan"
  warning: "magenta"
  danger: "bold red"

user:
  name: "username"
  email: ""
  organisation: ""

Configuration Commands #

View Configuration #

# Show full config
owi config list

# Show specific keys (dot-notation)
owi config get repositories.config
owi config get server-config user.name

# Check version info
owi config version

Use Local Config Only #

By default, OWILIX merges in the latest server config on startup (remote repositories and defaults). You can disable that behavior per command with --no-remote-config.

owi --no-remote-config remote ls all
owi --no-remote-config remote pull all/id=abc123

Modify Configuration #

# Set a value
owi config set user.email=user@example.com
owi config set repositories.config.local.options.path=/custom/path/{access}

Add Repositories #

# Add a LEXIS repository
owi config add-repo my-lexis -t lexis --project-id proj123

# Add a local file repository
owi config add-repo my-local -t file --path /data/owi/{access}

# Add an S3 repository
owi config add-repo my-s3 -t s3a \
  --endpoint https://s3.example.com \
  --path bucket/{access} \
  --key xxx --secret yyy

Select Repositories #

# Show current selection
owi config select --show

# Add repository to selection
owi config select --add my-s3

# Remove repository from selection
owi config select --remove old-repo

# Modify local selection
owi config select --type local --add my-local

Migration #

# Check if migration is needed
owi config version

# Preview migration changes
owi config migrate --dry-run

# Apply migration (creates backup)
owi config migrate

Repository Types #

Type Description
lexis LEXIS platform via HTTP API (py4lexis 5.x)
file Local filesystem storage
s3a S3-compatible object storage

LEXIS Repository Options #

lexis:
  repository: lexis
  options:
    project_id: "proj862c5962623246664c1fda27b7afb108"

Local File Repository Options #

local:
  repository: file
  options:
    path: ~/.owi/{access}
    collections_in_path: true

S3 Repository Options #

my-s3:
  repository: s3a
  options:
    protocol: s3a
    endpoint: https://s3.example.com
    key: ${S3_KEY}      # can use env var expansion
    secret: ${S3_SECRET}
    path: bucket/{access}
    async: true
    anonymous: false    # set true for public buckets

Configuration Migration #

OWILIX tracks configuration versions via server-config.version. When upgrading:

  1. Backup: Automatic backup at ~/.owi/owilix.cfg.bak
  2. Sequential migrations: V1→V2→V3→V4
  3. Preserves custom repos: User-added repositories are not modified

Migration History #

Version Changes
V1 → V2 Repository types lexis updated to lexis+http
V2 → V3 Removed deprecated it4i, lrz, lexishttp; added unified lexis
V3 → V4 Updated server URL to default-cfg-4-0

Environment Variables #

Variable Description
OWS_OWI_PATH Base path for OWILIX data (default: ~/.owi)
PY4LEXIS_TOKEN Refresh token for authentication
OWILIX_LEXIS_PROJECT_NAME Project name (default: openwebsearch)
OWILIX_LEXIS_PROJECT_ID Project ID
OWILIX_IRODS_CONNECTION_TIMEOUT iRODS timeout in seconds (default: 120)

Troubleshooting #

Reset to Defaults #

# Delete config and restart with fresh defaults
rm ~/.owi/owilix.cfg
owi config version

View Backup #

cat ~/.owi/owilix.cfg.bak

Disable Server Config Merging #

owi config set general.server-update-url=""

Or in Python:


Repository Management #

Use config add-repo to easily add new repository definitions:

owi config add-repo my-s3 -t s3a --endpoint https://s3.example.com --key xxx --secret yyy

Use config select to manage active repositories (multi-selection supported):

# Add a repository to the active list
owi config select --add my-s3

# Remove a repository
owi config select --remove lexis

# Show current selection
owi config select --show

Advanced Configuration #

For complex nested configurations or inserting entire sections (like JSON trees), use add-node:

# Insert a complex dictionary at a specific key
owi config add-node repositories.config.custom_repo \
  "{\"repository\": \"s3a\", \"options\": {\"endpoint\": \"...\", \"bucket\": \"...\"}}" \
  --json

You can manipulate the configuration safely via the OWIlixConfig class (multi-key syntax supported):

from owilix.core.manager import OWIlixConfig, OWIlixManager

# Point the manager to an alternative OWI path (where owilix.cfg lives)
mgr = OWIlixManager(owi_path="/srv/owi")
cfg = mgr.config  # type: OWIlixConfig

# 1) Change local repo path
cfg.add_config("repositories.config.local.options.path", "/data/owilix/{access}")
cfg.save_config()

# 2) Add a new S3 repo
cfg.add_config("repositories.config.backup-s3.repository", "s3a")
cfg.add_config("repositories.config.backup-s3.options.protocol", "s3a")
cfg.add_config("repositories.config.backup-s3.options.endpoint", "https://s3.backup.net")
cfg.add_config("repositories.config.backup-s3.options.path", "owi-archive/{access}")
cfg.add_config("repositories.config.backup-s3.options.async", True)

# 3) Activate it as a remote (append to list)
# Multi-key 'selected_remote#999' appends to list if index is out of bounds
cfg.add_config("repositories.selected_remote#999", "backup-s3")
cfg.save_config()

# 4) Load without remote server merge
cfg_no_remote = OWIlixConfig(cfg.config_file, no_remote=True)

Selecting repositories at runtime #

OWIlixConfig.get_repositories supports additive selection using +:

# Use defaults + one extra
remotes, locals_ = cfg.get_repositories(remote=["+backup-s3"])

# Override defaults entirely (only these)
remotes, locals_ = cfg.get_repositories(remote=["backup-s3", "it4i"],
                                        local=["local"])