OWILIX Repository Package #

The owilix.core.repository package provides abstractions for local and remote dataset storage.

See also: Core Manager | FSSPEC Integration

Package Structure #

owilix/core/repository/
├── __init__.py     # Package exports
├── base.py         # AbstractRepository, PerformanceMetric
├── file.py         # FileBasedRepository, LocalRepository
├── lexis.py        # LexisRepository (py4lexis 5.x HTTP)
└── aggregate.py    # AggregatedRepository

Quick Start #

from owilix.core.repository import LexisRepository, LocalRepository, AggregatedRepository

# Via manager (recommended)
from owilix.core import OWILIXManager
manager = OWILIXManager()
repos = manager.remote_data  # AggregatedRepository

# Direct instantiation
lexis = LexisRepository(manager, lexis_project_id="my-project")
local = LocalRepository(manager, path="~/.owi/{access}", repo_name="local")

Repository Types #

LexisRepository #

LEXIS HTTP repository using py4lexis 5.x session - no python-irodsclient required.

from owilix.core.repository import LexisRepository

repo = LexisRepository(
    manager=manager,
    lexis_project_id="proj862c5962623246664c1fda27b7afb108",  # Optional
    repo_name="lexis"
)

# List datasets
datasets = repo.list(access="public")

# Get files
files = repo.files(dataset, files_glob="**/*.parquet")

# Download
repo.get(dataset, "data.parquet", "/local/path")

Key features:

• Uses OWILexisDatasetAPI for DDI metadata
• Uses Http2IrodsFileSystem for file operations
• Gets URL from session.irods_http_api_url (no manual config)

FileBasedRepository #

Filesystem-based repository using fsspec.

from owilix.core.repository import FileBasedRepository

repo = FileBasedRepository(
    manager=manager,
    path="~/.owi/{access}",  # Must contain {access}
    protocol="file",  # or "s3"
    repo_name="local"
)

LocalRepository #

Alias for FileBasedRepository with protocol="file".

AggregatedRepository #

Aggregates multiple repositories with performance-based selection.

from owilix.core.repository import AggregatedRepository

agg = AggregatedRepository({
    "it4i": lexis_repo,
    "local": local_repo
})

# Lists from all repos, deduplicates by best performance
datasets = agg.list(access="public")

# Performance stats
agg.store_stats_to_file("stats.json")
agg.get_performance_stats()

Performance Tracking #

All repositories track list, upload, and download performance.

from owilix.core.repository import RepoPerformanceStats, PerformanceMetric

# Access stats
stats = repo.performance_stats
print(stats.list_response_time.get_overall_average())
print(stats.download_bandwidth.get_overall_average())

# Serialize
data = stats.to_json()
restored = RepoPerformanceStats.from_json(data)

Abstract Base Class #

.. autoclass:: owilix.core.repository.AbstractRepository
   :members: list, files, files_details, exists, put, get, rm, create, status

Configuration #

Repositories are configured in ~/.owi/owilix.cfg:

repositories:
  config:
    it4i:
      repository: lexis
      options:
        project_name: openwebsearch
        lexis_project_id: proj862c5962623246664c1fda27b7afb108
    local:
      repository: file
      options:
        path: ~/.owi/{access}
  selected_remote:
    - it4i
  selected_local:
    - local

Repository types:

• lexis+http: LexisRepository (HTTP API)
• file: LocalRepository
• s3a: S3FileBasedRepository

Legacy Aliases #

For backward compatibility:

• LEXISIrodsRepository → LexisRepository
• LEXISIrodsHTTPRepository → LexisRepository