# OWILIX Repository Package The `owilix.core.repository` package provides abstractions for local and remote dataset storage. **See also:** [Core Manager](core.md) | [FSSPEC Integration](fsspec_integration.md) ## Package Structure ``` owilix/core/repository/ ├── __init__.py # Package exports ├── base.py # AbstractRepository, PerformanceMetric ├── file.py # FileBasedRepository, LocalRepository ├── lexis.py # LexisRepository (py4lexis 5.x HTTP) └── aggregate.py # AggregatedRepository ``` ## Quick Start ```python from owilix.core.repository import LexisRepository, LocalRepository, AggregatedRepository # Via manager (recommended) from owilix.core import OWILIXManager manager = OWILIXManager() repos = manager.remote_data # AggregatedRepository # Direct instantiation lexis = LexisRepository(manager, lexis_project_id="my-project") local = LocalRepository(manager, path="~/.owi/{access}", repo_name="local") ``` ## Repository Types ### LexisRepository LEXIS HTTP repository using py4lexis 5.x session - **no python-irodsclient required**. ```python from owilix.core.repository import LexisRepository repo = LexisRepository( manager=manager, lexis_project_id="proj862c5962623246664c1fda27b7afb108", # Optional repo_name="lexis" ) # List datasets datasets = repo.list(access="public") # Get files files = repo.files(dataset, files_glob="**/*.parquet") # Download repo.get(dataset, "data.parquet", "/local/path") ``` **Key features:** - Uses `OWILexisDatasetAPI` for DDI metadata - Uses `Http2IrodsFileSystem` for file operations - Gets URL from `session.irods_http_api_url` (no manual config) ### FileBasedRepository Filesystem-based repository using fsspec. ```python from owilix.core.repository import FileBasedRepository repo = FileBasedRepository( manager=manager, path="~/.owi/{access}", # Must contain {access} protocol="file", # or "s3" repo_name="local" ) ``` ### LocalRepository Alias for `FileBasedRepository` with `protocol="file"`. ### AggregatedRepository Aggregates multiple repositories with performance-based selection. ```python from owilix.core.repository import AggregatedRepository agg = AggregatedRepository({ "it4i": lexis_repo, "local": local_repo }) # Lists from all repos, deduplicates by best performance datasets = agg.list(access="public") # Performance stats agg.store_stats_to_file("stats.json") agg.get_performance_stats() ``` ## Performance Tracking All repositories track list, upload, and download performance. ```python from owilix.core.repository import RepoPerformanceStats, PerformanceMetric # Access stats stats = repo.performance_stats print(stats.list_response_time.get_overall_average()) print(stats.download_bandwidth.get_overall_average()) # Serialize data = stats.to_json() restored = RepoPerformanceStats.from_json(data) ``` ## Abstract Base Class ```{eval-rst} .. autoclass:: owilix.core.repository.AbstractRepository :members: list, files, files_details, exists, put, get, rm, create, status ``` ## Configuration Repositories are configured in `~/.owi/owilix.cfg`: ```yaml repositories: config: it4i: repository: lexis options: project_name: openwebsearch lexis_project_id: proj862c5962623246664c1fda27b7afb108 local: repository: file options: path: ~/.owi/{access} selected_remote: - it4i selected_local: - local ``` **Repository types:** - `lexis+http`: LexisRepository (HTTP API) - `file`: LocalRepository - `s3a`: S3FileBasedRepository ## Legacy Aliases For backward compatibility: - `LEXISIrodsRepository` → `LexisRepository` - `LEXISIrodsHTTPRepository` → `LexisRepository`