# iRODS# FSSpec Integration

OWILIX uses the [Filesystem Spec (fsspec)](https://filesystem-spec.readthedocs.io/) library to provide a unified interface for accessing different storage backends.

## `Http2IrodsFileSystem`

The core component for LEXIS integration is `Http2IrodsFileSystem`. This custom implementation allows standard filesystem operations (ls, open, get) over the LEXIS iRODS HTTP API.

**Key Characteristics:**
- **HTTP-based**: Works through firewalls where standard iRODS TCP (1247) is blocked.
- **ReadOnly**: Primarily designed for reading/listing datasets.
- **Session-aware**: Uses `token_provider` callback to maintain valid authentication during long operations.

### Usage
Typically used internally by `LexisRepository`:
```python
fs = Http2IrodsFileSystem(
    irods_http_url="https://..../irods/api/v1",
    token_provider=session.get_token
)
fs.ls("/zone/project/...")
```
(fsspec) implementation for iRODS, enabling direct interaction with iRODS data objects and collections using standard Python file APIs. This integration is particularly optimized for data science workflows, including direct querying of Parquet files via DuckDB.

## Overview

The integration exposes the iRODS filesystem via the `http2irods` protocol. It supports:
- **Recursive File Listing**: Optimized implementations for fast directory traversal.
- **Range Requests**: Efficient partial reading of large files (crucial for Parquet/DuckDB).
- **Concurrency**: Support for asynchronous operations using `httpx`.
- **Integrations**: Seamless compatibility with `duckdb`, `pandas`, `dask`, and other fsspec-compatible libraries.

## Architecture

Two implementations are provided:

1.  **Sync (`Http2IrodsFileSystem`)**: 
    - Based on `irods_http_client` and `requests`.
    - Robust, thread-safe (via connection pooling).
    - Best for standard scripting and sequential access.

2.  **Async (`AsyncHttp2IrodsFileSystem`)**:
    - Based on `httpx`.
    - Supports `asyncio` for non-blocking operations.
    - Optimized for high-concurrency scenarios (e.g., fetching many small files).

## Usage

### Basic Usage

```python
from owilix.core.fsspec import Http2IrodsFileSystem

# Initialize
fs = Http2IrodsFileSystem(irods_client=client, url_base="https://ure.lexis.tech/...")

# List files
files = fs.ls("/zone/path/to/collection")

# Read file
with fs.open("/zone/path/to/file.txt", "rb") as f:
    content = f.read()
```

### DuckDB Integration

Query Parquet files stored in iRODS directly without downloading them locally:

```python
import duckdb
from owilix.core.fsspec import Http2IrodsFileSystem

# Register filesystem
fs = Http2IrodsFileSystem(...)
duckdb.register_filesystem(fs)

# Query
df = duckdb.query("SELECT * FROM read_parquet('http2irods:///zone/path/data.parquet') LIMIT 10").df()
```

## Performance

The implementation has been benchmarked against standard iRODS operations and optimized for throughput and latency.

### Listing Performance

The `find()` method uses **GenQuery** for optimized recursive listing:

- **Method**: `Queries.execute_genquery()` with pagination
- **Query**: `SELECT DATA_NAME, COLL_NAME, DATA_SIZE, DATA_MODIFY_TIME, DATA_CREATE_TIME WHERE COLL_NAME LIKE '{path}%'`
- **Speed**: **1.4s** for 1524 files (~87x faster than recursive approach)
- **Pagination**: Handles 360-row server limit automatically

**GenQuery vs Previous Approaches:**

| Method | Speed (1524 files) | HTTP Calls |
|--------|-------------------|------------|
| `iRODSCollecion` recursive | ~120s | O(directories) |
| `Collections.list(recurse=1)` | 5.1s | 1 (no type info) |
| **GenQuery** | **1.4s** | ~5 (paginated) |

### Metadata Caching

`find()` populates an internal cache (`_info_cache`) with file metadata. Subsequent `info()` calls for cached paths return instantly:

```python
# find() populates cache
files = fs.find("/zone/path", detail=True)

# info() uses cache - no HTTP call needed
info = fs.info(files[0]["name"])  # instant

# Invalidate cache if needed
fs.invalidate_cache()  # clear all
fs.invalidate_cache("/zone/path/file.txt")  # clear specific path
```

### GenQuery Fields

Available metadata from `find(detail=True)`:

| Field | Description |
|-------|-------------|
| `name` | Full path |
| `type` | "file" or "directory" |
| `size` | Size in bytes |
| `mtime` | Modification timestamp |
| `ctime` | Creation timestamp |


### Data Access (Parquet/DuckDB)

Benchmarks conducted on small Parquet files (~150KB - 3.5MB).

**1. Sequential Read Throughput (Sync vs Async)**
- **Sync (Requests)**: ~3.2 MB/s (Optimal chunk size 1MB).
- **Async (Httpx)**: ~0.7 MB/s (Single file sequential).
    - *Note*: Async sequential is slower due to HTTP/1.1 latency dominance on small chunks.

**2. Concurrent Read Throughput**
- **Async (Concurrent)**: **5.35 MB/s** (10 files in parallel).
- **Speedup**: ~1.7x faster than optimized Sync sequential.

**3. DuckDB Query Performance (Sync)**
- **Configuration**: 1MB Chunk Size (Optimal).
- **`count(*)`**: ~1.3s
- **`DESCRIBE`** (Schema): ~1.1s
- **`SELECT ... LIMIT 100`**: ~8.0s (High latency due to partial range requests overhead on small files).
- **`SELECT <column>`**: ~3.5s

### Optimization & Tunables

- **`chunk_size`**: Default 1MB. Tuning this affects prefetching.
    - 256KB: Slightly faster for `count(*)` (1.25s).
    - 1MB: Balanced best performance.
    - 4MB: No significant gain.
- **`max_connections`**: Controls connection pool size. Increase for high-concurrency async workloads.

## Async Support

Using `AsyncHttp2IrodsFileSystem` enables concurrent fetching:

```python
from owilix.core.fsspec import AsyncHttp2IrodsFileSystem

fs_async = AsyncHttp2IrodsFileSystem(...)
# fast concurrent reads
data = await fs_async._cat_file(path)
```

**Note**: The LEXIS iRODS server currently supports **HTTP/1.1**. While `httpx` is used, HTTP/2 multiplexing is not active, but client-side concurrency still yields significant performance gains.

## API Reference

### Http2IrodsFileSystem

```python
class Http2IrodsFileSystem(fsspec.AbstractFileSystem):
    """
    fsspec filesystem for iRODS HTTP API.
    
    Provides sync and async file access to iRODS via HTTP API.
    Supports range requests for efficient partial reads.
    
    Attributes:
        protocol (str): "http2irods"
        async_impl (bool): True
    """
    
    def __init__(self, irods_client, url_base, chunk_size=1024*1024, max_connections=10, **kwargs):
        """
        Initialize the filesystem.
        
        Args:
            irods_client: Initialized irods_http_client instance (with token)
            url_base (str): Base URL of the iRODS HTTP API
            chunk_size (int): Check size for buffering (default 1MB)
            max_connections (int): Max connections for connection pool
        """
        ...
```

### AsyncHttp2IrodsFileSystem

```python
class AsyncHttp2IrodsFileSystem(Http2IrodsFileSystem):
    """
    Experimental Async Http2IrodsFileSystem using httpx.
    Overrides async methods to use AsyncIrodsClient directly.
    """
    ...
```