iRODS# FSSpec Integration #

OWILIX uses the Filesystem Spec (fsspec) library to provide a unified interface for accessing different storage backends.

Http2IrodsFileSystem #

The core component for LEXIS integration is Http2IrodsFileSystem. This custom implementation allows standard filesystem operations (ls, open, get) over the LEXIS iRODS HTTP API.

Key Characteristics:

• HTTP-based: Works through firewalls where standard iRODS TCP (1247) is blocked.
• ReadOnly: Primarily designed for reading/listing datasets.
• Session-aware: Uses token_provider callback to maintain valid authentication during long operations.

Usage #

Typically used internally by LexisRepository:

fs = Http2IrodsFileSystem(
    irods_http_url="https://..../irods/api/v1",
    token_provider=session.get_token
)
fs.ls("/zone/project/...")

(fsspec) implementation for iRODS, enabling direct interaction with iRODS data objects and collections using standard Python file APIs. This integration is particularly optimized for data science workflows, including direct querying of Parquet files via DuckDB.

Overview #

The integration exposes the iRODS filesystem via the http2irods protocol. It supports:

• Recursive File Listing: Optimized implementations for fast directory traversal.
• Range Requests: Efficient partial reading of large files (crucial for Parquet/DuckDB).
• Concurrency: Support for asynchronous operations using httpx.
• Integrations: Seamless compatibility with duckdb, pandas, dask, and other fsspec-compatible libraries.

Architecture #

Two implementations are provided:

1. Sync (Http2IrodsFileSystem):

   • Based on irods_http_client and requests.
   • Robust, thread-safe (via connection pooling).
   • Best for standard scripting and sequential access.

2. Async (AsyncHttp2IrodsFileSystem):

   • Based on httpx.
   • Supports asyncio for non-blocking operations.
   • Optimized for high-concurrency scenarios (e.g., fetching many small files).

Usage #

Basic Usage #

from owilix.core.fsspec import Http2IrodsFileSystem

# Initialize
fs = Http2IrodsFileSystem(irods_client=client, url_base="https://ure.lexis.tech/...")

# List files
files = fs.ls("/zone/path/to/collection")

# Read file
with fs.open("/zone/path/to/file.txt", "rb") as f:
    content = f.read()

DuckDB Integration #

Query Parquet files stored in iRODS directly without downloading them locally:

import duckdb
from owilix.core.fsspec import Http2IrodsFileSystem

# Register filesystem
fs = Http2IrodsFileSystem(...)
duckdb.register_filesystem(fs)

# Query
df = duckdb.query("SELECT * FROM read_parquet('http2irods:///zone/path/data.parquet') LIMIT 10").df()

Performance #

The implementation has been benchmarked against standard iRODS operations and optimized for throughput and latency.

Listing Performance #

The find() method uses GenQuery for optimized recursive listing:

• Method: Queries.execute_genquery() with pagination
• Query: SELECT DATA_NAME, COLL_NAME, DATA_SIZE, DATA_MODIFY_TIME, DATA_CREATE_TIME WHERE COLL_NAME LIKE '{path}%'
• Speed: 1.4s for 1524 files (~87x faster than recursive approach)
• Pagination: Handles 360-row server limit automatically

GenQuery vs Previous Approaches:

Method	Speed (1524 files)	HTTP Calls
iRODSCollecion recursive	~120s	O(directories)
Collections.list(recurse=1)	5.1s	1 (no type info)
GenQuery	1.4s	~5 (paginated)

Metadata Caching #

find() populates an internal cache (_info_cache) with file metadata. Subsequent info() calls for cached paths return instantly:

# find() populates cache
files = fs.find("/zone/path", detail=True)

# info() uses cache - no HTTP call needed
info = fs.info(files[0]["name"])  # instant

# Invalidate cache if needed
fs.invalidate_cache()  # clear all
fs.invalidate_cache("/zone/path/file.txt")  # clear specific path

GenQuery Fields #

Available metadata from find(detail=True):

Field	Description
name	Full path
type	"file" or "directory"
size	Size in bytes
mtime	Modification timestamp
ctime	Creation timestamp

Data Access (Parquet/DuckDB) #

Benchmarks conducted on small Parquet files (~150KB - 3.5MB).

1. Sequential Read Throughput (Sync vs Async)

• Sync (Requests): ~3.2 MB/s (Optimal chunk size 1MB).
• Async (Httpx): ~0.7 MB/s (Single file sequential).
  • Note: Async sequential is slower due to HTTP/1.1 latency dominance on small chunks.

2. Concurrent Read Throughput

• Async (Concurrent): 5.35 MB/s (10 files in parallel).
• Speedup: ~1.7x faster than optimized Sync sequential.

3. DuckDB Query Performance (Sync)

• Configuration: 1MB Chunk Size (Optimal).
• count(*): ~1.3s
• DESCRIBE (Schema): ~1.1s
• SELECT ... LIMIT 100: ~8.0s (High latency due to partial range requests overhead on small files).
• SELECT <column>: ~3.5s

Optimization & Tunables #

• chunk_size: Default 1MB. Tuning this affects prefetching.
  • 256KB: Slightly faster for count(*) (1.25s).
  • 1MB: Balanced best performance.
  • 4MB: No significant gain.
• max_connections: Controls connection pool size. Increase for high-concurrency async workloads.

Async Support #

Using AsyncHttp2IrodsFileSystem enables concurrent fetching:

from owilix.core.fsspec import AsyncHttp2IrodsFileSystem

fs_async = AsyncHttp2IrodsFileSystem(...)
# fast concurrent reads
data = await fs_async._cat_file(path)

Note: The LEXIS iRODS server currently supports HTTP/1.1. While httpx is used, HTTP/2 multiplexing is not active, but client-side concurrency still yields significant performance gains.

API Reference #

Http2IrodsFileSystem #

class Http2IrodsFileSystem(fsspec.AbstractFileSystem):
    """
    fsspec filesystem for iRODS HTTP API.
    
    Provides sync and async file access to iRODS via HTTP API.
    Supports range requests for efficient partial reads.
    
    Attributes:
        protocol (str): "http2irods"
        async_impl (bool): True
    """
    
    def __init__(self, irods_client, url_base, chunk_size=1024*1024, max_connections=10, **kwargs):
        """
        Initialize the filesystem.
        
        Args:
            irods_client: Initialized irods_http_client instance (with token)
            url_base (str): Base URL of the iRODS HTTP API
            chunk_size (int): Check size for buffering (default 1MB)
            max_connections (int): Max connections for connection pool
        """
        ...

AsyncHttp2IrodsFileSystem #

class AsyncHttp2IrodsFileSystem(Http2IrodsFileSystem):
    """
    Experimental Async Http2IrodsFileSystem using httpx.
    Overrides async methods to use AsyncIrodsClient directly.
    """
    ...