# Advanced Usage and Configuration

## Viewing Logs

`owi admin logs` allows viewing different kinds of logs:

```bash
owi admin logs error          # Show error logs
owi admin logs all --lines 100 # Show all logs
```

## Debugging Problems

`owilix` usually only reports user information with a log level only on exceptions, errors or warning. 
However, to get more information, you can set teh `--loglevel DEBUG` option, which will log everything to the output.

e.g. 
```bash 
owilix --loglevel DEBUG remote ls all/access=public;subResourceType*=.*parq.*
```

Note that by prefixing "all:" to the log level, you can extend the log level to the root logger (and thus to all modules)


## Using the `LexisHTTP` repository (experimental)

`owilix` is primarily using direct file based repositories, particularly iRODS. 
However, this requires non-default ports to be open, which can cause problems in non-standard environments. 

To overcome this, `owilix` provides the so called `lexishttp`repository, which is deactivated by default. 
The repository is slower and does not support all commands as of know. But you should be able to list and pull datasets with it. 

The option `--remote lexishttp` activates the repository. 
Note that in order to keep the data consistent, no other datacenter which is available via the Lexis Portal should be active.

```bash 
owilix --remotes lexishttp remote ls all/access=public;dataCenter=it4i files=**/language=eng/index*
```

Note also, the filtering for data center changes. 
Usually, `owilix` is configured with one repository per data center in order to enable maximum parallel data transfer. 
However, the lexis http repo aggregates datasets over data centers. 
The specifier specifies the repository to be used, here lexishttp, which does not filter appropriately for the dataCenter. 
Consequently, filtering for datacenter must be done in the query part, as in the example above. 

## Administrative Commands

Should be handled with care

### Setting metadata directly 

Mainly intended to be used after an HPC execution workflow. Metadata are inferred automatically and placeholders in strings (e.g. {resourcetype}) are also replaced

There is an interactive mode for correcting metadata which can be turned off using `owilix --yes ......`

```bash
owilix admin set_irods_metadata irods://username:password@server:port/ZONE/path_in_zone metadta1=key1
```

Alternative, metdata can be set using a py4lexis session without specifying username:password above (but might require login via B2ACCESS browser login)

```bash
admin set_irods_metadata owilix://<ZONE>/[public|project]/<DatasetID> do_infer=True do_count=True metadata1=value1 metadata2=value2
```

## Example 1: Establishing a Session and Querying Data
```python
from py4lexis.session import LexisSession
import fsspec, irods_fsspec, duckdb
from py4lexis.lexis_irods import iRODS

# Authenticate
session = LexisSession(in_cli=True)

# Setup filesystem and retrieve data
irods = iRODS(session)
fs = fsspec.filesystem("irods", session=irods._iRODS__get_irods_session())
files = fs.glob('/IT4ILexisV2/public/proj862c5962623246664c1fda27b7afb108/**/*.parquet')

# Register filesystem in DuckDB and execute a query
parquet_files = ["irods://" + file for file in files]
duckdb.register_filesystem(fs)
df = duckdb.sql(f"SELECT COUNT(*) FROM read_parquet({parquet_files[:2]})").df()
print(df)
```

## Example 2: Running a Filtered Query
```python
query = f"SELECT url FROM read_parquet('{files[0]}') WHERE url LIKE 'https://www.ru.nl/' LIMIT 5"
physical_plan = duckdb.sql(f"EXPLAIN {query}").df().iloc[0, 1]
print(physical_plan)
```