owi-cli
Command line interface for accessing the Open Web Index shards and slices.
Star 0
Fork 0
Watch
owi-cli docs source advanced.md
3.6 kB · 92 lines
Markdown
at main

Advanced Usage and Configuration #

Viewing Logs #

owi admin logs allows viewing different kinds of logs:

owi admin logs error          # Show error logs
owi admin logs all --lines 100 # Show all logs

Debugging Problems #

owilix usually only reports user information with a log level only on exceptions, errors or warning. However, to get more information, you can set teh --loglevel DEBUG option, which will log everything to the output.

e.g.

owilix --loglevel DEBUG remote ls all/access=public;subResourceType*=.*parq.*

Note that by prefixing "all:" to the log level, you can extend the log level to the root logger (and thus to all modules)

Using the LexisHTTP repository (experimental) #

owilix is primarily using direct file based repositories, particularly iRODS. However, this requires non-default ports to be open, which can cause problems in non-standard environments.

To overcome this, owilix provides the so called lexishttprepository, which is deactivated by default. The repository is slower and does not support all commands as of know. But you should be able to list and pull datasets with it.

The option --remote lexishttp activates the repository. Note that in order to keep the data consistent, no other datacenter which is available via the Lexis Portal should be active.

owilix --remotes lexishttp remote ls all/access=public;dataCenter=it4i files=**/language=eng/index*

Note also, the filtering for data center changes. Usually, owilix is configured with one repository per data center in order to enable maximum parallel data transfer. However, the lexis http repo aggregates datasets over data centers. The specifier specifies the repository to be used, here lexishttp, which does not filter appropriately for the dataCenter. Consequently, filtering for datacenter must be done in the query part, as in the example above.

Administrative Commands #

Should be handled with care

Setting metadata directly #

Mainly intended to be used after an HPC execution workflow. Metadata are inferred automatically and placeholders in strings (e.g. {resourcetype}) are also replaced

There is an interactive mode for correcting metadata which can be turned off using owilix --yes ......

owilix admin set_irods_metadata irods://username:password@server:port/ZONE/path_in_zone metadta1=key1

Alternative, metdata can be set using a py4lexis session without specifying username:password above (but might require login via B2ACCESS browser login)

admin set_irods_metadata owilix://<ZONE>/[public|project]/<DatasetID> do_infer=True do_count=True metadata1=value1 metadata2=value2

Example 1: Establishing a Session and Querying Data #

from py4lexis.session import LexisSession
import fsspec, irods_fsspec, duckdb
from py4lexis.lexis_irods import iRODS

# Authenticate
session = LexisSession(in_cli=True)

# Setup filesystem and retrieve data
irods = iRODS(session)
fs = fsspec.filesystem("irods", session=irods._iRODS__get_irods_session())
files = fs.glob('/IT4ILexisV2/public/proj862c5962623246664c1fda27b7afb108/**/*.parquet')

# Register filesystem in DuckDB and execute a query
parquet_files = ["irods://" + file for file in files]
duckdb.register_filesystem(fs)
df = duckdb.sql(f"SELECT COUNT(*) FROM read_parquet({parquet_files[:2]})").df()
print(df)

Example 2: Running a Filtered Query #

query = f"SELECT url FROM read_parquet('{files[0]}') WHERE url LIKE 'https://www.ru.nl/' LIMIT 5"
physical_plan = duckdb.sql(f"EXPLAIN {query}").df().iloc[0, 1]
print(physical_plan)