PyArrow Streams #

owilix supports creating pyarrow streams from a select / where query and consuming it with different consumers, configured via the command line. Streaming offers a flexible way to work with OWI data that are either streamed from local files or remote files.

Streaming supports the following consumers (which can be again provide the stream to exgternal proceses)

Streaming to Stdout #

A standard case is to reuse the stream in another data via stdin. This can be done by using the owilix.core.stream.ConsumeToStdout consumer and the following command as example

owilix query stream --remote lrz:2023-12-3 select=url,title "where=url_suffix='at'" | nc -l 1234

Streaming to a host:port #

Another common use case is to stream the data to a network port. This can be done by using the owilix.core.stream.ConsumeToSocket consumer and the following command as example

owilix query stream --local all:2023-12-3 select=url,title "where=url_suffix='at'" consumer="owilix.core.stream.ConsumeToSocket" host="localhost" port=1234