Query Slice Command #

The query slice command is a powerful tool for creating new datasets (subsets) from existing local or remote datasets based on SQL queries. It leverages DuckDB to efficiently process parquet files and generate new datasets adhering to the OWI metadata schema.

Usage #

owi query slice [OPTIONS]

Key Arguments #

• Target Source:

  • -L, --local SPECIFIER: Slice from a local dataset.
  • -R, --remote SPECIFIER: Slice from a remote dataset (requires files to be accessible/pulled or creates empty structure for metadata slicing). Note: Slicing usually operates on downloaded files.

• Filtering & Selection:

  • --where TEXT: SQL WHERE clause to filter records (e.g., url_suffix='de').
  • --select TEXT: SQL SELECT clause to choose columns (default: *).

• Output Dataset:

  • --collection NAME: Name of the collection for the new dataset.
  • --access LEVEL: Access level for the new dataset (default: public). Options: public, project, user.
  • --id UUID: Target an existing dataset by ID. If specified, the slice result is appended/merged into this dataset.

• Execution Control:

  • --yes: Skip confirmation prompts.
  • --format FORMAT: Output format (table, json, jsonl).

Workflows #

1. Creating a New Slice #

To create a new dataset containing only German URLs from a source dataset:

owi query slice -L "all:latest" --where "url_suffix='de'" --collection "german_subset" --yes

This will:

1. Discover files in the source dataset.
2. Execute the DuckDB query SELECT * FROM source WHERE url_suffix='de'.
3. Write the result to a new dataset under ~/.owi/public/german_subset/<NEW_ID>.
4. Infer metadata (file counts, size) and save it.

2. Updating (Appending to) a Slice #

To add French URLs to the same dataset created above, use the --id flag:

# First, find the ID of the dataset created in step 1
owi local ls all/collectionName=german_subset

# Then slice and append
owi query slice -L "all:latest" --where "url_suffix='fr'" --collection "german_subset" --id <EXISTING_ID> --yes

3. Automated Slicing with JSON Output #

For scripts and automation pipelines, use --format json. This outputs a structured JSON array containing details of the created/updated dataset, while suppressing progress bars.

owi query slice -L "all:latest" ... --format json

Output Example:

[
  {
    "dataset": {
      "id": "15c682b7-7615-470e-8669-02fc667e3a62",
      "path": "/Users/user/.owi/public/german_subset/15c682b7-...",
      "collectionName": "german_subset",
      "access": "public",
      "title": "OWI-Open Web Index-german_subset...",
      "files": 150,
      "size": 104857600
    }
  }
]

Technical Details #

Access Levels #

The access parameter determines where the dataset is stored locally:

• public: ~/.owi/public/<collection>/
• project: ~/.owi/project/<collection>/
• user: ~/.owi/user/<collection>/

Fix Note: Previous issues with access being ignored (defaulting to None) have been resolved.

Data Provenance #

The sliced dataset includes provenance information in its metadata, recording the source dataset ID, the query used (select, where), and the timestamp of the slice operation.

CIFF Files #

If the --ciff flag is enabled (default), the command also slices associated CIFF (Common Index File Format) index files. For every partition directory of the sliced dataset, each matching source index.ciff.gz is filtered down to the documents that survived the slice (matched by the parquet id column against the index collection_docid), and the non-empty results are merged into a single index.ciff.gz for that partition.

Notes:

• If none of the sliced document ids appear in the source indexes, a well-formed but empty index is written and a warning is printed (sliced index ... contains 0 documents). An empty index usually means the source index uses a different document identifier scheme than the parquet id column.
• If a partition has no matching source index, a warning is printed and no index is written for that partition.
• Requires ciff-toolkit >= 0.2.2.
• Use --threads N to control the parallelism of the index filtering.