🧮 Open, serverless, and local friendly Data Platform for the Filecoin Ecosystem
README.md

Filecoin Data Portal #

Open datasets for the Filecoin ecosystem, built from open source code and published daily.

Website · Datasets · Docs · SKILL.md

Build License Stars

🚀 Quickstart #

Every dataset is a Parquet file, rebuilt daily. Browse them at filecoindataportal.xyz/datasets.

https://data.filecoindataportal.xyz/<dataset>.parquet

Query them with any Parquet reader. With DuckDB (or SQL Workbench in the browser):

select date, onboarded_pibs
from 'https://data.filecoindataportal.xyz/daily_network_metrics.parquet'
order by date desc
limit 10;

With Python:

import polars as pl

pl.read_parquet("https://data.filecoindataportal.xyz/miners.parquet")

All datasets are also available in one DuckDB file:

attach 'https://data.filecoindataportal.xyz/fdp.duckdb' as fdp (read_only);

🤖 Agents #

Point your agent to SKILL.md. It explains how to choose, query, and interpret datasets, and links the Markdown dataset catalog.

⚙️ Development #

Run the pipeline with uv. On Nix, nix develop provides a shell with pinned tool versions.

  • uv run fdp list: list assets
  • uv run fdp materialize: materialize all assets
  • uv run fdp materialize main.providers: materialize one asset (add --with-deps to refresh its dependencies)
  • uv run fdp query "select * from main.providers limit 10": query the local database
  • make check: lint and validate assets
  • make test: run data tests
  • make publish: publish datasets to R2
  • make restore SNAPSHOT_URL=...: replace local fdp.duckdb with a full-state snapshot

Secrets live in .env (load with uv run --env-file .env ...):

ENCODED_GOOGLE_APPLICATION_CREDENTIALS= # base64 Google service account JSON
R2_ACCESS_KEY_ID=
R2_SECRET_ACCESS_KEY=
R2_ACCOUNT_ID=
R2_BUCKET=
CLOUDFLARE_API_TOKEN= # needs Cache Purge on filecoindataportal.xyz
CLOUDFLARE_ZONE_ID=d301192e601454a9cf7a87b4cd6ee243

📃 Disclaimer #

Datasets are provided "as is", without express or implied warranties or guarantees. Use them at your own risk.

📝 Licenses #

The Filecoin Data Portal is licensed under the MIT License.