- HTML 30%
- Python 26%
- TypeScript 23%
- Astro 13%
- CSS 8%
- 2 more languages <1%
Filecoin Data Portal #
Open datasets for the Filecoin ecosystem, built from open source code and published daily.
Website · Datasets · Docs · SKILL.md
🚀 Quickstart #
Every dataset is a Parquet file, rebuilt daily. Browse them at filecoindataportal.xyz/datasets.
https://data.filecoindataportal.xyz/<dataset>.parquet
Query them with any Parquet reader. With DuckDB (or SQL Workbench in the browser):
select date, onboarded_pibs
from 'https://data.filecoindataportal.xyz/daily_network_metrics.parquet'
order by date desc
limit 10;
With Python:
import polars as pl
pl.read_parquet("https://data.filecoindataportal.xyz/miners.parquet")
All datasets are also available in one DuckDB file:
attach 'https://data.filecoindataportal.xyz/fdp.duckdb' as fdp (read_only);
🤖 Agents #
Point your agent to SKILL.md. It explains how to choose, query, and interpret datasets, and links the Markdown dataset catalog.
⚙️ Development #
Run the pipeline with uv. On Nix, nix develop provides a shell with pinned tool versions.
uv run fdp list: list assetsuv run fdp materialize: materialize all assetsuv run fdp materialize main.providers: materialize one asset (add--with-depsto refresh its dependencies)uv run fdp query "select * from main.providers limit 10": query the local databasemake check: lint and validate assetsmake test: run data testsmake publish: publish datasets to R2make restore SNAPSHOT_URL=...: replace localfdp.duckdbwith a full-state snapshot
Secrets live in .env (load with uv run --env-file .env ...):
ENCODED_GOOGLE_APPLICATION_CREDENTIALS= # base64 Google service account JSON
R2_ACCESS_KEY_ID=
R2_SECRET_ACCESS_KEY=
R2_ACCOUNT_ID=
R2_BUCKET=
CLOUDFLARE_API_TOKEN= # needs Cache Purge on filecoindataportal.xyz
CLOUDFLARE_ZONE_ID=d301192e601454a9cf7a87b4cd6ee243
📃 Disclaimer #
Datasets are provided "as is", without express or implied warranties or guarantees. Use them at your own risk.
📝 Licenses #
The Filecoin Data Portal is licensed under the MIT License.