···1212all the data from the none bsky pdses and then processes it into a 2 massive
1313json file.
14141515-To run the fetcher, run the following command:
1515+To run the fetcher, run the following commands:
1616+1717+1. First we need to fetch a new instance of the pdses list. This isn't needed
1818+ if you already have all the pdses you want to check.
1919+2020+```bash
2121+curl "https://raw.githubusercontent.com/mary-ext/atproto-scraping/refs/heads/trunk/state.json" -o data/data.json
2222+```
2323+2424+2. Then we can run the fetcher:
16251726```bash
1827pnpm run gen