[WIP] music platform user data scraper
teal-fm atproto
1# piper 2 3#### what is piper? 4 5piper is a teal-fm tool that will be used to scrape user data from variety of 6music providers. 7 8#### why doesn't it work? 9 10well its just a work in progress... we build in the open! 11 12## setup 13It is recommend to have port forward url while working with piper. Development or running from docker because of external callbacks. 14 15You have a couple of options 161. Setup the traditional port forward on your router 172. Use a tool like [ngrok](https://ngrok.com/) with the command `ngrok http 8080` or [Cloudflare tunnels](https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/get-started/create-remote-tunnel/) (follow the 2a. portion of the guide when you get to that point) 18 19Either way make note of what the publicly accessible domain name is for setting up env variables. It will be something like `https://piper.teal.fm` that you can access publicly 20 21#### env variables 22Copy [.env.template](.env.template) and name it [.env](.env) 23 24This is a break down of what each env variable is and what it may look like 25 26**_breaking piper/v0.0.2 changes env_** 27 28You now have to bring your own private key to run piper. Can do this via goat `goat key generate -t P-256`. You want the one that is labeled under "Secret Key (Multibase Syntax): save this securely (eg, add to password manager)" 29 30- `ATPROTO_CLIENT_SECRET_KEY` - Private key for oauth confidential client. This can be generated via goat `goat key generate -t P-256` 31- `ATPROTO_CLIENT_SECRET_KEY_ID` - Key ID for oauth confidential client. This needs to be persistent and unique, can use a timestamp. Here's one for you: `1758199756` 32 33 34- `SERVER_PORT` - The port piper is hosted on 35- `SERVER_HOST` - The server host. `localhost` is fine here, or `0.0.0.0` for docker 36- `SERVER_ROOT_URL` - This needs to be the pubically accessible url created in [Setup](#setup). Like `https://piper.teal.fm` 37- `SPOTIFY_CLIENT_ID` - Client Id from setup in [Spotify developer dashboard](https://developer.spotify.com/documentation/web-api/tutorials/getting-started) 38- `SPOTIFY_CLIENT_SECRET` - Client Secret from setup in [Spotify developer dashboard](https://developer.spotify.com/documentation/web-api/tutorials/getting-started) 39- `SPOTIFY_AUTH_URL` - most likely `https://accounts.spotify.com/authorize` 40- `SPOTIFY_TOKEN_URL` - most likely `https://accounts.spotify.com/api/token` 41- `SPOTIFY_SCOPES` - most likely `user-read-currently-playing user-read-email` 42- `CALLBACK_SPOTIFY` - The first part is your publicly accessible domain. So will something like this `https://piper.teal.fm/callback/spotify` 43 44- `ATPROTO_CLIENT_ID` - The first part is your publicly accessible domain. So will something like this `https://piper.teal.fm/.well-known/client-metadata.json` 45- `ATPROTO_METADATA_URL` - The first part is your publicly accessible domain. So will something like this `https://piper.teal.fm/.well-known/client-metadata.json` 46- `ATPROTO_CALLBACK_URL` - The first part is your publicly accessible domain. So will something like this `https://piper.teal.fm/callback/atproto` 47 48- `LASTFM_API_KEY` - Your lastfm api key. Can find out how to setup [here](https://www.last.fm/api) 49 50- `TRACKER_INTERVAL` - How long between checks to see if the registered users are listening to new music 51- `DB_PATH`= Path for the sqlite db. If you are using the docker compose probably want `/db/piper.db` to persist data 52 53 54 55## development 56 57make sure you have your env setup following [the env var setup](#env-variables) 58 59assuming you have go installed and set up properly: 60 61run some make scripts: 62 63``` 64 65make dev-setup 66``` 67 68install air: 69 70``` 71go install github.com/air-verse/air@latest 72``` 73 74run air: 75 76``` 77air 78``` 79air should automatically build and run piper, and watch for changes on relevant files. 80 81 82## tailwindcss 83 84To use tailwindcss you will have to install the tailwindcss cli. This will take the [./pages/static/base.css](./pages/static/base.css) and transform it into a [./pages/static/main.css](./pages/static/main.css) 85which is imported on the [./pages/templates/layouts/base.gohtml](./pages/templates/layouts/base.gohtml). When running the dev server tailwindcss will watch for changes and recompile the main.css file. 86 871. Install tailwindcss cli `npm install tailwindcss @tailwindcss/cli` 882. run `npx @tailwindcss/cli -i ./pages/static/base.css -o ./pages/static/main.css --watch` 89 90 91 92 93#### Lexicon changes 941. Copy the new or changed json schema files to the [lexicon folders](./lexicons) 952. run `make go-lexicons` 96 97Go types should be updated and should have the changes to the schemas 98 99#### docker 100We also provide a docker compose file to use to run piper locally. There are a few edits to the [.env](.env) to make it run smoother in a container 101`SERVER_HOST`- `0.0.0.0` 102`DB_PATH` = `/db/piper.db` to persist your piper db through container restarts 103 104Make sure you have docker and docker compose installed, then you can run piper with `docker compose up`