Scalable and distributed custom feed generator, ott - on that topic

Add cloudflare tunnel to figure out how to expose this from homelab

+9 -3
README.md
··· 4 4 5 5 The flow is as follows: 6 6 7 - 1. Two fluvio connector services consume wss streams, one for raw-posts and one for raw-likes 8 - 2. ott-filter consumes the keyed posts, and likes streams and keeps count on likes and other filters. 7 + 1. Two fluvio connector services consume wss streams, one for raw-posts and one for raw-likes. 8 + 2. ott-filter consumes the keyed posts and likes streams, and keeps count on likes and other filters. 9 9 It sends the passing posts to the fluvio topic posts. 10 10 3. ott-embed consumes the posts topic, embeds them with tei running on host and stores the vectors in a pg cluster 11 - 4. ott-xrpc listens to getFeedSkeleton requests, gets the users last likeed post and gets similar posts from the pg db. 11 + 4. ott-xrpc listens to getFeedSkeleton requests, gets the users last liked post and gets similar posts from the pg db. 12 12 13 13 Still work in progress, especially the ott-xrpc service isn't fleshed out yet. Also I intend to add a VIP stream so that all posts 14 14 liked by a feed user are guaranteed to pass the filter. ··· 17 17 This is still the intention. 18 18 19 19 # Setup 20 + Note that the ott-helm currently assumes that you intend to deploy through a cloudflare zerotrust tunnel, and defaults to starting up a cloudflared daemon 21 + forwarding to the ott-xrpc service. If your DNS isn't configured like that and you have no tunnel simply disable cloudflare in your helm values. 22 + 23 + Also note that this should likely not be done at all just yet, this software is in early development. 20 24 21 25 ## Install dependencies 22 26 ··· 46 50 ```shell 47 51 skaffold run 48 52 ``` 53 + 54 +
+38
helm/ott/templates/cloudflared_tunnel.yaml
··· 1 + {{- if .Values.cloudflare.enabled }} 2 + apiVersion: v1 3 + kind: Secret 4 + metadata: 5 + name: cloudflare-tunnel-token 6 + namespace: {{ .Values.cloudflared.namespace }} 7 + type: Opaque 8 + stringData: 9 + token: {{ required "cloudflared.token is required" .Values.cloudflared.token }} 10 + --- 11 + apiVersion: apps/v1 12 + kind: Deployment 13 + metadata: 14 + name: cloudflare-tunnel 15 + spec: 16 + replicas: 1 17 + selector: 18 + matchLabels: 19 + app: cloudflare-tunnel 20 + template: 21 + metadata: 22 + labels: 23 + app: cloudflare-tunnel 24 + spec: 25 + containers: 26 + - name: cloudflared 27 + image: "cloudflare/cloudflared:latest" 28 + args: 29 + - tunnel 30 + - run 31 + - --token=$(CLOUDFLARE_TUNNEL_TOKEN) 32 + env: 33 + - name: CLOUDFLARE_TUNNEL_TOKEN 34 + valueFrom: 35 + secretKeyRef: 36 + name: cloudflare-tunnel-token 37 + key: token 38 + {{- end }}
+16 -1
helm/ott/templates/deployments.yaml helm/ott/templates/services.yaml
··· 1 - {{- range .Values.deployments }} 1 + {{- range .Values.services }} 2 2 --- 3 3 apiVersion: apps/v1 4 4 kind: Deployment ··· 22 22 imagePullPolicy: {{ .image.pullPolicy }} 23 23 env: 24 24 {{- toYaml .env | nindent 10 }} 25 + --- 26 + apiVersion: v1 27 + kind: Service 28 + metadata: 29 + name: {{ .name }} 30 + labels: 31 + app: {{ .name }} 32 + spec: 33 + selector: 34 + app: {{ .name }} 35 + ports: 36 + - port: 80 37 + targetPort: 8080 38 + protocol: TCP 39 + type: ClusterIP 25 40 {{- end }}
+4 -1
helm/ott/values.yaml
··· 2 2 imageName: loaded-pg:17 3 3 migration_image_fqn: migration-pg:17 4 4 5 + cloudflare: 6 + enabled: true 7 + token: {} 5 8 6 - deployments: 9 + services: 7 10 - name: likes-connector 8 11 image: 9 12 fqn: likes-connector
+6 -5
skaffold.yaml
··· 85 85 valuesFiles: 86 86 - helm/ott/values.yaml 87 87 setValueTemplates: 88 + cloudflared.token: "{{.CLOUDFLARE_TUNNEL_TOKEN}}" 88 89 postgresql.migration_image_fqn: "{{.IMAGE_FULLY_QUALIFIED_migration_pg}}" 89 - deployments[0].image.fqn: "{{.IMAGE_FULLY_QUALIFIED_likes_connector}}" 90 - deployments[1].image.fqn: "{{.IMAGE_FULLY_QUALIFIED_posts_connector}}" 91 - deployments[2].image.fqn: "{{.IMAGE_FULLY_QUALIFIED_ott_filter}}" 92 - deployments[3].image.fqn: "{{.IMAGE_FULLY_QUALIFIED_ott_embed}}" 93 - deployments[4].image.fqn: "{{.IMAGE_FULLY_QUALIFIED_ott_xrpc}}" 90 + services[0].image.fqn: "{{.IMAGE_FULLY_QUALIFIED_likes_connector}}" 91 + services[1].image.fqn: "{{.IMAGE_FULLY_QUALIFIED_posts_connector}}" 92 + services[2].image.fqn: "{{.IMAGE_FULLY_QUALIFIED_ott_filter}}" 93 + services[3].image.fqn: "{{.IMAGE_FULLY_QUALIFIED_ott_embed}}" 94 + services[4].image.fqn: "{{.IMAGE_FULLY_QUALIFIED_ott_xrpc}}" 94 95 version: 0.1.0