tangled.org
Overview#
This pr implements the first part of this proposal and introduces one new environment variable
- LogBucket, when set logs will be uploaded to the specified bucket.
go.mod
go.mod
This file has not been changed.
go.sum
go.sum
This file has not been changed.
nix/gomod2nix.toml
nix/gomod2nix.toml
This file has not been changed.
nix/modules/spindle.nix
nix/modules/spindle.nix
This file has not been changed.
-1
nix/vm.nix
-1
nix/vm.nix
spindle/config/config.go
spindle/config/config.go
This file has not been changed.
spindle/engine/engine.go
spindle/engine/engine.go
This file has not been changed.
spindle/engine/s3.go
spindle/engine/s3.go
This file has not been changed.
History
10 rounds
3 comments
jobala.tngl.sh
submitted
#9
2 commits
expand
collapse
spindle/engine: store workflow logs in s3
spindle/engine: read secrets from environment file
no conflicts, ready to merge
expand 0 comments
jobala.tngl.sh
submitted
#8
10 commits
expand
collapse
spindle/engine: store workflow logs in s3
spindle/engine: read secrets from environment file
nix,knotmirror: listen & sync local knots from local knotmirror
Signed-off-by: Seongmin Lee <git@boltless.me>
lexicons:
sync.requestCrawl and knot.subscribeRepos
Signed-off-by: Seongmin Lee <git@boltless.me>
nix,knotserver,knotmirror/xrpc:
sync.requestCrawl support
knotmirror ingests repo creation event from atproto relay through tap,
not from knotstream. Therefore, knotserver will request
`sync.requestCrawl` with optional new repo information to notify the
repo creation event.
Knot will call `sync.requestCrawl` on following cases:
- on startup
- when `/event` stream has failed
- on repo creation (mandatory)
Signed-off-by: Seongmin Lee <git@boltless.me>
ogre: rename appview/ogcard to ogre
And move it out of appview.
Signed-off-by: Anirudh Oppiliappan <anirudh@tangled.org>
ogre: request jpeg from avatar service
This plays nice with satori and doesn't require wacky conversions etc.
Signed-off-by: Anirudh Oppiliappan <anirudh@tangled.org>
nix,knotmirror: listen & sync local knots from local knotmirror
Signed-off-by: Seongmin Lee <git@boltless.me>
nix,knotmirror: listen & sync local knots from local knotmirror
Signed-off-by: Seongmin Lee <git@boltless.me>
spindle/engine: remove unrelated changes
expand 0 comments
jobala.tngl.sh
submitted
#7
12 commits
expand
collapse
spindle/engine: store workflow logs in s3
spindle/engine: read secrets from environment file
nix,knotmirror: listen & sync local knots from local knotmirror
Signed-off-by: Seongmin Lee <git@boltless.me>
lexicons:
sync.requestCrawl and knot.subscribeRepos
Signed-off-by: Seongmin Lee <git@boltless.me>
nix,knotserver,knotmirror/xrpc:
sync.requestCrawl support
knotmirror ingests repo creation event from atproto relay through tap,
not from knotstream. Therefore, knotserver will request
`sync.requestCrawl` with optional new repo information to notify the
repo creation event.
Knot will call `sync.requestCrawl` on following cases:
- on startup
- when `/event` stream has failed
- on repo creation (mandatory)
Signed-off-by: Seongmin Lee <git@boltless.me>
ogre: rename appview/ogcard to ogre
And move it out of appview.
Signed-off-by: Anirudh Oppiliappan <anirudh@tangled.org>
ogre: request jpeg from avatar service
This plays nice with satori and doesn't require wacky conversions etc.
Signed-off-by: Anirudh Oppiliappan <anirudh@tangled.org>
nix,knotmirror: listen & sync local knots from local knotmirror
Signed-off-by: Seongmin Lee <git@boltless.me>
nix,knotmirror: listen & sync local knots from local knotmirror
Signed-off-by: Seongmin Lee <git@boltless.me>
spindle/engine: remove unrelated changes
spindle/engine: add overwritten changes
spindle/engine: fix typo
expand 0 comments
jobala.tngl.sh
submitted
#6
11 commits
expand
collapse
spindle/engine: store workflow logs in s3
spindle/engine: read secrets from environment file
nix,knotmirror: listen & sync local knots from local knotmirror
Signed-off-by: Seongmin Lee <git@boltless.me>
lexicons:
sync.requestCrawl and knot.subscribeRepos
Signed-off-by: Seongmin Lee <git@boltless.me>
nix,knotserver,knotmirror/xrpc:
sync.requestCrawl support
knotmirror ingests repo creation event from atproto relay through tap,
not from knotstream. Therefore, knotserver will request
`sync.requestCrawl` with optional new repo information to notify the
repo creation event.
Knot will call `sync.requestCrawl` on following cases:
- on startup
- when `/event` stream has failed
- on repo creation (mandatory)
Signed-off-by: Seongmin Lee <git@boltless.me>
ogre: rename appview/ogcard to ogre
And move it out of appview.
Signed-off-by: Anirudh Oppiliappan <anirudh@tangled.org>
ogre: request jpeg from avatar service
This plays nice with satori and doesn't require wacky conversions etc.
Signed-off-by: Anirudh Oppiliappan <anirudh@tangled.org>
nix,knotmirror: listen & sync local knots from local knotmirror
Signed-off-by: Seongmin Lee <git@boltless.me>
nix,knotmirror: listen & sync local knots from local knotmirror
Signed-off-by: Seongmin Lee <git@boltless.me>
spindle/engine: remove unrelated changes
spindle/engine: add overwritten changes
expand 0 comments
jobala.tngl.sh
submitted
#5
10 commits
expand
collapse
spindle/engine: store workflow logs in s3
spindle/engine: read secrets from environment file
nix,knotmirror: listen & sync local knots from local knotmirror
Signed-off-by: Seongmin Lee <git@boltless.me>
lexicons:
sync.requestCrawl and knot.subscribeRepos
Signed-off-by: Seongmin Lee <git@boltless.me>
nix,knotserver,knotmirror/xrpc:
sync.requestCrawl support
knotmirror ingests repo creation event from atproto relay through tap,
not from knotstream. Therefore, knotserver will request
`sync.requestCrawl` with optional new repo information to notify the
repo creation event.
Knot will call `sync.requestCrawl` on following cases:
- on startup
- when `/event` stream has failed
- on repo creation (mandatory)
Signed-off-by: Seongmin Lee <git@boltless.me>
ogre: rename appview/ogcard to ogre
And move it out of appview.
Signed-off-by: Anirudh Oppiliappan <anirudh@tangled.org>
ogre: request jpeg from avatar service
This plays nice with satori and doesn't require wacky conversions etc.
Signed-off-by: Anirudh Oppiliappan <anirudh@tangled.org>
nix,knotmirror: listen & sync local knots from local knotmirror
Signed-off-by: Seongmin Lee <git@boltless.me>
nix,knotmirror: listen & sync local knots from local knotmirror
Signed-off-by: Seongmin Lee <git@boltless.me>
spindle/engine: remove unrelated changes
expand 0 comments
jobala.tngl.sh
submitted
#4
9 commits
expand
collapse
spindle/engine: store workflow logs in s3
spindle/engine: read secrets from environment file
nix,knotmirror: listen & sync local knots from local knotmirror
Signed-off-by: Seongmin Lee <git@boltless.me>
lexicons:
sync.requestCrawl and knot.subscribeRepos
Signed-off-by: Seongmin Lee <git@boltless.me>
nix,knotserver,knotmirror/xrpc:
sync.requestCrawl support
knotmirror ingests repo creation event from atproto relay through tap,
not from knotstream. Therefore, knotserver will request
`sync.requestCrawl` with optional new repo information to notify the
repo creation event.
Knot will call `sync.requestCrawl` on following cases:
- on startup
- when `/event` stream has failed
- on repo creation (mandatory)
Signed-off-by: Seongmin Lee <git@boltless.me>
ogre: rename appview/ogcard to ogre
And move it out of appview.
Signed-off-by: Anirudh Oppiliappan <anirudh@tangled.org>
ogre: request jpeg from avatar service
This plays nice with satori and doesn't require wacky conversions etc.
Signed-off-by: Anirudh Oppiliappan <anirudh@tangled.org>
nix,knotmirror: listen & sync local knots from local knotmirror
Signed-off-by: Seongmin Lee <git@boltless.me>
nix,knotmirror: listen & sync local knots from local knotmirror
Signed-off-by: Seongmin Lee <git@boltless.me>
expand 0 comments
jobala.tngl.sh
submitted
#3
8 commits
expand
collapse
spindle/engine: store workflow logs in s3
spindle/engine: read secrets from environment file
nix,knotmirror: listen & sync local knots from local knotmirror
Signed-off-by: Seongmin Lee <git@boltless.me>
lexicons:
sync.requestCrawl and knot.subscribeRepos
Signed-off-by: Seongmin Lee <git@boltless.me>
nix,knotserver,knotmirror/xrpc:
sync.requestCrawl support
knotmirror ingests repo creation event from atproto relay through tap,
not from knotstream. Therefore, knotserver will request
`sync.requestCrawl` with optional new repo information to notify the
repo creation event.
Knot will call `sync.requestCrawl` on following cases:
- on startup
- when `/event` stream has failed
- on repo creation (mandatory)
Signed-off-by: Seongmin Lee <git@boltless.me>
ogre: rename appview/ogcard to ogre
And move it out of appview.
Signed-off-by: Anirudh Oppiliappan <anirudh@tangled.org>
ogre: request jpeg from avatar service
This plays nice with satori and doesn't require wacky conversions etc.
Signed-off-by: Anirudh Oppiliappan <anirudh@tangled.org>
nix,knotmirror: listen & sync local knots from local knotmirror
Signed-off-by: Seongmin Lee <git@boltless.me>
expand 0 comments
jobala.tngl.sh
submitted
#2
7 commits
expand
collapse
spindle/engine: store workflow logs in s3
spindle/engine: read secrets from environment file
nix,knotmirror: listen & sync local knots from local knotmirror
Signed-off-by: Seongmin Lee <git@boltless.me>
lexicons:
sync.requestCrawl and knot.subscribeRepos
Signed-off-by: Seongmin Lee <git@boltless.me>
nix,knotserver,knotmirror/xrpc:
sync.requestCrawl support
knotmirror ingests repo creation event from atproto relay through tap,
not from knotstream. Therefore, knotserver will request
`sync.requestCrawl` with optional new repo information to notify the
repo creation event.
Knot will call `sync.requestCrawl` on following cases:
- on startup
- when `/event` stream has failed
- on repo creation (mandatory)
Signed-off-by: Seongmin Lee <git@boltless.me>
ogre: rename appview/ogcard to ogre
And move it out of appview.
Signed-off-by: Anirudh Oppiliappan <anirudh@tangled.org>
ogre: request jpeg from avatar service
This plays nice with satori and doesn't require wacky conversions etc.
Signed-off-by: Anirudh Oppiliappan <anirudh@tangled.org>
expand 1 comment
jobala.tngl.sh
submitted
#1
expand 0 comments
jobala.tngl.sh
submitted
#0
1 commit
expand
collapse
spindle/engine: store workflow logs in s3
expand 2 comments
spindle/config/config.go:44-51 — this will result in SPINDLE_S3_LOG_BUCKET, but
nix/modules/spindle.nix:147 we reference SPINDLE_NIXERY_PIPELINES_LOG_BUCKET here.
spindle/engine/engine.go:60 — probably good to parameterize the bucket name from config as well!
spindle/engine/s3.go:20 — Go convention is camel case, so BaseS3Path is preferred here.
spindle/engine/s3.go:38 — also camel case here please: s3Key
spindle/engine/engine.go:154 — this is a no-op with just a single argument.
nix/modules/spindle.nix:148-150 — we should ideally use EnvironmentFile here. Refer to how secrets are passed through to the appview service, for example.
High level design concern: NewS3 creates a new S3 client for each workflow (on every defer). We should ideally initialize this upon spindle startup and passed through via the engine struct.
We should ideally initialize this upon spindle startup and passed through via the engine struct.
Doing this means that each engine will have to register a clean up task for uploading logs which is easy to forget. Doing it in spindle/engine means that new engines don't have to worry about how to upload logs.
sorry, but you will need to rebase with master before submitting!