tangled.org / core
Monorepo for Tangled tangled.org
fork atom

knotmirror: introduce knotmirror #1160

merged opened by boltless.me targeting master from sl/knotmirror

knotmirror is an external service that is intended to be used by appview. It will ingest all known git repos and provide xrpc methods to inspect them, so appview won't need to fetch individual knots on every page render.

Signed-off-by: Seongmin Lee git@boltless.me

STACK 9
#1184 appview/repo: avoid extra xrpc call on empty repo
1/3
0
#2
#1168 knotmirror: add knotBackoff and reachability test
1/3
0
#4
#1167 nix: add knotmirror module
1/3
0
#4
#1164 appview,knotmirror: use knotmirror to read the repository
1/3
5
#4
#1163 lexicons: introduce repo.temp xrpc methods
1/3
3
#4
#1160 knotmirror: introduce knotmirror
1/3
4
#4
#1159 tapc: add tap client package
1/3
2
#4
#1158 nix: add more nix packages/modules related to atproto
1/3
3
#4
#1157 go.mod: bump indigo version to fork
1/3
1
#4
ABANDONED PULLS 4
#1166 knotmirror: add manual resync/cancel
3
#1
#1165 knotmirror: move git logic into GitMirrorManager
0
#1
#1162 knotmirror/db: add timestamps to track syncs
1
#1
#1161 knotmirror: switch to postgres for concurrent writes
3
#1
Labels

None yet.

assignee

None yet.

Participants 3
AT URI
at://did:plc:xasnlahkri4ewmbuzly2rlc5/sh.tangled.repo.pull/3mh3qbj6xvc22
+4 -4
Interdiff #0 → #1
.gitignore

This file has not been changed.

cmd/knotmirror/main.go

This file has not been changed.

flake.nix

This file has not been changed.

go.mod

This file has not been changed.

go.sum

This file has not been changed.

knotmirror/adminpage.go

This file has not been changed.

knotmirror/config/config.go

This file has not been changed.

knotmirror/crawler.go

This file has not been changed.

knotmirror/db/db.go

This file has not been changed.

+2 -2
knotmirror/db/hosts.go
··· 27 27 host.LastSeq, 28 28 ); err != nil { 29 29 return fmt.Errorf("upserting host: %w", err) 30 - }; 30 + } 31 31 return nil 32 32 } 33 33 ··· 47 47 return nil, nil 48 48 } 49 49 return nil, err 50 - }; 50 + } 51 51 return &host, nil 52 52 } 53 53
knotmirror/db/repos.go

This file has not been changed.

knotmirror/git.go

This file has not been changed.

knotmirror/knotmirror.go

This file has not been changed.

knotmirror/knotstream/knotstream.go

This file has not been changed.

knotmirror/knotstream/metrics.go

This file has not been changed.

knotmirror/knotstream/scheduler.go

This file has not been changed.

+1 -1
knotmirror/knotstream/slurper.go
··· 100 100 101 101 // TODO: include `cancel` function to kill subscription by hostname 102 102 sub := &subscription{ 103 - hostname: host.Hostname, 103 + hostname: host.Hostname, 104 104 scheduler: NewParallelScheduler( 105 105 s.cfg.ConcurrencyPerHost, 106 106 host.Hostname,
knotmirror/knotstream/subscription.go

This file has not been changed.

knotmirror/metrics.go

This file has not been changed.

knotmirror/models/models.go

This file has not been changed.

knotmirror/readme.md

This file has not been changed.

knotmirror/resyncer.go

This file has not been changed.

+1 -1
knotmirror/tapclient.go
··· 78 78 79 79 status := models.RepoStatePending 80 80 errMsg := "" 81 - u, err := url.Parse("http://"+record.Knot) // parsing with fake scheme 81 + u, err := url.Parse("http://" + record.Knot) // parsing with fake scheme 82 82 if err != nil { 83 83 status = models.RepoStateSuspended 84 84 errMsg = "failed to parse knot url"
knotmirror/templates/base.html

This file has not been changed.

knotmirror/templates/hosts.html

This file has not been changed.

knotmirror/templates/repos.html

This file has not been changed.

nix/gomod2nix.toml

This file has not been changed.

nix/pkgs/knot-mirror.nix

This file has not been changed.

History

5 rounds 4 comments
sign up or login to add to the discussion
boltless.me submitted #4
diff interdiff
1 commit
expand
6596d8b7
knotmirror: introduce knotmirror
2/3 failed, 1/3 success
expand
build.yml
fmt.yml
test.yml
expand 0 comments
pull request successfully merged
boltless.me submitted #3
diff interdiff
1 commit
expand
944b5a3f
knotmirror: introduce knotmirror
expand 0 comments
boltless.me submitted #2
diff interdiff
1 commit
expand
6dda5665
knotmirror: introduce knotmirror
expand 0 comments
boltless.me submitted #1
diff
1 commit
expand
e620e86a
knotmirror: introduce knotmirror
expand 4 comments
oppi.li

cmd/knotmirror/main.go:49 we can have the entire thing config'd via env vars, no need for more flags IMO. knotmirror/crawler.go:11: what does the Crawler do? knotmirror/models/models.go:45-63: do we need the func and the global?

will post this as a preliminary review, and do a further code review shortly, i can see that PRs above this stack mutate the knotmirror modules, such as the postgres migration. would appreciate if the stack were cleaned up to remove such mutations as it is slightly hard to follow.

boltless.me

no need for more flags IMO

The intention was to make knotmirror service embeddable by allowing full config as an input of knotmirror.Run(). And while doing that, I thought it would be nice if we can have cli flags to override the config.

what does the Crawler do?

Crawler came from tap.Crawler which periodically schedules the resync job for known repos. As we don't have a way to check the repository state, I leave the implementation blank. It isn't strictly necessary and not valuable to implement in current structure. See TAN-283 on how I am going to check repository sync state. So yeah, it does nothing right now.

do we need the func and the global?

Uh, I think I lost one of my changes while reordering commits... I will fix that.

would appreciate if the stack were cleaned up to remove such mutations as it is slightly hard to follow.

will do.

oyster.cafe

knotmirror/knotstream/slurper.go:331

I think this is counting nanoseconds instead of seconds. Intentional?

boltless.me

good catch. Seems like I brought a bug from indigo/relay. Sorry for that, should check more carefully.

boltless.me submitted #0
diff
1 commit
expand
d0d0e77e
knotmirror: introduce knotmirror
expand 0 comments