tangled.org / core
Monorepo for Tangled tangled.org
fork atom

knotserver,knotmirror/xrpc: sync.requestCrawl support #1202

merged opened by boltless.me targeting master from sl/rvyxvsnxkkql

knotmirror ingests repo creation event from atproto relay through tap, not from knotstream. Therefore, knotserver will request sync.requestCrawl with optional new repo information to notify the repo creation event.

Knot will call sync.requestCrawl on following cases:

  • on startup
  • when /event stream has failed
  • on repo creation (mandatory)

Signed-off-by: Seongmin Lee git@boltless.me

STACK 3
#1202 knotserver,knotmirror/xrpc: sync.requestCrawl support
1/3
4
#6
#1201 lexicons: sync.requestCrawl and knot.subscribeRepos
1/3
0
#6
#1200 nix,knotmirror: listen & sync local knots from local knotmirror
1/3
0
#6
ABANDONED PULLS 13
#1206 appview,knotserver/xrpc: create repo on knot-side first
3/3
3
#1
#1205 lexicons: add required name field to repo.create
1/3
0
#1
#1203 nix,knotmirror: sync localhost knots
0
#1
#1199 appview: remove oauth.User type
3/3
0
#0
#1198 appview/pulls: remove dependencies to oauth.MultiAccountUser
2/3
0
#0
#1197 appview/pages: don't access MultiAccountUser.Active from template
2/3
0
#0
#1196 appview/oauth: remove .Handle from oauth.AccountInfo
2/3
0
#0
#1195 appview/oauth: remove AuthReturnInfo
2/3
0
#0
#1194 appview/{oauth,pages,state}: remove unused AddAccount session data
2/3
0
#0
#1193 appview/{pages,state}: simplify LoginPage params
2/3
0
#0
#1192 appview/{oauth,pages}: cleanup unused codes
2/3
0
#0
#1191 appview: don't trust oauth.ClientSessionData
2/3
0
#0
#1190 appview: use explicit TANGLED_PDS_USER_DOMAIN for user handle check
2/3
0
#0
Labels

None yet.

assignee

None yet.

Participants 2
AT URI
at://did:plc:xasnlahkri4ewmbuzly2rlc5/sh.tangled.repo.pull/3mhk6tzq7xq22
+13 -8
Interdiff #2 → #3
knotmirror/hostutil/hostutil.go

This file has not been changed.

knotmirror/knotmirror.go

This file has not been changed.

knotmirror/resyncer.go

This file has not been changed.

knotmirror/xrpc/sync_requestCrawl.go

This file has not been changed.

knotmirror/xrpc/xrpc.go

This file has not been changed.

knotserver/config/config.go

This file has not been changed.

knotserver/events.go

This file has not been changed.

knotserver/server.go

This file has not been changed.

+12 -8
knotserver/xrpc/create_repo.go
··· 122 122 repoPath, 123 123 ) 124 124 125 - // request crawl for this repository 126 - repoAt := fmt.Sprintf("at://%s/%s/%s", actorDid, tangled.RepoNSID, rkey) 125 + // HACK: request crawl for this repository 126 + // Users won't want to sync entire network from their local knotmirror. 127 + // Therefore, to bypass the local tap, requestCrawl directly to the knotmirror. 127 128 go func() { 128 - rCtx, rCancel := context.WithTimeout(context.Background(), 10*time.Second) 129 - defer rCancel() 130 - h.requestCrawl(rCtx, &tangled.SyncRequestCrawl_Input{ 131 - Hostname: h.Config.Server.Hostname, 132 - EnsureRepo: &repoAt, 133 - }) 129 + if h.Config.Server.Dev { 130 + repoAt := fmt.Sprintf("at://%s/%s/%s", actorDid, tangled.RepoNSID, rkey) 131 + rCtx, rCancel := context.WithTimeout(context.Background(), 10*time.Second) 132 + defer rCancel() 133 + h.requestCrawl(rCtx, &tangled.SyncRequestCrawl_Input{ 134 + Hostname: h.Config.Server.Hostname, 135 + EnsureRepo: &repoAt, 136 + }) 137 + } 134 138 }() 135 139 136 140 w.WriteHeader(http.StatusOK)
nix/modules/knot.nix

This file has not been changed.

+1
nix/vm.nix
··· 110 110 plcUrl = plcUrl; 111 111 jetstreamEndpoint = jetstream; 112 112 listenAddr = "0.0.0.0:6444"; 113 + dev = true; 113 114 }; 114 115 knotmirrors = [ 115 116 "http://localhost:7000"

History

7 rounds 4 comments
sign up or login to add to the discussion
boltless.me submitted #6
diff interdiff
1 commit
expand
bea612c9
nix,knotserver,knotmirror/xrpc: sync.requestCrawl support
2/3 timeout, 1/3 success
expand
build.yml
fmt.yml
test.yml
expand 1 comment
pull request successfully merged
boltless.me submitted #5
diff interdiff
1 commit
expand
c5293014
nix,knotserver,knotmirror/xrpc: sync.requestCrawl support
2/3 timeout, 1/3 success
expand
build.yml
fmt.yml
test.yml
expand 3 comments
oyster.cafe

knotmirror/hostutil/hostutil.go:11

nit: hostname

oyster.cafe

knotmirror/xrpc/sync_requestCrawl.go:22 this is unauthed right? so if this code goes through, I could curl -X POST /xrpc/sh.tangled.sync.requestCrawl -d "garbage" and crash knotmirror?

boltless.me

Thank you for checking. I should make a git hook to check unimplemented codes before push at this point...

boltless.me submitted #4
diff interdiff
1 commit
expand
c5293014
nix,knotserver,knotmirror/xrpc: sync.requestCrawl support
2/3 timeout, 1/3 success
expand
build.yml
fmt.yml
test.yml
expand 0 comments
boltless.me submitted #3
diff
1 commit
expand
e4ada809
nix,knotserver,knotmirror/xrpc: sync.requestCrawl support
expand 0 comments
boltless.me submitted #2
diff interdiff
1 commit
expand
16d4b9e6
knotserver,knotmirror/xrpc: sync.requestCrawl support
expand 0 comments
boltless.me submitted #1
diff interdiff
1 commit
expand
577fe4a5
knotserver,knotmirror/xrpc: sync.requestCrawl support
expand 0 comments
boltless.me submitted #0
diff
1 commit
expand
623108c4
knotserver,knotmirror/xrpc: sync.requestCrawl support
expand 0 comments