tangled.org / core
Monorepo for Tangled tangled.org
fork atom

knotserver,knotmirror/xrpc: sync.requestCrawl support #1202

merged opened by boltless.me targeting master from sl/rvyxvsnxkkql

knotmirror ingests repo creation event from atproto relay through tap, not from knotstream. Therefore, knotserver will request sync.requestCrawl with optional new repo information to notify the repo creation event.

Knot will call sync.requestCrawl on following cases:

  • on startup
  • when /event stream has failed
  • on repo creation (mandatory)

Signed-off-by: Seongmin Lee git@boltless.me

STACK 3
#1202 knotserver,knotmirror/xrpc: sync.requestCrawl support
1/3
4
#6
#1201 lexicons: sync.requestCrawl and knot.subscribeRepos
1/3
0
#6
#1200 nix,knotmirror: listen & sync local knots from local knotmirror
1/3
0
#6
ABANDONED PULLS 13
#1206 appview,knotserver/xrpc: create repo on knot-side first
3/3
3
#1
#1205 lexicons: add required name field to repo.create
1/3
0
#1
#1203 nix,knotmirror: sync localhost knots
0
#1
#1199 appview: remove oauth.User type
3/3
0
#0
#1198 appview/pulls: remove dependencies to oauth.MultiAccountUser
2/3
0
#0
#1197 appview/pages: don't access MultiAccountUser.Active from template
2/3
0
#0
#1196 appview/oauth: remove .Handle from oauth.AccountInfo
2/3
0
#0
#1195 appview/oauth: remove AuthReturnInfo
2/3
0
#0
#1194 appview/{oauth,pages,state}: remove unused AddAccount session data
2/3
0
#0
#1193 appview/{pages,state}: simplify LoginPage params
2/3
0
#0
#1192 appview/{oauth,pages}: cleanup unused codes
2/3
0
#0
#1191 appview: don't trust oauth.ClientSessionData
2/3
0
#0
#1190 appview: use explicit TANGLED_PDS_USER_DOMAIN for user handle check
2/3
0
#0
Labels

None yet.

assignee

None yet.

Participants 2
AT URI
at://did:plc:xasnlahkri4ewmbuzly2rlc5/sh.tangled.repo.pull/3mhk6tzq7xq22
+22 -1
Interdiff #1 → #2
knotmirror/hostutil/hostutil.go

This file has not been changed.

knotmirror/knotmirror.go

This file has not been changed.

knotmirror/resyncer.go

This file has not been changed.

+10 -1
knotmirror/xrpc/sync_requestCrawl.go
··· 61 61 } 62 62 record := out.Value.Val.(*tangled.Repo) 63 63 64 + knotUrl := record.Knot 65 + if !strings.Contains(record.Knot, "://") { 66 + if noSSL { 67 + knotUrl = "http://" + knotUrl 68 + } else { 69 + knotUrl = "https://" + knotUrl 70 + } 71 + } 72 + 64 73 repo := &models.Repo{ 65 74 Did: owner.DID, 66 75 Rkey: repoAt.RecordKey(), 67 76 Cid: (*syntax.CID)(out.Cid), 68 77 Name: record.Name, 69 - KnotDomain: record.Knot, 78 + KnotDomain: knotUrl, 70 79 State: models.RepoStatePending, 71 80 ErrorMsg: "", 72 81 RetryAfter: 0,
knotmirror/xrpc/xrpc.go

This file has not been changed.

knotserver/config/config.go

This file has not been changed.

knotserver/events.go

This file has not been changed.

knotserver/server.go

This file has not been changed.

knotserver/xrpc/create_repo.go

This file has not been changed.

+9
nix/modules/knot.nix
··· 115 115 ''; 116 116 }; 117 117 118 + knotmirrors = mkOption { 119 + type = types.listOf types.str; 120 + default = [ 121 + "https://mirror.tangled.network" 122 + ]; 123 + description = "List of knotmirror hosts to request crawl"; 124 + }; 125 + 118 126 server = { 119 127 listenAddr = mkOption { 120 128 type = types.str; ··· 263 271 "KNOT_SERVER_PLC_URL=${cfg.server.plcUrl}" 264 272 "KNOT_SERVER_JETSTREAM_ENDPOINT=${cfg.server.jetstreamEndpoint}" 265 273 "KNOT_SERVER_OWNER=${cfg.server.owner}" 274 + "KNOT_MIRRORS=${concatStringsSep "," cfg.knotmirrors}" 266 275 "KNOT_SERVER_LOG_DIDS=${ 267 276 if cfg.server.logDids 268 277 then "true"
+3
nix/vm.nix
··· 111 111 jetstreamEndpoint = jetstream; 112 112 listenAddr = "0.0.0.0:6444"; 113 113 }; 114 + knotmirrors = [ 115 + "http://localhost:7000" 116 + ]; 114 117 }; 115 118 services.tangled.spindle = { 116 119 enable = true;

History

7 rounds 4 comments
sign up or login to add to the discussion
boltless.me submitted #6
diff interdiff
1 commit
expand
bea612c9
nix,knotserver,knotmirror/xrpc: sync.requestCrawl support
2/3 timeout, 1/3 success
expand
build.yml
fmt.yml
test.yml
expand 1 comment
pull request successfully merged
boltless.me submitted #5
diff interdiff
1 commit
expand
c5293014
nix,knotserver,knotmirror/xrpc: sync.requestCrawl support
2/3 timeout, 1/3 success
expand
build.yml
fmt.yml
test.yml
expand 3 comments
oyster.cafe

knotmirror/hostutil/hostutil.go:11

nit: hostname

oyster.cafe

knotmirror/xrpc/sync_requestCrawl.go:22 this is unauthed right? so if this code goes through, I could curl -X POST /xrpc/sh.tangled.sync.requestCrawl -d "garbage" and crash knotmirror?

boltless.me

Thank you for checking. I should make a git hook to check unimplemented codes before push at this point...

boltless.me submitted #4
diff interdiff
1 commit
expand
c5293014
nix,knotserver,knotmirror/xrpc: sync.requestCrawl support
2/3 timeout, 1/3 success
expand
build.yml
fmt.yml
test.yml
expand 0 comments
boltless.me submitted #3
diff interdiff
1 commit
expand
e4ada809
nix,knotserver,knotmirror/xrpc: sync.requestCrawl support
expand 0 comments
boltless.me submitted #2
diff
1 commit
expand
16d4b9e6
knotserver,knotmirror/xrpc: sync.requestCrawl support
expand 0 comments
boltless.me submitted #1
diff interdiff
1 commit
expand
577fe4a5
knotserver,knotmirror/xrpc: sync.requestCrawl support
expand 0 comments
boltless.me submitted #0
diff
1 commit
expand
623108c4
knotserver,knotmirror/xrpc: sync.requestCrawl support
expand 0 comments