spindled/engine: store workflow logs in s3 #3468

Closed

jobala.tngl.sh 7mo ago

targeting master from jobala.tngl.sh/tangled:s3-logs

Closed opened by and assigned to

jobala.tngl.sh 7mo ago targeting master from jobala.tngl.sh/tangled:s3-logs

Overview #

This pr implements the first part of this proposal and introduces two new environment variables.

UploadWorkflowLogs, is set to true then the workflow's logs will be uploaded to s3
LogsBucket, specifies the logs bucket Stores workflow logs in s3

Commits 2

spindle/engine: move s3 implementation

+35 -25

spindle/engine: store workflow logs in s3

+188 -37

Comments

oppi.li 7mo ago

nice! couple of comments from my end (note that i have not yet tested this out myself locally):

spindle/config/config.go:42-43: i think the config option should be its own struct, and not in NixeryPipelines. we can have type S3 struct { .. } with just LogBucket as the only option. if LogBucket is the empty string, we can consider s3 to be disabled, and if not, we can consider it to be enabled.

spindle/engines/nixery/engine.go:290: spindle is structured to have multiple engines, of which nixery is one engine (and the only one presently). uploading logs here would mean future engines will have to duplicate this step. this logic is better implemented in engine/engine.go
spindle/utils/s3.go:1: i personally try to avoid a utils package. this could live at the top level IMO: spindle/s3.go.

jobala.tngl.sh 7mo ago

uploading logs here would mean future engines will have to duplicate this step. this logic is better implemented in engine/engine.go

Do you mean I should have something like this in nixery instead? Note, code below is pseudocode.

if uploadLogs {
engine.UploadLogs()
}

jobala.tngl.sh 7mo ago

nvm, I see what you mean.

v0 7mo ago

Overview #

Versions