seems like it's good on the write side

unclear if there's a win on reads

you're not the fastest but you are the most well-behaved.

it's fast but it's being allowed to use a lot of mem

it works? it's fast?

eh

redb blows up the disk a bit unsurprisingly

fast but this might be buffering in mem? need to verify.

just slightly slower than redb atm

small (~3%) perf hit for this, which is fine

righttttttttttt

pretty much no perf impact still

yay

this was not that bad

and a little refactor

idk why it was an independent function in iroh_car which this was loosely based off of

yay

the walker knows which block it needs to be able
to continue, so we pass that to the driver, and
have it load until that block is reached before
trying to walk again.

small apparent slowdown, lots of cleanup to do

don't interleave block loading and mst walking -- with current CARs there's no actual early blocks we can use, and it's slowwwwwww as it was written

hack in a "processing" hard-coding (save the record length): this does reduce memory (~half on huge repo) and speed things up (~2x on huge repo). obviously this is an extreme case of processing, most stuff will benefit less

file reader example: use buffering on the reader

all in this takes the huge repo from ~17s -> 0.45s on my dev machine.

and the code is so much worse

but, later problems.

it was trying to deserialize them as an array instead of as its normal bytes type