Right now the only way to read record content is per-record (RepoHandle::get_record/list_records), which does a non-sequential block lookup for each. To look at record contents across an entire collection (in my case, for analyzing things like word frequency and post activity), it takes forever (especially on large databases!) because of the random block lookups.
Instead, we can iterate the blocks keyspace directly, for cases where you just want the record content (and don't care about the associated DID/rkey). I've been using a patch that adds a method for this:
pub fn scan_collection_blocks(
&self,
collection: &str,
) -> impl Iterator<Item = Result<fjall::Slice>> + '_ {
let mut start = collection.as_bytes().to_vec();
start.push(crate::db::keys::SEP);
let mut end = start.clone();
if let Some(last) = end.last_mut() {
*last = last.saturating_add(1);
}
let bounds = (
std::ops::Bound::Included(start),
std::ops::Bound::Excluded(end),
);
self.state
.db
.blocks
.range::<Vec<u8>, _>(bounds)
.map(|item| item.into_inner().into_diagnostic().map(|(_key, value)| value))
}
For my patch this was just implemented on Hydrant, and it just returns the raw CBOR slice because I have been decoding into a specific record data shape myself. Also, I think a recent refactor made blocks private outside the db module so this would have to be adapted for that.
can add this yeah feel free to make a pr. i dont think the method should drop the key though and ideally want the key to be returned wrapped with a struct so we can do lazy parsing on it. value can stay as cbor binary