From a66a09e1cc4e36f8bbc34f486e623d78dec71017 Mon Sep 17 00:00:00 2001
From: Robin Berjon
- BDASL extends DASL CIDs with a new hash type that works better
- for large files but isn't available by default in browsers, and
- therefore not an appropriate option in most situations.
-
- BDASL extends DASL CIDs by adding BLAKE3 support ([[blake3]]).
- BLAKE3 is a powerful hashing framework that works well for
- progressive verification of large streams. Unfortunately, it
- isn't available in browser (and neither is streaming hashing in
- general) which makes it inappriopriate for inclusion as the
- primary hash function in DASL CIDs.
-
- It is recommended to avoid using BDASL CIDs in arbitrary open
- environments, and rather to focus on using such CIDs in specific
- cases in which participants are likely to know how to handle
- them.
-
- All the parsing works the same as for DASL CIDs ([[cid]]) with
- one modification.
-
- In the steps to
- decode a CID, the
- hash type may also be equal to
-
- Section 6.4 of the [[blake3]] spec outlines
- Streaming Verification, a process that allows the sender
- and receiver of a CID to incrementally verify data as it is
- being transferred. Applying this technique to CIDs is the core
- benefit of BDASL, which is well-suited to both fetching byte
- ranges within a CID, and verifying data where the cost of
- faulty transmission will impact the performance of an application.
- Verified streaming incurs minimal overhead on payloads of all
- sizes, and scales linearly as a small percentage of the size of
- the CID being verified.
-
- Streaming verification rounds to the nearest kilobyte for
- verification. For more details & a reference implementation, see
- [[iroh-blobs]].
-
- [[rfc9110]] Defines HTTP Range requests for fetching a single
- contiguous set of bytes from a larger source held by a server.
- Range requests use a Range header:
-
- In this context the HTTP server is acting as a trusted gateway,
- performing verificaiton on behalf of the user. The advantage of
- delegating trust here is interop with standard HTTP semantics.
- For trustless fetching, use verified streaming directly.
-
- For a given normalized byte range
-
- Start Chunk:
- Introduction
- Parsing BDASL CIDs
- 0x1e (BLAKE3) ([[blake3]]).
- Streaming Verification
- Verifying HTTP Range Requests
- Range: bytes={start}-{end}
- Both start and end values are optional, and when missing indicate
- the first and late byte, respectively.
- Fufilling requests
- (start: Option<u64>, end: Option<u64>),
- that byte range is mapped to a chunk range, which is the
- set of chunks that fully contains the set of bytes in the
- HTTP range request.
- the corresponding chunk range is calculated as:
-
-
- start is a number s, use ⌈s / 1024⌉ (ceiling division)start is empty, there is no lower bound, so begin the request from byte 0.
- End Chunk: -
stop is a number s, use ⌈s / 1024⌉ (ceiling division)stop is empty, no upper bound, so request to the end of the byte array- From here, construct a verified range request in accordance with - the verified streaming protocol, as chunks arrive, check if the - chunk responded intersects with either start or end chunks. -
- For a request of Range: bytes=512-1535
- two chunks will be retrieved, chunk 0 and chunk 1.
-
-
[512, 1024)chunk_data[512..1024] (512 bytes)[1024, 1536)chunk_data[0..512] (512 bytes)+ BDASL extends DASL CIDs with a new hash type that works better + for large files but isn't available by default in browsers, and + therefore not an appropriate option in most situations. +
++ BDASL extends DASL CIDs by adding BLAKE3 support ([[blake3]]). + BLAKE3 is a powerful hashing framework that works well for + progressive verification of large streams. Unfortunately, it + isn't available in browser (and neither is streaming hashing in + general) which makes it inappriopriate for inclusion as the + primary hash function in DASL CIDs. +
++ It is recommended to avoid using BDASL CIDs in arbitrary open + environments, and rather to focus on using such CIDs in specific + cases in which participants are likely to know how to handle + them. +
++ All the parsing works the same as for DASL CIDs ([[cid]]) with + one modification. +
+
+ In the steps to
+ decode a CID, the
+ hash type may also be equal to
+ 0x1e (BLAKE3) ([[blake3]]).
+
+ Section 6.4 of the [[blake3]] spec outlines + Streaming Verification, a process that allows the sender + and receiver of a CID to incrementally verify data as it is + being transferred. Applying this technique to CIDs is the core + benefit of BDASL, which is well-suited to both fetching byte + ranges within a CID, and verifying data where the cost of + faulty transmission will impact the performance of an application. + Verified streaming incurs minimal overhead on payloads of all + sizes, and scales linearly as a small percentage of the size of + the CID being verified. +
++ Streaming verification rounds to the nearest kilobyte for + verification. For more details & a reference implementation, see + [[iroh-blobs]]. +
+
+ [[rfc9110]] Defines HTTP Range requests for fetching a single
+ contiguous set of bytes from a larger source held by a server.
+ Range requests use a Range header:
+ Range: bytes={start}-{end}
+ Both start and end values are optional, and when missing indicate
+ the first and late byte, respectively.
+
+ In this context the HTTP server is acting as a trusted gateway, + performing verificaiton on behalf of the user. The advantage of + delegating trust here is interop with standard HTTP semantics. + For trustless fetching, use verified streaming directly. +
+
+ For a given normalized byte range
+ (start: Option<u64>, end: Option<u64>),
+ that byte range is mapped to a chunk range, which is the
+ set of chunks that fully contains the set of bytes in the
+ HTTP range request.
+ the corresponding chunk range is calculated as:
+
+ Start Chunk: +
start is a number s, use ⌈s / 1024⌉ (ceiling division)start is empty, there is no lower bound, so begin the request from byte 0.+ End Chunk: +
stop is a number s, use ⌈s / 1024⌉ (ceiling division)stop is empty, no upper bound, so request to the end of the byte array+ From here, construct a verified range request in accordance with + the verified streaming protocol, as chunks arrive, check if the + chunk responded intersects with either start or end chunks. +
+ For a request of Range: bytes=512-1535
+ two chunks will be retrieved, chunk 0 and chunk 1.
+
+
[512, 1024)chunk_data[512..1024] (512 bytes)[1024, 1536)chunk_data[0..512] (512 bytes)