Crisp Byte

What Does SDBD Actually Look Like?

To turn the semantics into a real file format, we need to define what they look like as actual data. How do you write an SDBD to a stream or a disk? How do we turn our HTTP-like semantics into a real document? That question almost answers itself. I'll keep stealing from HTTP.

Consider the options #

There are three versions of HTTP that each represent headers in a different format. HTTP/1.1 uses a human-readable plain text format. It's basically the format you'll see in browser development tools and blog posts about HTTP headers. HTTP/2 uses a compressible binary format called HPACK. HTTP/3 uses a variant of HPACK called QPACK that's designed to be less trouble for the underlying QUIC protocol.

I'm going to decide that SDBD will be a binary format. No part of it needs to be plain text. I'm going to choose HPACK over QPACK. Mostly that's because I think it should be easier to implement the proof of concept with HPACK due to available tools.

Represent #

It's time to put it together. I already have plans for updating the format, so the very first byte of SDBD will be a version number. We'll start simply with 0x01. The rest of this section will describe a version 1 document.

Next comes the metadata. We need to know how long the metadata is, so we'll use two bytes to store an unsigned int to tell us the length of the metadata section in bytes. (But what happens if the metadata is more than 64KB? We'll worry about that later.) The metadata headers will be encoded with HPACK.

After the metadata comes the data, which better be the exact length defined by content-length, or bad things will happen.

That's it. If we encode the original example with this format, this is the result:

0000	01 3a 00 00 89 21 ea 49  6a 4a d5 0e 92 ff 8d 5f   .:...!.IjJ....._
0010	ff f8 20 74 d7 41 57 4f  94 af 1d 9f 0f 0b 02 62   .. t.AWO.......b
0020	72 0f 10 94 49 7c a5 8a  e8 19 aa fb 50 93 8e c4   r...I|......P...
0030	15 30 5a 85 86 82 18 df  0f 0d 02 39 35 a1 20 04   .0Z........95. .
0040	00 20 50 6e eb 2b e9 96  6c 96 ac 25 3e a8 69 22   . Pn.+..l..%>.i"
0050	88 0b ea c9 42 ee de f8  24 4d b3 bd 08 d5 80 db   ....B...$M......
0060	f0 b9 02 31 a3 be 91 b7  65 bd 28 82 7f 92 1c 88   ...1....e.(.....
0070	f2 38 1d eb 56 54 b9 93  d3 59 62 6a 7f 65 30 f4   .8..VT...Ybj.e0.
0080	dd 5c 62 80 01 74 42 12  6e 34 a8 1c e5 9d f4 91   .\b..tB.n4......
0090	3c 34 83 10 00 1a 70 2e  bb f5 aa 01               <4....p.....

It is finally time to get to the good part. Let's write some code.