UTXO TTL disk/network encoding

[dergoegge]

We currently use 4 byte integers for storing the UTXO time-to-live values on disk/network.
Should we use VARINTs?
~60% of UTXOS live less than 255 blocks so one byte would be enough. This would reduce bandwidth but add a small serialization overhead.

[kcalvinalvin]

In my btcd implementation, I'm just going with byteOrder.PutUint32() for the moment. It's a fairly simple code change though so maybe I'll benchmark some stuff to see if there's a meaningful enough overhead that will force us not to change.

For future reference, to implement VARINT serialization the code can be found in bridgenode/compress.go. This is a copy with slight modifications since the relevant functions putVLQ() deserializeVLQ() and serializeSizeVLQ() are unexported in btcd.

[adiabat]

In some cases we can get some efficiency by going to varints, but probably not for the flat file TTL values.

The way the TTL values are written, we allocate space for the TTL before we know what it is. We allocate 4 empty bytes of space for every output in the block. Then when a UTXO is spent in a later block, we seek back to that location and write the TTL value.

If we wanted to use varints there, we'd have to either expand or contract the whole file each time we wrote a TTL that wasn't the same size as our initial estimate. With the current flat file implementation this would not be practical as it would mean rewriting many megabytes thousands of times per block. We could batch them but it'd still be a lot of work.

Also TTL values are only a few kilobytes per block.

For network transmission though, it could make sense to use varints.

Or we can just use 3 bytes everywhere since it will take hundreds of years to get to 16 million blocks.

[dergoegge]

Ah right, i did not think this through.
I guess 4 bytes is ok for now and if we really want to save bandwidth we can switch to 3.

For network transmission though, it could make sense to use varints.

Maybe allthough you could not serve directly from the flatfile, which might be slow.