Add VarInt support

[fboemer]

Adds VarInt support

[karulont]

Did you try with changing the intermediate type from UInt64 to T in the decoding?

[fboemer]

Did you try with changing the intermediate type from UInt64 to T in the decoding?

We'd need some extra logic to check for overflow shifting & T == UInt8 (typeSpecificOverflow was failing otherwerise)
This would work, but it's more complicated.
So I think it's better to leave as is.

    /// Decode a varint from a byte array
    /// Returns (decoded value, number of bytes consumed)
    static func decode<T: FixedWidthInteger & UnsignedInteger>(_ bytes: some Collection<UInt8>) throws
        -> (decoded: T, bytesConsumed: Int)
    {
        precondition(T.bitWidth >= 7)
        var result: T = 0
        var shift = 0
        var index = 0

        while index < bytes.count {
            let byte = bytes[bytes.index(bytes.startIndex, offsetBy: index)]
            index += 1

            // Check for overflow (varint can be at most 10 bytes for 64-bit)
            if shift >= T.bitWidth {
                throw VarIntError.overflow
            }

            // Check if adding these bits would overflow
            let bitsToAdd = T(byte & 0x7F)
            
            // Check if shifting would cause overflow
            if shift > 0 && (T.bitWidth - shift) < 7 {
                // If we're trying to shift beyond the bit width, check if higher bits are set
                if bitsToAdd >> (T.bitWidth - shift) > 0 {
                    throw VarIntError.overflow
                }
            }
            
            // Add the 7 data bits
            let shiftedBits = bitsToAdd << shift
            
            // Check for overflow when adding to result
            if T.max - result < shiftedBits {
                throw VarIntError.overflow
            }
            
            result |= shiftedBits

            // If continuation bit is not set, we're done
            if (byte & 0x80) == 0 {
                return (result, index)
            }

            shift += 7
        }

        // Ran out of bytes while continuation bit was set
        throw VarIntError.truncated
    }

[karulont]

Did you try with changing the intermediate type from UInt64 to T in the decoding?

We'd need some extra logic to check for overflow shifting & T == UInt8 (typeSpecificOverflow was failing otherwerise) This would work, but it's more complicated. So I think it's better to leave as is.

Yeah, fine to leave at is. Just a note: that it will not work for types larger than UInt64.

[fboemer]

Did you try with changing the intermediate type from UInt64 to T in the decoding?

We'd need some extra logic to check for overflow shifting & T == UInt8 (typeSpecificOverflow was failing otherwerise) This would work, but it's more complicated. So I think it's better to leave as is.

Yeah, fine to leave at is. Just a note: that it will not work for types larger than UInt64.

Good point, I'll add a precondition check for this