Implement streaming lz4 compession #1611

zhwufd · 2021-11-20T18:00:50Z

Compress method Compress size(B) Compress time(us) Decompress time(us) Compress throughput(MB/s) Decompress throughput(MB/s) Compress ratio
Snappy 128 0.630420 0.676840 193.633312 180.353278 37.500000%
Gzip 128 4.886340 0.804860 24.981952 151.666517 47.656250%
Zlib 128 4.376520 0.665340 27.892095 183.470575 38.281250%
Lz4 128 0.614140 0.359200 198.766263 339.839400 58.115935%

          Snappy                1024            0.690060            0.697920                   1415.184911                   1399.247048                     8.789062%
            Gzip                1024            7.144860            1.752040                    136.680425                    557.385962                     6.640625%
            Zlib                1024            6.703100            1.304000                    145.688189                    748.897623                     5.468750%
             Lz4                1024            0.781000            0.520440                   1250.400128                   1876.417070                     7.810039%

          Snappy               16384            4.155420            4.253955                   3760.149466                   3673.052421                     4.962158%
            Gzip               16384           55.165820           19.621680                    283.236974                    796.313070                     0.781250%
            Zlib               16384           48.379395           13.014209                    322.968077                   1200.610811                     0.708008%
             Lz4               16384            3.494336            3.105762                   4471.521994                   5030.971921                     1.227403%

          Snappy               32768            7.633301            8.515039                   4093.903921                   3669.977292                     4.824829%
            Gzip               32768          105.522363           37.225879                    296.145756                    839.469770                     0.537109%
            Zlib               32768           92.468164           24.807813                    337.954152                   1259.683819                     0.500488%
             Lz4               32768            6.346191            4.902539                   4924.213280                   6374.248038                     1.037690%

          Snappy              524288          121.535938          143.765625                   4114.009488                   3477.882839                     4.747772%
            Gzip              524288         2128.425000          564.970312                    234.915489                    885.002254                     0.305939%
            Zlib              524288         1917.210938          367.104688                    260.795508                   1362.009304                     0.303650%
             Lz4              524288          105.067188           66.967187                   4758.859658                   7466.343125                     0.857538%

LZ4 compress/decompress throughput outperform snappy, gzip and zlib.

zhwufd · 2021-11-25T04:16:02Z

本地Makefile编译没问题，travis-ci编译不过。

chenzhangyi · 2021-11-25T04:26:14Z

src/butil/third_party/lz4/lz4.c

@@ -0,0 +1,2495 @@
+/*


需要更新下LICENSE

LICENSE更新完成了。

这个文件需要加上额外的namespace，不然如果应用方也使用了lz4的话，会造成链接冲突

chenzhangyi · 2021-11-25T04:27:04Z

src/brpc/policy/lz4_compress.cpp

+    size_t ref_cnt = in.backing_block_num(); 
+    LZ4_stream_t* lz4_stream = LZ4_createStream();
+    butil::IOBuf block_buf;
+    std::vector<size_t> block_metas;


不要用size_t, 这个跨架构大小不是确定的。这里得用int32或者int16.

chenzhangyi · 2021-11-25T04:28:42Z

src/brpc/policy/lz4_compress.cpp

+        block_metas.emplace_back(src_block_size);
+    }
+    size_t nblocks = block_metas.size() / 2;
+    out->append(&nblocks, sizeof(size_t));


不能直接这么写，需要转换成网络序。另外同上，这里不能用size_t

chenzhangyi · 2021-11-25T04:30:16Z

src/brpc/policy/lz4_compress.cpp

+        return false;
+    }
+    std::vector<size_t> block_metas(nblocks * 2, 0);
+    buf_iter.copy_and_forward(block_metas.data(), nblocks * 2 * sizeof(size_t));


这些都得实现为序列化，这里如果考虑压缩的话，可以用varint encoding(protobuf应该有类似的接口)

chenzhangyi · 2021-11-25T04:30:42Z

src/brpc/policy/lz4_compress.cpp

+        return false;
+    }
+    LZ4_streamDecode_t* lz4_stream_decode = LZ4_createStreamDecode();
+    char* in_scratch = new char[max_block];


用DEFNIE_SMALL_ARRAY，大部分情况下应该不会很大.

chenzhangyi · 2021-11-25T04:31:54Z

src/brpc/policy/lz4_compress.cpp

+            LZ4_freeStreamDecode(lz4_stream_decode);
+            return false;
+        }
+        out->append_user_data(out_buf, dst_block_size, [](void *d) {


iobuf as zero copy stream这里无法使用么, 必须依赖user_data?

chenzhangyi · 2021-11-25T04:33:00Z

src/brpc/global.cpp

@@ -386,6 +387,11 @@ static void GlobalInitializeOrDieImpl() {
    if (RegisterCompressHandler(COMPRESS_TYPE_SNAPPY, snappy_compress) != 0) {
        exit(1);
    }
+    const CompressHandler lz4_compress =
+        { Lz4Compress, Lz4Decompress, "lz4" };


lz4有 frame, block, stream不同模式(记忆中block似乎是压缩比最高的). 这里要么就改成lz4s. 而不是直接用lz4.

chenzhangyi · 2021-11-25T04:37:18Z

src/brpc/policy/lz4_compress.cpp

+    butil::IOBufAsZeroCopyOutputStream wrapper(&serialized_pb);
+    if (res.SerializeToZeroCopyStream(&wrapper)) {
+        return Lz4Compress(serialized_pb, buf);
+    }


建议写一下文档先讨论下对应压缩的wire format, 这里似乎并不是效率最高的。结合zero copy stream，这里应该能做到一边序列化一边压缩(至少传输格式上需要能保留这种实现可能)。现在还是多构造了一次中间数据。

ehds · 2023-11-15T08:42:16Z

这个PR有考虑继续跟进吗？

zhwufd · 2023-11-16T00:09:55Z

这个PR有考虑继续跟进吗？

我这周再优化下。

wanghenshui · 2024-07-19T03:39:04Z

这个PR有考虑继续跟进吗？

zhwufd added 4 commits November 20, 2021 00:15

Add lz4.h and lz4.c

8902b77

Add lz4 compress

d5fac35

Add unit test

46acbad

Fix memory leak

f76aed4

zhwufd changed the title ~~Add streaming lz4 compession~~ Implement streaming lz4 compession Nov 22, 2021

zhwufd closed this Nov 22, 2021

zhwufd reopened this Nov 22, 2021

zhwufd added 2 commits November 22, 2021 12:47

Fix bazel compile issue

f1cea3d

Fix make compile issue

5a0720d

zhwufd closed this Nov 25, 2021

zhwufd reopened this Nov 25, 2021

chenzhangyi requested changes Nov 25, 2021

View reviewed changes

zhwufd added 3 commits November 27, 2021 11:57

Fix license issue for lz4

16c48e1

Change lz4 to lz4s, change size_t to int32_t

7003321

Add lz4hc and lz4frame

660c0ac

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement streaming lz4 compession #1611

Implement streaming lz4 compession #1611

zhwufd commented Nov 20, 2021

zhwufd commented Nov 25, 2021

chenzhangyi Nov 25, 2021

zhwufd Nov 27, 2021

chenzhangyi Nov 28, 2021

chenzhangyi Nov 25, 2021

chenzhangyi Nov 25, 2021

chenzhangyi Nov 25, 2021

chenzhangyi Nov 25, 2021

chenzhangyi Nov 25, 2021

chenzhangyi Nov 25, 2021

chenzhangyi Nov 25, 2021

ehds commented Nov 15, 2023

zhwufd commented Nov 16, 2023

wanghenshui commented Jul 19, 2024

Implement streaming lz4 compession #1611

Are you sure you want to change the base?

Implement streaming lz4 compession #1611

Conversation

zhwufd commented Nov 20, 2021

zhwufd commented Nov 25, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ehds commented Nov 15, 2023

zhwufd commented Nov 16, 2023

wanghenshui commented Jul 19, 2024