[JUST FOR COMPARATION] All encryption changes #21
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Do not run actions on `push` to save resources. - Skip to run `build-macos-java-static` and `build-macos-java-static-universal` jobs because they will cause link errors when OpenSSL introduced and they are not the Pegasus use cases. We can resolve the issue later. - Fix the error flags used in `build-linux-encrypted_env-no_compression`.
apache/incubator-pegasus#1575 Cherry-pick from tikv@113b363 Summary: Introduce `KeyManagedEncryptedEnv` which wraps around `EncryptedEnv` but provides an `KeyManager` API to enable key management per file. Also implements `AESBlockCipher` with OpenSSL. Test Plan: not tested yet. will update. Signed-off-by: Yi Wu <[email protected]> Signed-off-by: tabokie <[email protected]>
apache/incubator-pegasus#1575 Cherry-pick from tikv@3d44a33 Summary: Instead of using openssl's raw `AES_encrypt` and `AES_decrypt` API, which is a low level call to encrypt or decrypt exact one block (16 bytes), we change to use the `EVP_*` API. The former is deprecated, and will use the default C implementation without AES-NI support. Also the EVP API is capable of handing CTR mode on its own. Test Plan: will add tests Signed-off-by: Yi Wu <[email protected]> --------- Signed-off-by: Yi Wu <[email protected]> Co-authored-by: yiwu-arbug <[email protected]>
apache/incubator-pegasus#1575 Cherry-pick from tikv@2360562 Summary: Fix NewRandomRWFile and ReuseWritableFile misuse of `GetFile()` and `NewFile()`. See inline comments. Test Plan: manual test with tikv Signed-off-by: Yi Wu <[email protected]> Co-authored-by: yiwu-arbug <[email protected]>
apache/incubator-pegasus#1575 Cherry-pick from tikv@93e89a5 fix bug: tikv/tikv#9115 Summary: we need to update encryption metadata via encryption::DataKeyManager, which cannot combine with the actual file operation into one atomic operation. In RenameFile, when the src_file has been removed, power is off, then we may lost the file info of src_file next restart. Signed-off-by: Xintao [[email protected]](mailto:[email protected]) Signed-off-by: Xintao <[email protected]> Co-authored-by: Xintao <[email protected]>
apache/incubator-pegasus#1575 Cherry-pick from tikv@bbd27cf used LinkFile instead of RenameFile api of key manager. But LinkFile needs check the dst file information, in RenameFile logic, we don't care about that. So just skip encryption for current file. Signed-off-by: Xintao [[email protected]](mailto:[email protected])
apache/incubator-pegasus#1575 Cherry-pick from tikv@1868d12 Signed-off-by: Xintao <[email protected]> Signed-off-by: tabokie <[email protected]>
apache/incubator-pegasus#1575 Cherry-pick from tikv@4cebfc1 * Add SM4-CTR encryption algorithm * Adjust block size for sm4 encryption * Add UT for SM4 encryption * Adjust macros indentation for sm4 * Fix format for adding sm4 Signed-off-by: Jarvis Zheng <[email protected]>
apache/incubator-pegasus#1575 Cherry-pick from tikv@9464766 In some env, user installed openssl by yum install, and the openssl software may compiled with OPENSSL_NO_SM4 flag, so although the version is >= 1.1.1, but we still could not use sm4 in that situation. Signed-off-by: Jarvis Zheng <[email protected]>
apache/incubator-pegasus#1575 Cherry-pick from tikv@acc624f * hook delete dir in encrypted env * add a comment Signed-off-by: tabokie <[email protected]> Co-authored-by: Xinye Tao <[email protected]>
apache/incubator-pegasus#1575 Cherry-pick from tikv@14f36f8 (without compaction related code) * fix renaming encrypted directory Signed-off-by: tabokie <[email protected]>
apache/incubator-pegasus#1575 After all encryption related patches been cherry-picked from [tikv](https://github.com/tikv/rocksdb/commits/6.29.tikv) and merged, now we will improve the encrytion, including: - Fix action job `build-linux-encrypted_env-no_compression-no_openssl` to build binaries without openssl and compression libs correctly. - Fix action job `build-linux-encrypted_env-openssl` to export the `ENCRYPTED_ENV` enviroment variable correctly. - Don not skip tests which are skipped by TiKV. - Refactor `AESCTRCipherStream` and `AESEncryptionProvider` to support manage file key by the file itself, according to the design docs in [Data at rest encryption](apache/incubator-pegasus#1575). - Remove all KeyManager related codes. - Replace KeyManager tests by AES encryption tests. - Refactor encryption/encryption_test.cc and add more tests. - Make it possible to construct AESEncryptionProvider object via `EncryptionProvider::CreateFromString()` by registering a factory in "encryption" library. It's possible to construct an object by URI: `AES`, `AES://test` or `AES:<instance_key>,<EncryptionMethod>`. - `ldb` tool support to parse `--fs_uri` flags as the URI mentioned above. - Add tests to create AESEncryptionProvider object in `CreateEncryptedEnvTest.CreateEncryptedFileSystem` - `db_bench` support to run benchmark with encryption enabled, by adding new flags for `db_bench`, they are `encryption_method` and `encryption_instance_key`. - Move code from the exported header directory (i.e. include/rocksdb/encryption.h) to rocksdb internal (i.e. encryption/encryption.h), do not expose them to users. - Code format. Review hint: #17 shows all the code changes from the base branch (i.e. `pegasus-kv:v8.3.2-pegasus`), you can review it together to make sure the request branch `acelyc111:pk_enc_new` doesn't have vice effect on the base. Manual test: ``` // Generate some data. ./db_bench --encryption_method=AES128CTR --encryption_instance_key=test_instance_key --num=10000 // Dump WAL OK ./tools/ldb --fs_uri="provider=AES; id=EncryptedFileSystem" dump_wal --walfile=/tmp/rocksdbtest-1000/dbbench/000004.log ./tools/ldb --fs_uri="provider=AES://test; id=EncryptedFileSystem" dump_wal --walfile=/tmp/rocksdbtest-1000/dbbench/000004.log ./tools/ldb --fs_uri="provider=AES:test_instance_key,AES128CTR; id=EncryptedFileSystem" dump_wal --walfile=/tmp/rocksdbtest-1000/dbbench/000004.log // Dump WAL failed. Pass bad provider parameters to --fs_uri, e.g. ./tools/ldb --fs_uri="provider=AES1:test_instance_key,1AES128CTR; id=EncryptedFileSystem" dump_wal --walfile=/tmp/rocksdbtest-1000/dbbench/000004.log ./tools/ldb --fs_uri="provider=AES:bad_test_instance_key,AES128CTR; id=EncryptedFileSystem" dump_wal --walfile=/tmp/rocksdbtest-1000/dbbench/000004.log ./tools/ldb --fs_uri="provider=AES:test_instance_key,AES192CTR; id=EncryptedFileSystem" dump_wal --walfile=/tmp/rocksdbtest-1000/dbbench/000004.log // The same to other ldb tools. ```
apache/incubator-pegasus#1575 1. Update the status badge to pegasus-kv/rocksdb's own site. 2. Also aim to check whether all tests could pass after cherry-picking encryption related patches to 8.5.3 branch.
Return a pointer rather than Status for GetEVPCipher() when SM4 not supported.
Before this patch, there is no validation of the encryption key length
and method, so if using an encryption method but with a not matched
length
encryption key, it would cause encrypt and decrypt not match bug.
This patch fixes this bug and adds some related modifacations,
including:
- Add validation of encryption key length and method
- Use `RegisterOptions` to initialize the `AESEncryptionProvider`
objects
instead of `AddFactory`
- Still keep `AddFactory` to create `AESEncryptionProvider` objects
- The `AddFactory` will only create `AESEncryptionProvider` objects
successfully
when the uri match `AES`
- Remove the unit tests added in previous patched, we can use the
enviroment
variables instead, say `TEST_FS_URI` and `ENCRYPTED_ENV`
- Remove the new add gflags in `db_bench` tool, we can use `--fs_uri`
instead
- Add more unit tests in `encryption/encryption_test.cc`
- Add or update related CI jobs `build-linux-encrypted_env-openssl*`
The functionality can be checked by the following script as well.
```
#!/usr/bin/env bash
set -ex
# 1. build
mkdir build && cd build
cmake -DWITH_LZ4=1 -DCMAKE_BUILD_TYPE=Debug -DWITH_TESTS=1 -DROCKSDB_BUILD_SHARED=0 -DWITH_OPENSSL=1 ..
uris=("provider={id=AES;hex_instance_key=0123456789ABCDEF0123456789ABCDEF;method=AES128CTR};id=EncryptedFileSystem"
"provider={id=AES;hex_instance_key=0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF;method=AES192CTR};id=EncryptedFileSystem"
"provider={id=AES;hex_instance_key=0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF;method=AES256CTR};id=EncryptedFileSystem"
"provider={id=AES;hex_instance_key=0123456789ABCDEF0123456789ABCDEF;method=SM4CTR};id=EncryptedFileSystem")
for uri in ${uris[*]}; do
echo "${uri}"
# Set env (select 1 of the 4)
export TEST_FS_URI=${uri}
# Run unit tests
./env_basic_test --gtest_filter=*CustomEnv*
./env_test --gtest_filter=CreateEnvTest.CreateEncryptedFileSystem
# Run benchmarks
./db_bench --fs_uri="${uri}" --benchmarks="fillseq,readrandom,readseq" --compression_type=lz4 --num=1000000
# Run ldb tools
./tools/ldb --fs_uri="${uri}" --db=/tmp/rocksdbtest-1000/dbbench/ put k v
./tools/ldb --fs_uri="${uri}" --db=/tmp/rocksdbtest-1000/dbbench/ get k
ls -l /tmp/rocksdbtest-1000/dbbench | grep "log" | awk '{print $NF}' | xargs -i ./tools/ldb --fs_uri="${uri}" dump_wal --walfile=/tmp/rocksdbtest-1000/dbbench/{} | head
./tools/ldb --fs_uri="${uri}" --db=/tmp/rocksdbtest-1000/dbbench/ scan --hex | head
./tools/ldb --fs_uri="${uri}" --db=/tmp/rocksdbtest-1000/dbbench/ dump --hex | head
./tools/ldb --fs_uri="${uri}" --db=/tmp/rocksdbtest-1000/dbbench/ manifest_dump | head
./tools/ldb --fs_uri="${uri}" --db=/tmp/rocksdbtest-1000/dbbench/ list_live_files_metadata | head
done
# Unset the env
unset TEST_FS_URI
```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.