repo-sync-2024-02-08T11:49:27+0800 (#125)

.bazelversion

@@ -1 +1 @@
-6.5.0
+6.5.0

.clang-format

@@ -17,3 +17,6 @@ IncludeCategories:
 SeparateDefinitionBlocks: Always
 EmptyLineBeforeAccessModifier: LogicalBlock
 InsertNewlineAtEOF: true
+PointerAlignment: Right
+ReferenceAlignment: Pointer
+QualifierAlignment: Left

README.md

@@ -33,6 +33,100 @@ Each algorithm includes a variety of different implementations, and some impleme
 hardware accelerators. For more details, please refer to
 the [Document](https://www.secretflow.org.cn/docs/heu/latest/zh-Hans/getting_started/algo_choice )
 
+
+### Layout
+
+```text
+.
+├── .circleci  # CI/CD configuration files for circleci
+├── .github    # Configuration files for GitHub
+├── docs       # HEU documentation in Sphinx format
+├── heu        # All the implementation code for HEU
+│   ├── algorithms     # Holds the implementations of all algorithms
+│   ├── experimental   # Contains some standalone experimental code
+│   ├── library        # Implements the application layer functionality of the HE Library
+│   │   ├── algorithms # Legacy algorithms not yet migrated to SPI (this directory will be deprecated after migration)
+│   │   ├── benchmark  # Contains benchmarking code
+│   │   ├── numpy      # Implements a set of Numpy-like interfaces
+│   │   └── phe        # PHE Dispatcher implementation (to be deprecated, with replaced by SPI)
+│   ├── pylib          # Python bindings
+│   └── spi            # Defines the HEU software and hardware access layer interface (SPI)
+│       ├── he         # Contains HE SPI and related Sketches
+│       └── poly       # Defines polynomial interfaces and related Sketches
+└── third_party        # Contains third-party libraries required for compilation; libraries will be automatically downloaded during build
+
+```
+
+HEU is currently transitioning from the old Dispatcher architecture to an SPI-based framework. The
+main modules and their code path mappings for both architectures are as follows:
+
+Dispatcher-based architecture:
+
+```text
+     Python APIs (Python binding)
+          PATH: heu/pylib
+                  │
+                  ├───────────────────┐
+                  │                   ▼
+                  │    Tensor Lib with Numpy-like API
+                  │        PATH: heu/library/numpy
+                  │                   │
+                  │     ┌─────────────┘
+                  │     │
+                  ▼     ▼
+     PHE Dispatcher & PHE C++ API
+        PATH: heu/library/phe
+                  │
+                  │
+                  ▼
+ Various PHE algorithm implementations
+     PATH: heu/library/algorithms
+```
+
+SPI-based architecture:
+
+```text
+    Python APIs (Python binding)
+         PATH: heu/pylib
+                 │
+                 ├───────────────────┐
+                 │                   ▼
+                 │    Tensor Lib with Numpy-like API
+                 │           PATH: heu/numpy
+                 │                   │
+                 │ ┌─────────────────┘
+                 │ │
+                 ▼ ▼
+         HE SPI (C++ APIs)
+         PATH: heu/spi/he
+                 │
+                 │
+                 ▼
+ Various HE algorithm implementations
+        PATH: heu/algorithms
+```
+
+For a more detailed introduction to SPI, please [click here](heu/spi/README.md)
+
+### 2024 Work Plan
+
+Architecture Transition Milestones:
+
+- [x] HE SPI: Designed a unified interface that supports all PHE/FHE algorithms.
+- [ ] Implementation of SPI Sketches. (in progress)
+- [ ] Migration of existing algorithms to SPI.
+- [ ] Automated testing framework for PHE/FHE algorithms.
+- [ ] Transition of Tensor Lib's underlying layer from Dispatcher to SPI.
+- [ ] Transition of PyLib's underlying layer from Dispatcher to SPI.
+
+FHE Milestones:
+
+- [ ] Integration of Microsoft SEAL
+- [ ] Integration of OpenFHE.
+- [ ] Support for GPU-accelerated CKKS algorithm.
+- [ ] Provides FHE interfaces in Tensor Lib.
+- [ ] Provides FHE interfaces in PyLib.
+
 ## Compile and install
 
 ### Environmental requirements

README_cn.md

@@ -26,6 +26,98 @@ https://www.secretflow.org.cn/docs/heu/
 
 其中每一类算法又包含多种不同的实现，部分实现支持硬件加速器，详见[文档](https://www.secretflow.org.cn/docs/heu/latest/zh-Hans/getting_started/algo_choice)
 
+### 目录结构
+
+```text
+.
+├── .circleci  # 存放 circleci 的 CI/CD 配置文件
+├── .github    # 存放 github 配置文件
+├── docs       # 存放 HEU 文档，sphinx 格式
+├── heu        # 存放 HEU 所有实现代码
+│   ├── algorithms     # 存放所有算法实现，包括 PHE 和 FHE
+│   ├── experimental   # 存放一些独立的实验性代码
+│   ├── library        # HE Library 应用层功能实现
+│   │   ├── algorithms # 未迁移至 SPI 下的存量算法，迁移完成后该目录废弃
+│   │   ├── benchmark  # 存放基准测试代码
+│   │   ├── numpy      # 实现一套 Numpy-like 接口
+│   │   └── phe        # PHE Dispatcher 实现（即将废弃，功能由 SPI 代替）
+│   ├── pylib          # 实现 Python binding
+│   └── spi            # 定义 HEU 软硬件接入层接口（SPI）
+│       ├── he         # 存放 HE SPI，以及相关的 Sketches
+│       └── poly       # 定义多项式接口，以及相关的 Sketches
+└── third_party        # 存放编译所需的第三方库链接，库本身会在编译时自动下载
+```
+
+目前 HEU 正在从老的 Dispatcher 架构切换成基于 SPI 的架构，两套架构主要的模块及其代码路径的映射关系如下：
+
+基于 Dispatcher 的架构：
+
+```text
+     Python APIs (Python binding)
+          PATH: heu/pylib
+                  │
+                  ├───────────────────┐
+                  │                   ▼
+                  │    Tensor Lib with Numpy-like API
+                  │        PATH: heu/library/numpy
+                  │                   │
+                  │     ┌─────────────┘
+                  │     │
+                  ▼     ▼
+     PHE Dispatcher & PHE C++ API
+        PATH: heu/library/phe
+                  │
+                  │
+                  ▼
+ Various PHE algorithm implementations
+     PATH: heu/library/algorithms
+```
+
+基于 SPI 的架构：
+
+```text
+    Python APIs (Python binding)
+         PATH: heu/pylib
+                 │
+                 ├───────────────────┐
+                 │                   ▼
+                 │    Tensor Lib with Numpy-like API
+                 │           PATH: heu/numpy
+                 │                   │
+                 │ ┌─────────────────┘
+                 │ │
+                 ▼ ▼
+         HE SPI (C++ APIs)
+         PATH: heu/spi/he
+                 │
+                 │
+                 ▼
+ Various HE algorithm implementations
+        PATH: heu/algorithms
+```
+
+关于 SPI 更详细的介绍请 [点击此处](heu/spi/README.md)
+
+### 2024 工作计划
+
+架构切换里程碑：
+
+- [x] HE SPI：设计一套大一统接口，同时支持所有的 PHE/FHE 算法
+- [ ] 实现 SPI Sketches （doing）
+- [ ] 存量算法迁移至 SPI
+- [ ] HE 算法自动化测试框架
+- [ ] Tensor Lib 底层从 Dispatcher 切到 SPI
+- [ ] PyLib 底层从 Dispatcher 切到 SPI
+
+FHE 里程碑
+
+- [ ] 集成 SEAL，提供 BFV/BGV/CKKS 算法功能
+- [ ] 集成 OpenFHE
+- [ ] 支持 GPU 加速的 CKKS 算法
+- [ ] Tensor Lib 开放 FHE 接口
+- [ ] PyLib 开放 FHE 接口
+
+
 ## 编译和安装
 
 ### 环境要求

WORKSPACE

@@ -7,7 +7,7 @@ load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")
 
 SECRETFLOW_GIT = "https://github.com/secretflow"
 
-YACL_COMMIT_ID = "61a0d980f0895245015b5c128d74eb3cbf758704"
+YACL_COMMIT_ID = "cbe5436f27ff9c4c9c7f7f6664f9fb9f6c852ec2"
 
 git_repository(
     name = "yacl",

heu/library/algorithms/dj/secret_key.cc

@@ -30,7 +30,7 @@ void SecretKey::Init(const MPInt& p, const MPInt& q, uint32_t s) {
   for (auto i = 2u; i <= s + 1; ++i) {
     lut_->pq_pow[i] = {lut_->pq_pow[i - 1].P * p, lut_->pq_pow[i - 1].Q * q};
   }
-  auto const& [ps, qs] = lut_->pq_pow[s];
+  const auto& [ps, qs] = lut_->pq_pow[s];
   pp_ = ps * ps.InvertMod(qs);
   inv_pq_ = {q.InvertMod(ps), p.InvertMod(qs)};
   lut_->precomp.resize(s + 1);
@@ -65,10 +65,10 @@ std::string SecretKey::ToString() const {
 MPInt SecretKey::Decrypt(const MPInt& ct) const {
   MPInt2 z, ls;
   // compute z = c^d mod n^(s+1)
-  auto const& [ps1, qs1] = lut_->pq_pow[s_ + 1];
+  const auto& [ps1, qs1] = lut_->pq_pow[s_ + 1];
   z = {(ct % ps1).PowMod(lambda_, ps1), (ct % qs1).PowMod(lambda_, qs1)};
   //  compute ls = L(z) mod n^s
-  auto const& [ps, qs] = lut_->pq_pow[s_];
+  const auto& [ps, qs] = lut_->pq_pow[s_];
   ls = {inv_pq_.P.MulMod((z.P - MPInt::_1_) / n_.P, ps),
         inv_pq_.Q.MulMod((z.Q - MPInt::_1_) / n_.Q, qs)};
 

heu/library/algorithms/paillier_gpu/decryptor.cc

@@ -47,8 +47,9 @@ std::vector<Plaintext> Decryptor::Decrypt(ConstSpan<Ciphertext> cts) const {
   std::vector<Plaintext> ptx_res(count);
   for (unsigned int i = 0; i < count; i++) {
     ptx_res[i] = Plaintext(0, 512);
-    ptx_res[i].FromMagBytes(yacl::ByteContainerView((uint8_t*)(gpts[i].m), 512),
-                            algorithms::Endian::little);
+    ptx_res[i].FromMagBytes(
+        yacl::ByteContainerView((uint8_t *)(gpts[i].m), 512),
+        algorithms::Endian::little);
 
     // if the value is negative (the judgment condition is greater than n/2),
     // then -n

heu/library/phe/base/variant_helper.h

@@ -44,7 +44,7 @@ Overloaded(Ts...) -> Overloaded<Ts...>;
 
 #define HE_DISPATCH_RET(T, ...)                        \
   ::heu::lib::phe::Overloaded {                        \
-    [](const std::monostate&) -> T {                   \
+    [](const std::monostate &) -> T {                  \
       YACL_THROW("illegal variable (no schema info)"); \
     },                                                 \
         HE_FOR_EACH(__VA_ARGS__),                      \

heu/spi/he/BUILD.bazel

@@ -2,39 +2,48 @@ load("@yacl//bazel:yacl.bzl", "yacl_cc_library", "yacl_cc_test")
 
 yacl_cc_library(
     name = "he",
-    srcs = ["kit.cc"],
-    hdrs = ["kit.h"],
+    srcs = ["he_kit.cc"],
+    hdrs = ["he_kit.h"],
     visibility = ["//visibility:public"],
     deps = [
         ":decryptor",
         ":encoder",
         ":encryptor",
         ":evaluator",
+        ":item_manipulator",
     ],
 )
 
 yacl_cc_library(
-    name = "base",
-    srcs = ["base.cc"],
-    hdrs = ["base.h"],
+    name = "item",
+    srcs = ["item.cc"],
+    hdrs = ["item.h"],
     deps = [
         "@yacl//yacl/utils/spi",
     ],
 )
 
+yacl_cc_library(
+    name = "item_manipulator",
+    hdrs = ["item_manipulator.h"],
+    deps = [
+        ":item",
+    ],
+)
+
 yacl_cc_library(
     name = "encryptor",
     hdrs = ["encryptor.h"],
     deps = [
-        ":base",
+        ":item",
     ],
 )
 
 yacl_cc_library(
     name = "decryptor",
     hdrs = ["decryptor.h"],
     deps = [
-        ":base",
+        ":item",
     ],
 )
 
@@ -46,20 +55,20 @@ yacl_cc_library(
         "word_evaluator.h",
     ],
     deps = [
-        ":base",
+        ":item",
     ],
 )
 
 yacl_cc_library(
     name = "encoder",
     srcs = ["encoder.h"],
-    deps = [":base"],
+    deps = [":item"],
 )
 
 yacl_cc_test(
-    name = "base_test",
-    srcs = ["base_test.cc"],
+    name = "item_test",
+    srcs = ["item_test.cc"],
     deps = [
-        ":base",
+        ":item",
     ],
 )