[Client encryption]: Performance and Reliability improvements for StreamProcessor #5385
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… compression, null-skip, malformed number, comments)
…r paths (format, compression, base64, decompression)
…ei partial skip fallback
… for unknown type markers with diagnostics; improve UTF-8 and deserialization error messages with path context; unify stream ownership (caller-owned); make initial buffer size per-instance with legacy default.
…update usages in encrypt/decrypt partials and tests.
…ssor; reduce allocations: use ValueTextEquals for _ei, precompute top-level encrypted path maps to avoid string concat and lookups.
…strument decryptor to record bytesRead/bytesWritten/propertiesDecrypted/compressedPathsDecompressed/elapsedMs; use CountingStream for non-seekable outputs.
…; share JsonWriterOptions (Indented=false) across encrypt/decrypt for throughput predictability.
…ool on growth and always return final rented buffer to prevent leaks
…ip for _ei value using state (no allocations), replacing brittle isIgnoredBlock handling
…String/Number tokens with pooled buffers; size ciphertext buffer from sequence length
…ll base64 strings to reduce pooling on decrypt path
…oss-buffer _ei skip (scalar), multi-segment string/number tokens, and small/large ciphertexts across buffers
…line test; select baseline by TFM (net8 vs net6) and re-enable when API stabilizes
…ero-alloc top-level name match in decryptor+encryptor; leave logger args unchanged
…ion and remove runtime ICollection checks; update producers/consumers and tests
… simplify decrypt streaming path
Eliminates all code, tests, and metadata related to payload compression in the custom Cosmos encryption implementation. CompressionOptions, compressed path handling, and associated validation logic are removed from both runtime and test code, restricting encryption to the MDE format only.
Added the Utf8JsonWriter parameter to the EncryptionPipelineState constructor and updated its initialization. This change ensures the writer is available for encryption processing within the pipeline state.
- Remove Bucket() method and manual power-of-two bucketing logic - Simplify buffer allocation to rely on ArrayPool's internal bucketing - Update EnsureCapacity to use direct size capping instead of bucketing - Simplify buffer growth logic in streaming decrypt path
- Remove small seekable stream fast path optimization - Simplify DecryptStreamAsync to always use StreamProcessAsync - Remove SmallPayloadMaxBytes constant no longer needed - Streamline code path for all stream sizes
Introduced a local helper method to validate Base64 decoding and replaced repeated decoding logic with this method for improved readability and maintainability. Removed redundant comments and streamlined variable usage in the decryption process.
…cation helper; integrate across encryptor/decryptor + emulator tests; include boolean/numeric checks
…pe marker decoding (AssertEncryptedDocument) and update tests
Updated using statements to include System.Text.Json and relevant namespaces. Improved type declarations for variables in foreach loops and method bodies for clarity and consistency.
Changed DecodeValidated to a static method and added cipher, expectedLen, and pathLabel as parameters. Updated all calls to pass the required arguments, improving clarity and reducing reliance on outer scope.
Changed ValidateRawEncryptedAsync to accept an IReadOnlyDictionary of encrypted properties instead of an IEnumerable of plaintext values. Updated all test calls to provide explicit property-value mappings, improving clarity and flexibility in encrypted document validation.
Simplifies the initialization of plainMap in ValidateRawEncryptedAsync by removing unnecessary comments and explicit type casting. This improves code clarity in the StreamProcessorEmulatorTests.
Replaces strict type comparison with 'is not' to allow MdeEncryptor hierarchy. Updates exception message to reflect support for MdeEncryptor subclasses.
Removed the Ignore attribute from the ContractChanges test in ContractEnforcementTests, allowing it to run again. This change reflects that contract enforcement is no longer temporarily ignored.
adamnova
requested review from
FabianMeiswinkel,
Pilchie,
kirankumarkolli and
sboshra
as code owners
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
… handling Replaces '#if ENCRYPTION_CUSTOM_PREVIEW && NET8_0_OR_GREATER' with '#if NET8_0_OR_GREATER' in source and test files, enabling code for all NET8_0_OR_GREATER builds. Adds comprehensive unit tests for CandidatePaths and extensive tests for StreamProcessorEncryptor/Decryptor to cover edge cases involving Utf8JsonReader's ValueSequence, fragmented streams, and error handling. Also comments out obsolete assertions related to removed compression support.
Updated the IMdeJsonProcessorAdapter interface and its implementations to include a CosmosDiagnosticsContext parameter in the EncryptAsync method. This change enables telemetry and diagnostics support during encryption operations.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pull Request Template
Description
Please include a summary of the change and which issue is fixed. Include samples if adding new API, and include relevant motivation and context. List any dependencies that are required for this change.
Type of change
Please delete options that are not relevant.
Closing issues
#4678