mount: fuse fs performance fix, docstring #9246
Merged
+17
−2
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## 1.4-maint #9246 +/- ##
==========================================
Coverage 80.59% 80.59%
==========================================
Files 38 38
Lines 11256 11256
Branches 1771 1771
==========================================
Hits 9072 9072
Misses 1615 1615
Partials 569 569 ☔ View full report in Codecov by Sentry. |
ThomasWaldmann
changed the title
ThomasWaldmann
force-pushed
the
fuse-fixes-1.4
branch
from
7321daa to
da51e3d
Compare
ThomasWaldmann
force-pushed
the
fuse-fixes-1.4
branch
from
da51e3d to
0017e34
Compare
ThomasWaldmann
changed the title
### Performance Comparison: `bytearray.extend()` vs. Concatenation The efficiency difference between `meta.extend(bytes(N))` and `meta = meta + bytes(N)` stems from how Python manages memory and objects during these operations. #### 1. In-Place Modification vs. Object Creation - **`bytearray.extend()`**: This is an **in-place** operation. If the current memory block allocated for the `bytearray` has enough extra capacity (pre-allocated space), Python simply writes the new bytes into that space and updates the length. If it needs more space, it uses `realloc()`, which can often expand the existing memory block without moving the entire data set to a new location. - **Concatenation (`+`)**: This creates a **completely new** `bytearray` object. It allocates a new memory block large enough to hold the sum of both parts, copies the contents of `meta`, copies the contents of `bytes(N)`, and then reassigns the variable `meta` to this new object. #### 2. Computational Complexity - **`bytearray.extend()`**: In the best case (when capacity exists), it is **O(K)**, where K is the number of bytes being added. In the worst case (reallocation), it is **O(N + K)**, but Python uses an over-allocation strategy (growth factor) that amortizes this cost, making it significantly faster on average. - **Concatenation (`+`)**: It is always **O(N + K)** because it must copy the existing `N` bytes every single time. As the `bytearray` grows larger (e.g., millions of items in a backup), this leads to **O(N²)** total time complexity across multiple additions, as you are repeatedly copying an ever-growing buffer. #### 3. Memory Pressure and Garbage Collection - Concatenation briefly requires memory for **both** the old buffer and the new buffer simultaneously before the old one is garbage collected. This increases the peak memory usage of the process. - `extend()` is more memory-efficient as it minimizes the need for multiple large allocations and relies on the underlying memory manager's ability to resize buffers efficiently. In the context of `borg mount`, where `meta` can grow to be many megabytes or even gigabytes for very large repositories, using concatenation causes a noticeable slowdown as the number of archives or files increases, whereas `extend()` remains performant.
ThomasWaldmann
force-pushed
the
fuse-fixes-1.4
branch
from
b6dfcc9 to
4eed3ec
Compare
ThomasWaldmann
changed the title
|
Backport failed for Please cherry-pick the changes locally and resolve any conflicts. git fetch origin master
git worktree add -d .worktree/backport-9246-to-master origin/master
cd .worktree/backport-9246-to-master
git switch --create backport-9246-to-master
git cherry-pick -x 646f279062faad0b88c2feeff2e0d9a9cf675a2a 4eed3ecd5b378c8cd899fd3fe89df001f7d43a65 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.