Skip to content

feat: overwrite_partial_chunks #682

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dodamih wants to merge 4 commits into
base: master
Choose a base branch
from
Open

feat: overwrite_partial_chunks #682

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
f055b75
feat: overwrite_partial_chunks
dodamih Jan 1, 2026
4b441c6
chore: refactor, add typing, add tests
dodamih Jan 2, 2026
526394e
feat: raw encoding optimisation
dodamih Jan 2, 2026
63aee8d
chore: remove unnecessary conditional
dodamih Jan 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions cloudvolume/chunks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -321,6 +321,8 @@ def encode_compressed_segmentation(
return cseg.compress(subvol, block_size=block_size, order=order)

def encode_raw(subvol):
if not subvol.flags['F_CONTIGUOUS']:
Copy link
Contributor

@william-silversmith william-silversmith Jan 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand this optimization. It's true that tobytes will run very fast, but that's because the expensive part is getting done in asfortranarray. Secondly, asfortranarray doesn't do anything when it's already in fortran order so the if statement isn't necessary.

Copy link
Author

@dodamih dodamih Jan 3, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested this and asfortranarray + tobytes is much faster than tobytes for C order arrays above a certain size (70% speedup for 64MB), and 50% speedup for non-contiguous slices. I forget if C order arrays can even make it this far without conversion, but non-contiguous slices definitely can. I will remove the if statement, but the performance benefit is there if you can accept the small memory cost of making a copy of the chunk - guessing that tobytes doesn't bother making a contiguous copy.

As a side note, what I really need is an option to skip the encoding and just keep the volume in memory because I'm spending nontrivial amount of time (20% of the entire task time) serialising / deserialising temporary volumes being written to mem://, but I'm not sure how you feel about extending to just keeping raw arrays in memory. The nontrivial time being spent on encoding was why I was looking into the optimisation.

subvol = np.asfortranarray(subvol)
return subvol.tobytes('F')

def encode_kempressed(subvol):
Expand Down