feat: overwrite_partial_chunks #682
Conversation
william-silversmith
left a comment
william-silversmith
left a comment
There was a problem hiding this comment.
This is a great start! I think it's pretty close. Can you add a test or two that uses this flag?
|
Fixed the things you mentioned. |
cloudvolume/chunks.py
Outdated
| return cseg.compress(subvol, block_size=block_size, order=order) | ||
|
|
||
| def encode_raw(subvol): | ||
| if not subvol.flags['F_CONTIGUOUS']: |
There was a problem hiding this comment.
I don't understand this optimization. It's true that tobytes will run very fast, but that's because the expensive part is getting done in asfortranarray. Secondly, asfortranarray doesn't do anything when it's already in fortran order so the if statement isn't necessary.
There was a problem hiding this comment.
I tested this and asfortranarray + tobytes is much faster than tobytes for C order arrays above a certain size (70% speedup for 64MB), and 50% speedup for non-contiguous slices. I forget if C order arrays can even make it this far without conversion, but non-contiguous slices definitely can. I will remove the if statement, but the performance benefit is there if you can accept the small memory cost of making a copy of the chunk - guessing that tobytes doesn't bother making a contiguous copy.
As a side note, what I really need is an option to skip the encoding and just keep the volume in memory because I'm spending nontrivial amount of time (20% of the entire task time) serialising / deserialising temporary volumes being written to mem://, but I'm not sure how you feel about extending to just keeping raw arrays in memory. The nontrivial time being spent on encoding was why I was looking into the optimisation.
No description provided.