High speed ndtiff writer for Python

[LukasMicroscopy]

Hi!

The question is about writing ndtiff fast with python.

We use the ndtiff format as our main format in the lab. Mainly by micromanager, but with the availability of the ndtiff writer on python also as format after post processing.
Due to a Python project with high speed recordings, we need a high-performance file writer and we wish to use the ndtiff format.
Unfortunately, I stumbled over some limitations with the ndtiff Python implementation.

When a new ndtiff-file (ndtiff_file.py) is created, the file is first written with zeroes in maximum file size (Line 67 – 70). This takes quite some time. I guess this is done, to be able to work in read/write mode, to read file content while writing. Changing this to write only would eliminate this file creation time, but I guess the functionality to read while writing is lost?

The native file IO of python is blocking. And I couldn’t change this behaviour (by setting non-blocking flags). Therefore, the file writing process takes a lot of time, blocking in threaded mode. Running ndtiff in multiprocessing is not an option, because copying the data also takes some time. I used asyncio and aiofile to overcome this issue on very simplified file writing tests.

My Question is:
Is there interest in having a write only fast asynchronous ndtiff writer?
if yes, how should I proceed to implement one?
Should I try to implement one within the ndtiff package and make a pull request or
Fork the Projekt, for a separated write_only implementation or are there other thoughts?

Thanks!
Best Lukas

[henrypinkard]

Glad to hear you've found the format useful. At this point the python writer is the least optimized and least tested among the Java/Python readers/writers. I would not be surprised if there are substantial improvements to be had, and if you're interested in experimenting with it and finding them, pull requests welcome! So so long as it passes the tests changes are fair game

When a new ndtiff-file (ndtiff_file.py) is created, the file is first written with zeroes in maximum file size (Line 67 – 70). This takes quite some time.

This was copied from the Java NDTiff writer, where doing so greatly increased performance because repeated calls to resize were expensive. But it may not be the case with the python version. I don't see a problem with changing it.

The native file IO of python is blocking. And I couldn’t change this behaviour (by setting non-blocking flags). Therefore, the file writing process takes a lot of time, blocking in threaded mode

It be worth playing around with disabling the global interpreter lock, since this could allow the code to operate in a truly multithreaded way

[LukasMicroscopy]

Hi!
Thanks for the feedback and advice!
I have multiple changes (tiny and "big"), should i make independent pull requests or on big one?
I tested Python 3.13 and it works fast when threaded, even without disabling the GIL. This is not the case up to Python 3.12.
AsyncIO is still a little bit faster, but i think not worth the hassle for a full and clean implementation.

The changes so far and well tested:
Commenting out the flush command to allocating the file size on the file system: The 4Gb file is then reserved without writing the full space.
Adding a Lock when a new file is generated. This was necessary to make the library thread save (i don't get why it is needed at all, but it works)
Tiny bugfix: somehow the tiff TAG for "samples per pixel" always gets 16 even the data is 8bit.
In question if wanted: orjson is much faster than json, specially with big metadata, but this would mean another external dependency.
Adding zlib compression (tiff comp. 8) option: This would be very useful for us. I tested also the compatibility with standard tiff libraries and it works.
Last: i still have to observe/debug a metadata read error i get when reading very large (multiple 4gb files) ndtiff datasets.

Thanks for your work and help!

[henrypinkard]

All sounds great! Either 1 big or multiple smaller ones -- whatever is easiest on your end. Have you run the NDTiff tests yet? you can set this up locally, but it might just be easier at first to open a PR and see the result in github actions

[LukasMicroscopy]

Ok, i don't know what's easier for me. I did run some tests more or less manually. Is there a good howto to set up the tests locally? I would like to learn that too. I guess i will need some additional days until i start making the pull requests.

[henrypinkard]

I think you just need to navigate to python/ndstorage and do pytest -v .? You probably need to download the test data https://zenodo.org/records/7065552. I'm not sure I ever wrote down the exact sequence

[LukasMicroscopy]

Hi!
The pytest -v . worked find. For some reason i had to call it with python -m pytest -v . in the venv, also downloaded the test data and copied them to the test_data folder.
I tested python 3.13. It is noticeable faster. The non blocking file writes seemed to work in a "simple" threaded test. But in a more complex project with multiple threads it failed to be ether non blocking or stable.
I also tried syncio and implemented a syncio file writer into ndtiff storage. There the problem is, that this approach only works, if the whole python project is a syncio project. That is not a good solution for us. If syncio is running in a thread the non blocking features of syncio are neglected.
The next step for me now is to convert the python ndtiff storage into a cython project. I will let you know when i have something working.
I created some pull requests today, with the tiny things i encountered while working on the code. I also added zlib compression support which includes the tiny changes too.

[henrypinkard]

Nice work!