Re-stage to allow variable sized chunks.

[danielballan]

Error report from @xianghuix:

The two routines you like to compare are fly_scan in 40-scan.py, and fly_scan2 in 98-user_scan.py. fly_scan2 is the new routine. Here is the problem description

One example data set is that with uid ‘ec0d583d’. The command used for this scan is
RE(fly_scan2(0.1, relative_rot_angle=200, period=0.1, chunk_size=360, out_x=600, out_y=None, out_z=None, out_r=None, rs=6, relative_move_flag=1, note=’fly_test’, binning=[1, 1], simu=True)
With this command, it is supposed to save 360 sample images as a single chunk, then 20 background images and 20 dark images. I checked the HDF5 writer plugin during scan and saw it saved all total 400 images. However, in bluesky, I checked the images with command
imgs = list(db[‘ec0d583d’].data(“Andor_image”))
and found ‘imgs’ is composed of three elements, each of them includes 20 images. The first element in imgs should include 360 images. Where do other images go?
The reason I think it save 20 images in each group is that the Andor camera ‘Andor.cam.num_images’ was set to 20. However, HDF5 writer indeed streamed 360 images in the first group. Why does db only find 20 of them?

I believe @cryos and I understand the cause. In chronological order:

1. Before fly_scan2 is run, the detector's num_images is 20 (left as it was at the end of the previous scan).
2. The detector is staged. Its num_images is read at this point (still 20) and written down in a Resource document. See 'frame_per_point': 20 below.

{'spec': 'AD_HDF5',
'resource_path': '2020/08/12/d5ad62cb-0a36-4d34-b1ca_000000.h5',
'root': '/NSLS2/xf18id1/DATA/Andor',
'resource_kwargs': {'frame_per_point': 20},
'path_semantics': 'posix',
'uid': 'a8879517-9e90-4a66-b11a-91fec639ee4a',
'run_start': 'ec0d583d-37af-445a-95f9-f6dae5014ed2'},

3. The detector's num_images is set to 360 and then images are acquired. All 360 frames are written via the HDF5 plugin, but databroker will only be "aware" of 20 of them.
4. The detector's num_images is set to 20, and 20 frames each of background and dark are taken.

I believe that this issue has come up before. It led to the prototype bluesky/area-detector-handlers#11 which would allow one HDF5 to contain chunks of unequal sizes (e.g. [360, 20, 20] in this case). That work hasn't yet been pushed through to completion (cc @tacaswell @mrakitin).

In this PR, I propose changes to fly_scan2 that workaround the issue. By paying a small performance penalty (likely ~1 second) we can write the 360 frames to a different HDF5 file from the rest. A better fix would be to complete and deploy bluesky/area-detector-handlers#11, but I offer this possibility as a short-term workaround.

I have not tested this as I don't have access to an Andor for testing. Please give it a try.

[xianghuix]

Do you suggest to have two different plans in fly_scan2, so run sample scan acquisitions with one plan and save as one file, then run another scan to acquire reference and dark images and save in another file? Is this correct? Thanks, Xianghui

…

On Aug 13, 2020, at 2:29 PM, Dan Allan ***@***.***> wrote: Error report from @xianghuix <https://github.com/xianghuix>: The two routines you like to compare are fly_scan in 40-scan.py, and fly_scan2 in 98-user_scan.py. fly_scan2 is the new routine. Here is the problem description One example data set is that with uid ‘ec0d583d’. The command used for this scan is RE(fly_scan2(0.1, relative_rot_angle=200, period=0.1, chunk_size=360, out_x=600, out_y=None, out_z=None, out_r=None, rs=6, relative_move_flag=1, note=’fly_test’, binning=[1, 1], simu=True) With this command, it is supposed to save 360 sample images as a single chunk, then 20 background images and 20 dark images. I checked the HDF5 writer plugin during scan and saw it saved all total 400 images. However, in bluesky, I checked the images with command imgs = list(db[‘ec0d583d’].data(“Andor_image”)) and found ‘imgs’ is composed of three elements, each of them includes 20 images. The first element in imgs should include 360 images. Where do other images go? The reason I think it save 20 images in each group is that the Andor camera ‘Andor.cam.num_images’ was set to 20. However, HDF5 writer indeed streamed 360 images in the first group. Why does db only find 20 of them? I believe @cryos <https://github.com/cryos> and I understand the cause. In chronological order: Before fly_scan2 is run, the detector's num_images is 20 (left as it was at the end of the previous scan). The detector is staged. Its num_images is read at this point (still 20) and written down in a Resource document. See 'frame_per_point': 20 below. {'spec': 'AD_HDF5', 'resource_path': '2020/08/12/d5ad62cb-0a36-4d34-b1ca_000000.h5', 'root': '/NSLS2/xf18id1/DATA/Andor', 'resource_kwargs': {'frame_per_point': 20}, 'path_semantics': 'posix', 'uid': 'a8879517-9e90-4a66-b11a-91fec639ee4a', 'run_start': 'ec0d583d-37af-445a-95f9-f6dae5014ed2'}, The detector's num_images is set to 360 and then images are acquired. All 360 frames are written via the HDF5 plugin, but databroker will only be "aware" of 20 of them. The detector's num_images is set to 20, and 20 frames each of background and dark are taken. I believe that this issue has come up before. It led to the prototype bluesky/area-detector-handlers#11 <bluesky/area-detector-handlers#11> which would allow one HDF5 to contain chunks of unequal sizes (e.g. [360, 20, 20] in this case). That work hasn't yet been pushed through to completion (cc @tacaswell <https://github.com/tacaswell> @mrakitin <https://github.com/mrakitin>). In this PR, I propose changes to fly_scan2 that workaround the issue. By paying a small performance penalty (likely ~1 second) we can write the 360 frames to a different HDF5 file from the rest. A better fix would be to complete and deploy bluesky/area-detector-handlers#11 <bluesky/area-detector-handlers#11>, but I offer this possibility as a short-term workaround. I have not tested this as I don't have access to an Andor for testing. Please give it a try. You can view, comment on, or merge this pull request online at: #5 <#5> Commit Summary Restage to allow variable sized chunks. File Changes M startup/98-user_scan.py <https://github.com/NSLS-II-FXI/profile_collection/pull/5/files#diff-7653447c90c20382515cd05167089c89> (29) Patch Links: https://github.com/NSLS-II-FXI/profile_collection/pull/5.patch <https://github.com/NSLS-II-FXI/profile_collection/pull/5.patch> https://github.com/NSLS-II-FXI/profile_collection/pull/5.diff <https://github.com/NSLS-II-FXI/profile_collection/pull/5.diff> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABOSZFQ3NTKTKEMNT7FOO73SAQWJHANCNFSM4P6XDR5A>.

[danielballan]

No, not at all. It's still one plan, and it still generates one "Run" (one entry in databroker). It just happens to write to two different HDF5 files. That detail will not change how you execute the plan or how you access its data from databroker.

[mrakitin]

Nice workaround! I suggested one correction to a typo in a comment.