feat: rgb, v2 #141

gselzer · 2025-02-20T00:40:24Z

This PR revives the work done in #41, but updated for the new viewer. I'm guessing that the design can be further refined, and there are more bugs to be ironed out.

Still lots of cleaning, testing, bugfixing to do. Notably, there were many comments on #41 that should be gone through and addressed here.

Note that histograms are broken :)

These are conventionally RGB(A) images in Python

codecov · 2025-02-20T23:23:25Z

Codecov Report

Attention: Patch coverage is 87.90323% with 15 lines in your changes missing coverage. Please review.

Project coverage is 80.74%. Comparing base (bd210ab) to head (d37cec9).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
src/ndv/models/_data_display_model.py	73.33%	8 Missing ⚠️
src/ndv/controllers/_array_viewer.py	75.00%	7 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #141      +/-   ##
==========================================
+ Coverage   80.64%   80.74%   +0.10%     
==========================================
  Files          45       45              
  Lines        4195     4270      +75     
==========================================
+ Hits         3383     3448      +65     
- Misses        812      822      +10

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Otherwise the rgb image shows in grayscale :)

src/ndv/data.py

It's useful for testing higher-dim RGB data, as might come from a RGB camera MDA via Micro-Manager in the real world

I did not realize that both images and volumes use PyGFXImageHandle

gselzer · 2025-02-21T23:46:19Z

@tlambert03 I'm generally happy with the functionality here and would appreciate your thoughts and high-level review. Couple points I'd like your opinions on:

I've changed the LutKey type alias into ChannelKey, which now is basically anything Hashable. The reason that I need this is because I need a specific identifier for RGB mode - it cannot be an integer, and must be differentiable from None - to trigger an appropriate RGB LutView and to correctly slice (read: not slice) the data. However this typing may be too broad - what do you think?
On a broader level, I'm not quite satisfied with how much RGB mode gets special-cased - in a sense, it needs special casing because it's the only mode we (want to) support where the channels must be combined for meaning - but it would be nice to minimize the special cases.
The histogram right now is lacking because there's no special RGB support - I'm thinking this may best be solved in another PR, because it's not technically wrong right now.
There's an obnoxious case where, if you do not specify RGB mode on viewer construction (i.e. remove the data model from being passed to imshow in the new example, you'll start out in composite mode where the channel is one of the visible axes. I try to make switching to RGB do the right thing and change the visible axes (in _ArrayDataDisplayModel._on_channel_mode_change), but then there's no way to go back to the previously shown axes. This feels unfortunate/wrong (hence the leftover FIXME), but maybe it's just necessary?
This could just be random, but I feel like recent commits have more tests that sporadically fail. I didn't edit any tests so I'm not sure what would be causing this - if you have any ideas I'd be interested!

gselzer · 2025-03-12T18:09:28Z

@tlambert03 did you ever get a chance to take a look at this?

src/ndv/models/_data_display_model.py

examples/rgb.py

tlambert03

beyond the few small comments, i think everything in here looks good! nice job adding a new feature with minimal lines :)

tlambert03 · 2025-03-17T20:14:37Z

remaining segfaults do seem to be specific to this PR. locally, I can run make test extras=pyqt,pygfx py=3.10 over and over on main with no issue, and this PR is giving segfault each time. Will try to dig

tlambert03 · 2025-03-17T20:16:26Z

when running with lldb, I get this:

tests/test_examples.py::test_example[rgb.py] Process 56522 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x27c)
    frame #0: 0x0000000129340fa4 QtWidgets`QWidgetPrivate::drawWidget(QPaintDevice*, QRegion const&, QPoint const&, QFlags<QWidgetPrivate::DrawWidgetFlag>, QPainter*, QWidgetRepaintManager*) + 3152
QtWidgets`QWidgetPrivate::drawWidget:
->  0x129340fa4 <+3152>: ldr    w9, [x8, #0x228]
    0x129340fa8 <+3156>: tbnz   w9, #0x1e, 0x129340fd4 ; <+3200>
    0x129340fac <+3160>: ldrh   w8, [x23, #0x18]
    0x129340fb0 <+3164>: cbnz   w8, 0x129340fe4 ; <+3216>
(lldb)  bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x27c)
  * frame #0: 0x0000000129340fa4 QtWidgets`QWidgetPrivate::drawWidget(QPaintDevice*, QRegion const&, QPoint const&, QFlags<QWidgetPrivate::DrawWidgetFlag>, QPainter*, QWidgetRepaintManager*) + 3152
    frame #1: 0x000000012933ffe0 QtWidgets`QWidgetPrivate::paintOnScreen(QRegion const&) + 404
    frame #2: 0x000000012933fdb8 QtWidgets`QWidgetPrivate::syncBackingStore() + 196
    frame #3: 0x00000001293512f0 QtWidgets`QWidget::event(QEvent*) + 1420
    frame #4: 0x0000000128ab1160 QtWidgets.abi3.so`sipQWidget::event(QEvent*) + 224
    frame #5: 0x0000000129300298 QtWidgets`QApplicationPrivate::notify_helper(QObject*, QEvent*) + 336
    frame #6: 0x0000000129301d18 QtWidgets`QApplication::notify(QObject*, QEvent*) + 3260
    frame #7: 0x0000000128a995d0 QtWidgets.abi3.so`sipQApplication::notify(QObject*, QEvent*) + 248
    frame #8: 0x000000012845e390 QtCore`QCoreApplication::notifyInternal2(QObject*, QEvent*) + 200
    frame #9: 0x000000012845f49c QtCore`QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) + 532
    frame #10: 0x000000012e86914c libqcocoa.dylib`___lldb_unnamed_symbol1725 + 360
    frame #11: 0x000000012e867800 libqcocoa.dylib`___lldb_unnamed_symbol1718 + 1008

so there's something going on during the paint event

tlambert03 · 2025-03-17T20:34:53Z

wow... i think i might have fixed it with this change. What I think is happening is that running examples/rgb.py, on pygfx, hits the warning in the LutView:

/Users/talley/dev/self/ndv/src/ndv/views/bases/_lut_view.py:93: UserWarning: Cannot set colormap on an RGB image
  self.set_colormap(model.cmap)

I believe that warning was getting upcast in tests to an exception. For some reason, simply letting the application process events (where it would have called exec() is enough to let it fail gracefully).

However, it raises a new point. @gselzer, it probably shouldn't show a warning to simply call ndv.imshow(ndv.data.rgba()), so perhaps that pygfx warning is either too much? or we should make it a bit more specific such that it only warns if the user is somehow specifically trying to do something they shouldn't (which, i think might be a bit hard to detect at the moment). I'm fine just removing the warning...

tlambert03 · 2025-03-17T20:39:23Z

one more thing, could we add a simple test (in addition to the example test) in test_controller that adds an RGB image, and asserts some basic things, such as that the magic is working.

gselzer · 2025-03-17T20:42:02Z

However, it raises a new point. @gselzer, it probably shouldn't show a warning to simply call ndv.imshow(ndv.data.rgba()), so perhaps that pygfx warning is either too much?

Yeah, it's probably too much - I'm a fan of silently ignoring changes to the colormap.

one more thing, could we add a simple test (in addition to the example test) in test_controller that adds an RGB image, and asserts some basic things, such as that the magic is working.

Sure! I can give that a shot tomorrow!

...I yield...for now

gselzer · 2025-03-18T22:40:13Z

@tlambert03 I've added a RGB magic test, and have removed the warning for setting the cmap on an RGB image. I additionally noticed that the wx/jupyter RGB lut views had a misplaced label (likely from the main merge), and fixed them.

This took a lot longer than I thought, as I was plagued by qt widget leaks. I'm fairly certain they're coming from the _QLutView itself (specifically from the cmap combo box), however I was not successful in pinpointing the issue. I'm pretty sure they would also arise anytime we create an ArrayView with data passed through the constructor (calling ArrayViewer._fully_synchronize_view), but we don't ever test this pathway without allowing leaks. Maybe something to look at in the future...

tlambert03 · 2025-03-18T23:59:47Z

I'm fairly certain they're coming from the _QLutView itself (specifically from the cmap combo box), however I was not successful in pinpointing the issue. I'm pretty sure they would also arise anytime we create an ArrayView with data passed through the constructor (calling ArrayViewer._fully_synchronize_view), but we don't ever test this pathway without allowing leaks. Maybe something to look at in the future...

thanks, that's a great lead

tlambert03

LGTM!

tlambert03 · 2025-03-19T00:01:19Z

src/ndv/controllers/_array_viewer.py

@@ -563,7 +598,15 @@ def _get_values_at_world_point(self, x: int, y: int) -> dict[ChannelKey, float]:
        values: dict[ChannelKey, float] = {}
        for key, ctrl in self._lut_controllers.items():
            if (value := ctrl.get_value_at_index((y, x))) is not None:
-                values[key] = value
+                # Handle RGB
+                if key == "RGB" and isinstance(value, np.ndarray):


just nothing for the future. I don't love the addition of another special unique key here (I recognize that None was already that, and I don't have an immediate alternate suggestion at the moment). absolutely fine for now, and maybe forever... just noticed it :)

Yeah...I'm also not a fan, as I mentioned here, but I wrote that for lack of a better option...very open to improving it later!

Ah yes sorry I forgot you pointed that out. Agreed though, it’s a fine workaround

gselzer added 2 commits February 19, 2025 17:44

RGBA in Qt

831119f

Note that histograms are broken :)

First cut for wx, jupyter, pygfx

2b3e6bc

gselzer added the enhancement New feature or request label Feb 20, 2025

gselzer self-assigned this Feb 20, 2025

gselzer mentioned this pull request Feb 20, 2025

feat: RGB support #41

Closed

gselzer added 4 commits February 20, 2025 09:37

Remove rgb example cruft

337dc28

Enable RGBA view for (X, Y, C) images

688db52

These are conventionally RGB(A) images in Python

Merge remote-tracking branch 'upstream/main' into rgb-v2

b1a7086

Many small fixes

6a9601c

gselzer mentioned this pull request Feb 20, 2025

chore: Purge v1 #142

Merged

gselzer added 2 commits February 20, 2025 17:41

Merge remote-tracking branch 'upstream/main' into rgb-v2

a0501a4

examples: Use ArrayDisplayModel for RGB mode

8aef4fb

Otherwise the rgb image shows in grayscale :)

gselzer commented Feb 20, 2025

View reviewed changes

src/ndv/data.py Outdated Show resolved Hide resolved

gselzer added 5 commits February 21, 2025 08:17

Remove RGBA test dataset

9808015

Add RGBA back as a 3D RGB dataset

3761b9b

It's useful for testing higher-dim RGB data, as might come from a RGB camera MDA via Micro-Manager in the real world

Clean up pygfx RGB changeset

2ebbf5d

pygfx: Fix conditional

b99f90c

vispy: Simplify set_data dims check

b0a74c4

gselzer force-pushed the rgb-v2 branch from 6da8610 to b2243b6 Compare February 21, 2025 21:53

gselzer added 4 commits February 21, 2025 15:53

Wx: Fix RGB lutview label

b2243b6

Enable hovering on RGB images

ab75df5

Fix hovering

ff25b5e

pygfx: fix bug with image/volume cmap

235e21a

I did not realize that both images and volumes use PyGFXImageHandle

gselzer added 2 commits February 26, 2025 10:41

Merge remote-tracking branch 'upstream/main' into rgb-v2

9d24110

Fix histograms on Jupyter, Wx

5676550

gselzer marked this pull request as ready for review March 12, 2025 21:07

tlambert03 reviewed Mar 14, 2025

View reviewed changes

src/ndv/models/_data_display_model.py Outdated Show resolved Hide resolved

tlambert03 reviewed Mar 14, 2025

View reviewed changes

examples/rgb.py Outdated Show resolved Hide resolved

tlambert03 approved these changes Mar 14, 2025

View reviewed changes

gselzer and others added 10 commits March 14, 2025 13:21

Revert process_request to a static method

4f388c3

ArrayView.add_lut_view: Remove default param value

0b688d5

Vispy: Refactor Image handle dims check

41479b0

Add RGB magic

8463d28

Merge remote-tracking branch 'upstream/main' into rgb-v2

5ce2960

style(pre-commit.ci): auto fixes [...]

f889141

Fix magic

54b9a55

small changes to magic

f6c3d65

Merge branch 'main' into rgb-v2

7e04883

Merge branch 'main' into rgb-v2

951f55d

fix segfault

7929362

gselzer added 3 commits March 18, 2025 12:36

Add RGB magic test

84ba796

use any_app for test

a0a45f5

Resolve widget leaks

2fb7de3

gselzer force-pushed the rgb-v2 branch from 05cac54 to 2fb7de3 Compare March 18, 2025 19:35

gselzer added 4 commits March 18, 2025 16:01

Actually fix widget leaks

c04bc41

Allow leaks

6eb70ad

...I yield...for now

Downgrade RGB cmap warnings

630a3f4

Fix wx/jupyter RGB LutViews

d37cec9

tlambert03 approved these changes Mar 19, 2025

View reviewed changes

tlambert03 merged commit d11d9a0 into pyapp-kit:main Mar 19, 2025
51 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: rgb, v2 #141

feat: rgb, v2 #141

gselzer commented Feb 20, 2025 •

edited

Loading

codecov bot commented Feb 20, 2025 •

edited

Loading

gselzer commented Feb 21, 2025 •

edited

Loading

gselzer commented Mar 12, 2025

tlambert03 left a comment

tlambert03 commented Mar 17, 2025

tlambert03 commented Mar 17, 2025

tlambert03 commented Mar 17, 2025

tlambert03 commented Mar 17, 2025

gselzer commented Mar 17, 2025

gselzer commented Mar 18, 2025 •

edited

Loading

tlambert03 commented Mar 18, 2025

tlambert03 left a comment

tlambert03 Mar 19, 2025

gselzer Mar 19, 2025

tlambert03 Mar 19, 2025

feat: rgb, v2 #141

feat: rgb, v2 #141

Conversation

gselzer commented Feb 20, 2025 • edited Loading

codecov bot commented Feb 20, 2025 • edited Loading

Codecov Report

gselzer commented Feb 21, 2025 • edited Loading

gselzer commented Mar 12, 2025

tlambert03 left a comment

Choose a reason for hiding this comment

tlambert03 commented Mar 17, 2025

tlambert03 commented Mar 17, 2025

tlambert03 commented Mar 17, 2025

tlambert03 commented Mar 17, 2025

gselzer commented Mar 17, 2025

gselzer commented Mar 18, 2025 • edited Loading

tlambert03 commented Mar 18, 2025

tlambert03 left a comment

Choose a reason for hiding this comment

tlambert03 Mar 19, 2025

Choose a reason for hiding this comment

gselzer Mar 19, 2025

Choose a reason for hiding this comment

tlambert03 Mar 19, 2025

Choose a reason for hiding this comment

gselzer commented Feb 20, 2025 •

edited

Loading

codecov bot commented Feb 20, 2025 •

edited

Loading

gselzer commented Feb 21, 2025 •

edited

Loading

gselzer commented Mar 18, 2025 •

edited

Loading