Negadoctor module order

[dtrtuser]

Is your feature request related to a problem? Please describe.

Currentl module order V5.0 puts negadoctor very late in the pipeline. This is causing many modules like exposure, tone equalizer and color calibration to behave in a counter-intuitive way.

Describe the solution you'd like

By moving negadoctor before the exposure module, the inversion of the image would happen sooner in the pipeline and the subsequent modules would behave as with a regular positive image.

Alternatives

Additional context

[sayoder]

I agree with this suggestion. From the perspective of a frequent negadoctor user and not a Darktable developer, I think negadoctor could actually benefit from a functionality split. The settings in the "film properties" module should be applied early in the pixel pipeline.

To me, it would make sense to ditch the color correction tab entirely, and let us control this with color calibration and the other excellent color modules that already exist in DT. Of course, this presents a conflict; color calibration doesn't like to be enabled at the same time as white balance. Maybe the way around this is to allow setting white balance for the light source in negadoctor itself.

The "print properties" tab, in my view, is a less-powerful version of exposure + filmic rgb. We have far less control over the curves and no good way to handle highlights. In a typical digital workflow, we can boost exposure for the mid-tones, then pull back highlights with filmic. This gets very dangerous with negadoctor, because boosting exposure or density permanently compresses highlights in a non-recoverable manner. I don't understand exactly what's happening technically, but I think this has something to do with negadoctor operating in display-referred space.

I guess I wish we had some way to invert the negative very early in the pipeline, setting D min and D max and perhaps entering a color temperature for the light source, staying in scene-referred space, and then from there use normal Darktable tools as if we're working with a positive digital image. I recognize there may be many technical reasons I don't understand that make this non-feasible.

Ultimately, to solve any of these issues, it will take someone who has a lot of time and energy to rethink this part of the pipeline. I am a programmer but with limited C knowledge, and I've spent a bit of time trying to decipher the negadoctor code, but it would be a massive effort for me. Maybe in the fall when I'm unemployed I could take a look at it.

[Jiyone]

Negadoctor's place is good.

This is causing many modules like exposure, tone equalizer and color calibration to behave in a counter-intuitive way.

Yes it's counter-intuitive, but it's working as intended : Negadoctor inverts the image so the result is inverted. See the pipeline as a pile that have to be read from bottom to top.
Anyway, those mentioned modules can be duplicated / moved after Negadoctor if needed.
Also, color calibration shouldn't be used before Negadoctor with color negative film because it's useless, but exposure may be needed to adjust the exposure of the film scan at its normal place.

To me, it would make sense to ditch the color correction tab entirely, and let us control this with color calibration and the other excellent color modules that already exist in DT.

This will not be the same, the color correction tab works in film's log space. You can neutralize tones with that panel, then color grade using an other module.

We have far less control over the curves and no good way to handle highlights. In a typical digital workflow, we can boost exposure for the mid-tones, then pull back highlights with filmic.

You can do in Negadoctor what you are talking about. :D boost exposure for mid-tones and use the specular highlights tool to pull back highlights, while adjusting the grade (which is a power operation) to get correct contrast.
Using Filmic with Negadoctor is a nonsense since Negadoctor replaces Filmic, filmic's existence is to mimic a film's tonal curve and ... well you already have a film tonal curve in film scans...

This gets very dangerous with negadoctor, because boosting exposure or density permanently compresses highlights in a non-recoverable manner. I don't understand exactly what's happening technically, but I think this has something to do with negadoctor operating in display-referred space.

No it's not operating in display-referred space at all. Negadoctor's code is based on Kodak's Cineon which takes linear data of film. The goal is to have your image in the 0-1 range and the specular highlights tool helps compressing the highlight tones to not loose them.

[sayoder]

No it's not operating in display-referred space at all

Negadoctor does work in display-reffered space, though -- you can see this information when you hover over the module -- its input and output are linear display-referred and non-linear display-referred, respectively.

You can do in Negadoctor what you are talking about. :D boost exposure for mid-tones and use the specular highlights tool to pull back highlights, while adjusting the grade (which is a power operation) to get correct contrast.

The negadoctor tools to handle this are much less powerful than tools that already exist in Darktable, though. There's no way to define the highlight masking, no way to actually shape the curves. The print properties controls are 1-dimensional and they irreversibly compress data in the image. The specular highlights controls are like going back to the Stone Age compared to the highlight reconstruction that exists in filmic rgb.

Also, color calibration shouldn't be used before Negadoctor with color negative film because it's useless,

Consider this use case: I am photographing a shoot in some lighting conditions, and on the first image of my roll I take a photograph of my physical Calibrite Color Checker. In a negadoctor-based workflow, how can I use the photo of that Color Checker to calibrate colors for the rest of my film roll? I simply can't. There's no way to do it. I think that's bizarre and points to an issue with the philosophy of negadoctor.

I make these comments as someone who loves Darktable and negadoctor. I think they're great tools that help me produce some beautiful images. But I think negadoctor's design philosophy fundamentally conflicts with Darktable's design philosophy.

[Jiyone]

Negadoctor does work in display-reffered space, though

Yes because your scan picture is in scene-referred, not the film data.
You are thinking Negadoctor as a scene-referred converter from a film scan image. However, this is not right nor possible because the image on film si not scene-referred either, it's the equivalent of an image output from filmic or sigmoid, which tries to mimic the tone curve of film.

The density (=invert transmittance on a digitized picture) is not a straight line with the corresponding log Exposure

On the other hand, the image of the digitized film is scene-referred, especially if you are digitizing with a DSLR camera (so Darktable will need all modules necessary to correct the sensor, colors, lens ... as usual).

Consider this use case: I am photographing a shoot in some lighting conditions, and on the first image of my roll I take a photograph of my physical Calibrite Color Checker.
In a negadoctor-based workflow, how can I use the photo of that Color Checker to calibrate colors for the rest of my film roll? I simply can't. There's no way to do it. I think that's bizarre and points to an issue with the philosophy of negadoctor.

There is absolutely nothing wrong if you want to do that. I do that occasionally but you have to move the Color Calibration AFTER Negadoctor like this :

If you let Color calibration BEFORE Negadoctor, it's like wanting to do it on the non inverted image :

There's no way to define the highlight masking, no way to actually shape the curves. The print properties controls are 1-dimensional and they irreversibly compress data in the image.

If you want a linear* representation of your highlight to edit them after Negadoctor, you can set the Paper gloss setting to 100% and set exposure to not clip them, and use an exposure module instance to set exposure for the mid-tones, then do what you think is necessary to recover the highlights.
But you will see soon that the Paper gloss setting will already do a good job.

*linear regarding the scan data + inversion, not the image on film because that part of the curve is not supposed to be linear on film.

The specular highlights controls are like going back to the Stone Age compared to the highlight reconstruction that exists in filmic rgb.

The setting in Negadoctor seems very simple but that's because film in general handle specular highlights way better than digital.