Translate TWS core algorithms to ImageJ Ops

[ctrueden]

We would like to create ops that perform pixel-wise classification based on input parameters—such as a list of features to compute, and initial training set—and producing an output ImgLabeling of the classification. Then the UI for TWS can be refactored to call the ops and produce the same results as before, but backed by the ops framework.

A related part of this work could be to also convert the feature computations to plugins, so that the available features become extensible. But this is a secondary project.

@iarganda How do you feel about @LeonYang5114 working on this project?
@dietzc Do you have specific ideas, preferences, etc., for how a set of pixel-wise classification ops would be structured?
@LeeKamentsky Does this sound like the direction you had in mind during our chat today?

[LeeKamentsky]

My preference would be to not have it be quite as monolithic - I'd like to see an Op that had an image as input and an image whose channel hyperplanes were class probabilities for each of the classes and perhaps that primitive could be fed into another to do the segmentation.

We usually break things into two pieces - a pixel classification step followed by a segmentation step. That lets you tailor the segmentation step to the shapes of the objects - generally, there's no one-size-fits-all for segmentation. So in CellProfiler, we'd turn the pixel classification op into a CellProfiler module and do our own segmentation.

I do like the idea of feeding in a feature bank. Some pixel classification problems require a lot of context, so you'd include filters with large kernels to capture more pixel context. Others can pick up on local edges or textures and so smaller kernels are sufficient.

[dietzc]

Hi all,

actually we (@EikeHeinz, @hornm and me) are currently working on several things related to this problem.

(a) We started reimplementing a subset of the features we have in trainable segmentation. Basically, all the filter based features. So, actually these features are just using existing ops. If the ops were missing, we implemented them (see e.g. imagej/imagej-ops#349 as a first PR of several)

(b) We discussed how we want to implement the learner and the predictor. We came up with with the following plan: The input for the learner are two n-dimensional RAIs. One of type CompositeType (or FeatureType if it doesn't work out with CompositeType) and one of type LabelingType for the class information. Basically, the learner will return a Computer which can map a n-dimensional RAI<CompositeType> to a n+1 dimensional RAI<FloatType> were the additional dimension reflects the class probabilities.

(c) it was very clear for us that the implementations in (b) should be independent from the underlying machine learning library, e.g. we don't want to restrict us to WEKA or so. even if we will use WEKA for our reference implementation.

(d) we can use all these "components" to compose whatever high-level-op we like and add some gui later.

Does this make sense?

Christian

[dietzc]

ah maybe one more thing to add to (b) there will be helpers which allow one to calculate arbitrary features on images (whatever we have), stack them as one RAI with n+1 dimensions and create a RAI<CompositeType> of it. It can all be done with Views.

[dietzc]

About postprocessing of probability images: so many ways to derive segmentations from them. Easiest would be to simply perform a max projection over the n+1 dimensional RAI ... Which btw could also be a n-d of CompositeType.

[iarganda]

Hello everybody!

Everything looks good to me.

@ctrueden I agree with the modularity proposed by @LeeKamentsky (first create probability maps and then labels) and it is true that features could have a way of being called as a mini-plugin or filter. All features should be identified on running time (for example by extending a specific class) so adding new features would be as easy as creating a new class.

If at least @hornm and @LeonYang5114 are going to work on this as well, maybe it would be a good idea to have a meeting by videoconference. What do you think?

[dietzc]

All features should be identified on running time (for example by extending a specific class) so adding new features would be as easy as creating a new class.

We named this interface 'PixelFeature'.

[hornm]

Synchronizing our efforts via an online-conference makes absolutely sense! Just for everyone's information: if everything works out as @iarganda and I planing it, I am going to work on porting the 'Trainable Segmentation' plugin (gui and algorithms) to ImageJ, Ops and KNIME in June for a month. However, I will hardly be able to work on that earlier. Hence, I am happy to continue and build on related work that has been done till then.

[rejsmont]

If I could pop into the discussion - an op that trains a classifier based on training image set and classification op that reuse the classifier would be great!