update monai version

Author: davidiommi

README.md

@@ -1,11 +1,12 @@
 ![Salmon-logo-1](images/salmon.JPG)
 # SALMON v.2: Segmentation deep learning ALgorithm based on MONai toolbox
 - SALMON is a computational toolbox for segmentation using neural networks (3D patches-based segmentation)
-- SALMON is based on  NN-UNET and MONAI: PyTorch-based, open-source frameworks for deep learning in healthcare imaging. 
+- SALMON is based on MONAI 0.7.0 : PyTorch-based, open-source frameworks for deep learning in healthcare imaging. 
 (https://github.com/Project-MONAI/MONAI)
 (https://github.com/MIC-DKFZ/nnUNet)
+(https://arxiv.org/abs/2103.10504)
 
-This is my "open-box" version if I want to modify the parameters for some particular task, while the two above are hard-coded.
+This is my "open-box" version if I want to modify the parameters for some particular task, while the two above are hard-coded. The monai 0.5.0 folder contains the previous versions based on the old monai version.
 
 *******************************************************************************
 ## Requirements
@@ -20,7 +21,7 @@ Labels are resampled and resized to the corresponding image, to avoid array size
 
 - check_loader_patches: Shows example of patches fed to the network during the training.  
 
-- networks.py: The architecture available for segmentation is a nn-Unet.
+- networks.py: The architectures available for segmentation are nn-Unet and UneTR (based on Visual transformers)
 
 - train.py: Runs the training
 
@@ -93,6 +94,7 @@ with same size and resolution of the source image.
 ### Tips:
 - Use and modify "check_loader_patches.py" to check the patches fed during training. 
 - The "networks.py" calls the nn-Unet, which adapts itself to the input data (resolution and patches size). The script also saves the graph of you network, so you can visualize it. 
+- "networks.py" includes also UneTR (based on Visual transformers). This is experimental. For more info check (https://arxiv.org/abs/2103.10504) and https://github.com/Project-MONAI/tutorials/blob/master/3d_segmentation/unetr_btcv_segmentation_3d.ipynb
 - Is it possible to add other networks, but for segmentation the U-net architecture is the state of the art.
 
 ### Sample script inference
@@ -108,9 +110,27 @@ The example segment the prostate (1 channel input) in the transition zone and pe
 The gif files with some example images are shown above.
 
 Some note:
-- You must add an additional channel for the background. Example: 0 background, 1 prostate, 2 prostate tumor = 3 out channels in total.
 - Tensorboard can show you all segmented channels, but for now the metric is the Mean-Dice (of all channels). If you want to evaluate the Dice score for each channel you 
   have to modify a bit the plot_dice function. I will do it...one day...who knows...maybe not
 - The loss is the DiceLoss + CrossEntropy. You can modify it if you want to try others (https://docs.monai.io/en/latest/losses.html#diceloss)
 
-Check more examples at https://github.com/Project-MONAI/tutorials/blob/master/3d_segmentation/spleen_segmentation_3d.ipynb.
+Check more examples at https://github.com/Project-MONAI/tutorials/blob/master/3d_segmentation/.
+
+UneTR Notes from the authors:
+
+Feature_size and pos_embed are the parameters that need to changed to adopt it for your application of interest. Other parameters that are mentioned come from Vision Transformer (ViT) default hyper-parameters (original architecture). In addition, the new revision of UNETR paper with more descriptions is now publicly available. Please check for more details:
+https://arxiv.org/pdf/2103.10504.pdf
+
+Now let's look at each of these hyper-parameters in the order of importance:
+
+feature_size : In UNETR, we multiply the size of the CNN-based features in the decoder by a factor of 2 at every resolution ( just like the original UNet paper). By default, we set this value to 16 ( to make the entire network lighter). However using larger values such as 32 can improve the segmentation performance if GPU memory is not an issue. Figure2 of the paper also shows this in details.
+
+pos_embed: this determines how the image is divided into non-overlapping patches. Essentially, there are 2 ways to achieve this ( by setting it to conv or perceptron). Let's further dive into it for more information:
+First is by directly applying a convolutional layer with the same stride and kernel size of the patch size and with feature size of the hidden size in the ViT model. Second is by first breaking the image into patches by properly resizing the tensor ( for which we use einops) and then feed it into a perceptron (linear) layer with a hidden size of the ViT model. Our experiments show that for certain applications such as brain segmentation with multiple modalities (e.g. 4 modes such as T1,T2 etc.), using the convolutional layer works better as it takes into account all modes concurrently. For CT images ( e.g. BTCV multi-organ segmentation), we did not see any difference in terms of performance between these two approaches.
+
+hidden_size : this is the size of the hidden layers in the ViT encoder. We follow the original ViT model and set this value to 768. In addition, the hidden size should be divisible by the number of attention heads in the ViT model.
+
+num_heads : in the multi-headed self-attention block, this is the number of attention heads. Following the ViT architecture, we set it to 12.
+
+mlp_dim : this is the dimension of the multi-layer perceptrons (MLP) in the transformer encoder. Again, we follow the ViT model and set this to 3072 as default value to be consistent with their architecture.
+

check_loader_patches.py

@@ -75,7 +75,7 @@ def plot3d(image):
 
         LoadImaged(keys=['image', 'label']),
         AddChanneld(keys=['image', 'label']),
-        Orientationd(keys=["image", "label"], axcodes="RAS"),
+        # Orientationd(keys=["image", "label"], axcodes="RAS"),
         # ThresholdIntensityd(keys=['image'], threshold=-135, above=True, cval=-135),
         # ThresholdIntensityd(keys=['image'], threshold=215, above=False, cval=215),
         CropForegroundd(keys=['image', 'label'], source_key='image', start_coord_key='foreground_start_coord',

init.py

@@ -17,12 +17,13 @@ def initialize(self, parser):
         parser.add_argument('--labels_folder', type=str, default='./Data_folder/labels')
         parser.add_argument('--increase_factor_data',  default=1, help='Increase data number per epoch')
         parser.add_argument('--preload', type=str, default=None)
-        parser.add_argument('--gpu_ids', type=str, default='0,1', help='gpu ids: e.g. 0  0,1,2, 0,2. use -1 for CPU')
+        parser.add_argument('--gpu_ids', type=str, default='2,3', help='gpu ids: e.g. 0  0,1,2, 0,2. use -1 for CPU')
         parser.add_argument('--workers', default=8, type=int, help='number of data loading workers')
 
         # dataset parameters
-        parser.add_argument('--patch_size', default=(160, 160, 32), help='Size of the patches extracted from the image')
-        parser.add_argument('--spacing', default=[2.25, 2.25, 3], help='Original Resolution')
+        parser.add_argument('--network', default='unetr', help='nnunet, unetr')
+        parser.add_argument('--patch_size', default=(256, 256, 16), help='Size of the patches extracted from the image')
+        parser.add_argument('--spacing', default=[0.7, 0.7, 3], help='Original Resolution')
         parser.add_argument('--resolution', default=None, help='New Resolution, if you want to resample the data in training. I suggest to resample in organize_folder_structure.py, otherwise in train resampling is slower')
         parser.add_argument('--batch_size', type=int, default=4, help='batch size, depends on your machine')
         parser.add_argument('--in_channels', default=1, type=int, help='Channels of the input')
@@ -31,6 +32,7 @@ def initialize(self, parser):
         # training parameters
         parser.add_argument('--epochs', default=1000, help='Number of epochs')
         parser.add_argument('--lr', default=0.01, help='Learning rate')
+        parser.add_argument('--benchmark', default=True)
 
         self.initialized = True
         return parser

installation_commands_.txt

@@ -5,7 +5,7 @@
 2) install pytorch    conda install pytorch==1.5.0 torchvision==0.6.0 cudatoolkit=10.1 -c pytorch    # this for cuda (check your cuda version)
 
 3)    conda install git pip
-4)    pip install git+git://github.com/davidiommi/MONAI    # dowload libraries
+4)    pip install git+https://github.com/davidiommi/MONAI_0_7_0.git    # dowload libraries
 5)    pip install -r requirements.txt                      # dowload libraries
 
 

previous_version/LICENSE monai 0.5.0/LICENSE

@@ -0,0 +1,116 @@
+![Salmon-logo-1](images/salmon.JPG)
+# SALMON v.2: Segmentation deep learning ALgorithm based on MONai toolbox
+- SALMON is a computational toolbox for segmentation using neural networks (3D patches-based segmentation)
+- SALMON is based on  NN-UNET and MONAI: PyTorch-based, open-source frameworks for deep learning in healthcare imaging. 
+(https://github.com/Project-MONAI/MONAI)
+(https://github.com/MIC-DKFZ/nnUNet)
+
+This is my "open-box" version if I want to modify the parameters for some particular task, while the two above are hard-coded.
+
+*******************************************************************************
+## Requirements
+Follow the steps in "installation_commands.txt". Installation via Anaconda and creation of a virtual env to download the python libraries and pytorch/cuda.
+*******************************************************************************
+## Python scripts and their function
+
+- organize_folder_structure.py: Organize the data in the folder structure (training,validation,testing) for the network. 
+Labels are resampled and resized to the corresponding image, to avoid array size conflicts. You can set here a new image resolution for the dataset. 
+
+- init.py: List of options used to train the network. 
+
+- check_loader_patches: Shows example of patches fed to the network during the training.  
+
+- networks.py: The architecture available for segmentation is a nn-Unet.
+
+- train.py: Runs the training
+
+- predict_single_image.py: It launches the inference on a single input image chosen by the user.
+*******************************************************************************
+## Usage
+### Folders structure:
+
+Use first "organize_folder_structure.py" to create organize the data.
+Modify the input parameters to select the two folders: images and labels folders with the dataset. Set the resolution of the images here before training.
+
+    .
+	├── Data_folder                   
+	|   ├── CT               
+	|   |   ├── 1.nii 
+    |   |   ├── 2.nii 	
+	|   |   └── 3.nii                     
+	|   ├── CT_labels                         
+	|   |   ├── 1.nii 
+    |   |   ├── 2.nii 	
+	|   |   └── 3.nii  
+
+Data structure after running it:
+
+	.
+	├── Data_folder  
+	|   ├── CT  
+	|   ├── CT_labels 
+	|   ├── images              
+	|   |   ├── train             
+	|   |   |   ├── image1.nii              
+	|   |   |   └── image2.nii                     
+	|   |   └── val             
+	|   |   |   ├── image3.nii             
+	|   |   |   └── image4.nii
+	|   |   └── test             
+	|   |   |   ├── image5.nii              
+	|   |   |   └── image6.nii
+	|   ├── labels              
+	|   |   ├── train             
+	|   |   |   ├── label1.nii              
+	|   |   |   └── label2.nii                     
+	|   |   └── val             
+	|   |   |   ├── label3.nii             
+	|   |   |   └── label4.nii
+	|   |   └── test             
+	|   |   |   ├── label5.nii              
+	|   |   |   └── label6.nii
+	
+*******************************************************************************
+### Training:
+- Modify the "init.py" to set the parameters and start the training/testing on the data. Read the descriptions for each parameter.
+- Afterwards launch the "train.py" for training. Tensorboard is available to monitor the training ("runs" folder created)	
+- Check and modify the train_transforms applied to the images  in "train.py" for your specific case. (e.g. In the last update there is a HU windowing for CT images)
+
+Sample images: the following images show the segmentation of carotid artery from MRI sequence
+
+![Image](images/image.gif)![result](images/result.gif)
+
+Sample images: the following images show the multi-label segmentation of prostate transition zone and peripheral zone from MRI sequence
+
+![Image1](images/prostate.gif)![result1](images/prostate_inf.gif)!
+
+*******************************************************************************
+### Inference:
+- Launch "predict_single_image.py" to test the network. Modify the parameters in the parse section to select the path of the weights, images to infer and result. 
+- You can test the model on a new image, with different size and resolution from the training. The script will resample it before the inference and give you a mask
+with same size and resolution of the source image.
+*******************************************************************************
+### Tips:
+- Use and modify "check_loader_patches.py" to check the patches fed during training. 
+- The "networks.py" calls the nn-Unet, which adapts itself to the input data (resolution and patches size). The script also saves the graph of you network, so you can visualize it. 
+- Is it possible to add other networks, but for segmentation the U-net architecture is the state of the art.
+
+### Sample script inference
+- The label can be omitted (None) if you segment an unknown image. You have to add the --resolution if you resampled the data during training (look at the argsparse in the code).
+```console
+python predict_single_image.py --image './Data_folder/image.nii' --label './Data_folder/label.nii' --result './Data_folder/prova.nii' --weights './best_metric_model.pth'
+```
+*******************************************************************************
+### Multi-channel segmentation: 
+
+The subfolder "multi_label_segmentation_example" include the modified code for multi_labels scenario.
+The example segment the prostate (1 channel input) in the transition zone and peripheral zone (2 channels output). 
+The gif files with some example images are shown above.
+
+Some note:
+- You must add an additional channel for the background. Example: 0 background, 1 prostate, 2 prostate tumor = 3 out channels in total.
+- Tensorboard can show you all segmented channels, but for now the metric is the Mean-Dice (of all channels). If you want to evaluate the Dice score for each channel you 
+  have to modify a bit the plot_dice function. I will do it...one day...who knows...maybe not
+- The loss is the DiceLoss + CrossEntropy. You can modify it if you want to try others (https://docs.monai.io/en/latest/losses.html#diceloss)
+
+Check more examples at https://github.com/Project-MONAI/tutorials/blob/master/3d_segmentation/spleen_segmentation_3d.ipynb.

monai 0.5.0/README.md

@@ -0,0 +1,106 @@
+# Copyright 2020 MONAI Consortium
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#     http://www.apache.org/licenses/LICENSE-2.0
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import sys
+from glob import glob
+import tempfile
+import numpy as np
+import matplotlib.pyplot as plt
+import nibabel as nib
+import torch
+from torch.utils.data import DataLoader
+from init import Options
+import monai
+from monai.data import ArrayDataset, GridPatchDataset, create_test_image_3d
+from monai.transforms import (Compose, LoadImaged, AddChanneld, Transpose, Resized, CropForegroundd, CastToTyped,RandGaussianSmoothd,
+                              ScaleIntensityd, ToTensord, RandSpatialCropd, Rand3DElasticd, RandAffined, SpatialPadd,
+    Spacingd, Orientationd, RandZoomd, ThresholdIntensityd, RandShiftIntensityd, RandGaussianNoised, BorderPadd,RandAdjustContrastd, NormalizeIntensityd,RandFlipd, ScaleIntensityRanged)
+
+
+class IndexTracker(object):
+    def __init__(self, ax, X):
+        self.ax = ax
+        ax.set_title('use scroll wheel to navigate images')
+
+        self.X = X
+        rows, cols, self.slices = X.shape
+        self.ind = self.slices//2
+
+        self.im = ax.imshow(self.X[:, :, self.ind],cmap= 'gray')
+        self.update()
+
+    def onscroll(self, event):
+        print("%s %s" % (event.button, event.step))
+        if event.button == 'up':
+            self.ind = (self.ind + 1) % self.slices
+        else:
+            self.ind = (self.ind - 1) % self.slices
+        self.update()
+
+    def update(self):
+        self.im.set_data(self.X[:, :, self.ind])
+        self.ax.set_ylabel('slice %s' % self.ind)
+        self.im.axes.figure.canvas.draw()
+
+
+def plot3d(image):
+    original=image
+    original = np.rot90(original, k=-1)
+    fig, ax = plt.subplots(1, 1)
+    tracker = IndexTracker(ax, original)
+    fig.canvas.mpl_connect('scroll_event', tracker.onscroll)
+    plt.show()
+
+
+if __name__ == "__main__":
+
+    opt = Options().parse()
+
+    train_images = sorted(glob(os.path.join(opt.images_folder, 'train', 'image*.nii')))
+    train_segs = sorted(glob(os.path.join(opt.labels_folder, 'train', 'label*.nii')))
+
+    data_dicts = [{'image': image_name, 'label': label_name}
+                  for image_name, label_name in zip(train_images, train_segs)]
+
+    monai_transforms = [
+
+        LoadImaged(keys=['image', 'label']),
+        AddChanneld(keys=['image', 'label']),
+        Orientationd(keys=["image", "label"], axcodes="RAS"),
+        # ThresholdIntensityd(keys=['image'], threshold=-135, above=True, cval=-135),
+        # ThresholdIntensityd(keys=['image'], threshold=215, above=False, cval=215),
+        CropForegroundd(keys=['image', 'label'], source_key='image', start_coord_key='foreground_start_coord',
+                        end_coord_key='foreground_end_coord', ),  # crop CropForeground
+        NormalizeIntensityd(keys=['image']),
+        ScaleIntensityd(keys=['image']),
+        # Spacingd(keys=['image', 'label'], pixdim=opt.resolution, mode=('bilinear', 'nearest')),
+
+        SpatialPadd(keys=['image', 'label'], spatial_size=opt.patch_size, method= 'end'),
+        RandSpatialCropd(keys=['image', 'label'], roi_size=opt.patch_size, random_size=False),
+        ToTensord(keys=['image', 'label','foreground_start_coord', 'foreground_end_coord'],)
+    ]
+
+    transform = Compose(monai_transforms)
+    check_ds = monai.data.Dataset(data=data_dicts, transform=transform)
+    loader = DataLoader(check_ds, batch_size=1, shuffle=True, num_workers=0, pin_memory=torch.cuda.is_available())
+    check_data = monai.utils.misc.first(loader)
+    im, seg, coord1, coord2 = (check_data['image'][0], check_data['label'][0],check_data['foreground_start_coord'][0],
+                      check_data['foreground_end_coord'][0])
+
+    print(im.shape, seg.shape, coord1, coord2)
+
+    vol = im[0].numpy()
+    mask = seg[0].numpy()
+
+    print(vol.shape, mask.shape)
+    plot3d(vol)
+    plot3d(mask)