README.md

Deep Learning Models

A collection of various deep learning architectures, models, and tips for TensorFlow and PyTorch in Jupyter Notebooks.

Traditional Machine Learning

Title	Dataset	Description	Notebooks
Perceptron	2D toy data	TBD
Logistic Regression	2D toy data	TBD
Softmax Regression (Multinomial Logistic Regression)	MNIST	TBD
Softmax Regression with MLxtend's plot_decision_regions on Iris	Iris	TBD

Multilayer Perceptrons

Title	Dataset	Description	Notebooks
Multilayer Perceptron	MNIST	TBD
Multilayer Perceptron with Dropout	MNIST	TBD
Multilayer Perceptron with Batch Normalization	MNIST	TBD
Multilayer Perceptron with Backpropagation from Scratch	MNIST	TBD

Convolutional Neural Networks

Basic

Title	Dataset	Description	Notebooks
Convolutional Neural Network	TBD	TBD
CNN with He Initialization	TBD	TBD

Concepts

Title	Dataset	Description	Notebooks
Replacing Fully-Connected by Equivalent Convolutional Layers	TBD	TBD

---

AlexNet

Title	Dataset	Description	Notebooks
AlexNet Trained on CIFAR-10	TBD	TBD
AlexNet with Grouped Convolutions Trained on CIFAR-10	TBD	TBD

DenseNet

Title	Description	Daset	Notebooks
DenseNet-121 Digit Classifier Trained on MNIST	TBD	TBD
DenseNet-121 Image Classifier Trained on CIFAR-10	TBD	TBD

Fully Convolutional

Title	Dataset	Description	Notebooks
"All Convolutionl Net" -- A Fully Convolutional Neural Network	TBD	TBD

LeNet

Title	Dataset	Description	Notebooks
LeNet-5 on MNIST	TBD	TBD
LeNet-5 on CIFAR-10	TBD	TBD
LeNet-5 on QuickDraw	TBD	TBD

MobileNet

Title	Dataset	Description	Notebooks
MobileNet-v2 on Cifar-10	TBD	TBD
MobileNet-v3 small on Cifar-10	TBD	TBD
MobileNet-v3 large on Cifar-10	TBD	TBD
MobileNet-v3 large on MNIST via Embetter	TBD	TBD

Network in Network

Title	Dataset	Description	Notebooks
Network in Network Trained on CIFAR-10	TBD	TBD

VGG

Title	Dataset	Description	Notebooks
Convolutional Neural Network VGG-16 Trained on CIFAR-10	TBD	TBD
VGG-16 Smile Classifier	CelebA	TBD
VGG-16 Dogs vs Cats Classifier	TBD	TBD
Convolutional Neural Network VGG-19	TBD	TBD

ResNet

Title	Dataset	Description	Notebooks
ResNet and Residual Blocks	MNIST	TBD
ResNet-18 Digit Classifier	MNIST	TBD
ResNet-18 Gender Classifier	CelebA	TBD
ResNet-34 Digit Classifier	MNIST	TBD
ResNet-34 Object Classifier	QuickDraw	TBD
ResNet-34 Gender Classifier	CelebA	TBD
ResNet-50 Digit Classifier	MNIST	TBD
ResNet-50 Gender Classifier	CelebA	TBD
ResNet-101 Gender Classifier	CelebA	TBD
ResNet-101	CIFAR-10	TBD
ResNet-152 Gender Classifier	CelebA	TBD

---

Transformers

Title	Dataset	Description	Notebooks
Multilabel DistilBERT	Jigsaw Toxic Comment Challenge	DistilBERT classifier fine-tuning
DistilBERT as feature extractor	IMDB movie review	DistilBERT classifier with sklearn random forest and logistic regression
DistilBERT as feature extractor using embetter	IMDB movie review	DistilBERT classifier with sklearn random forest and logistic regression using the scikit-learn embetter library
Fine-tune DistilBERT I	IMDB movie review	Fine-tune only the last 2 layers of DistilBERT classifier
Fine-tune DistilBERT II	IMDB movie review	Fine-tune the whole DistilBERT classifier

---

Ordinal Regression and Deep Learning

Please note that the following notebooks below provide reference implementations to use the respective methods. They are not performance benchmarks.

Title	Dataset	Description	Notebooks
Baseline multilayer perceptron	Cement	A baseline multilayer perceptron for classification trained with the standard cross entropy loss
CORAL multilayer perceptron	Cement	Implementation of Rank Consistent Ordinal Regression for Neural Networks with Application to Age Estimation 2020
CORN multilayer perceptron	Cement	Implementation of Deep Neural Networks for Rank-Consistent Ordinal Regression Based On Conditional Probabilities 2022
Binary extension multilayer perceptron	Cement	Implementation of Ordinal Regression with Multiple Output CNN for Age Estimation 2016
Reformulated squared-error multilayer perceptron	Cement	Implementation of A simple squared-error reformulation for ordinal classification 2016
Class distance weighted cross-entropy loss	Cement	Implementation of Class Distance Weighted Cross-Entropy Loss for Ulcerative Colitis Severity Estimation 2022

---

Normalization Layers

Title	Dataset	Description	Notebooks
BatchNorm before and after Activation for Network-in-Network CIFAR-10 Classifier	TBD	TBD
Filter Response Normalization for Network-in-Network CIFAR-10 Classifier	TBD	TBD

Metric Learning

Title	Dataset	Description	Notebooks
Siamese Network with Multilayer Perceptrons	TBD	TBD

Autoencoders

Fully-connected Autoencoders

Title	Dataset	Description	Notebooks
Autoencoder (MNIST)	TBD	TBD
Autoencoder (MNIST) + Scikit-Learn Random Forest Classifier	TBD	TBD

Convolutional Autoencoders

Title	Dataset	Description	Notebooks
Convolutional Autoencoder with Deconvolutions / Transposed Convolutions	TBD	TBD
Convolutional Autoencoder with Deconvolutions and Continuous Jaccard Distance	TBD	TBD
Convolutional Autoencoder with Deconvolutions (without pooling operations)	TBD	TBD
Convolutional Autoencoder with Nearest-neighbor Interpolation	TBD	TBD
Convolutional Autoencoder with Nearest-neighbor Interpolation -- Trained on CelebA	TBD	TBD
Convolutional Autoencoder with Nearest-neighbor Interpolation -- Trained on Quickdraw	TBD	TBD

Variational Autoencoders

Title	Dataset	Description	Notebooks
Variational Autoencoder	TBD	TBD
Convolutional Variational Autoencoder	TBD	TBD

Conditional Variational Autoencoders

Title	Dataset	Description	Notebooks
Conditional Variational Autoencoder (with labels in reconstruction loss)	TBD	TBD
Conditional Variational Autoencoder (without labels in reconstruction loss)	TBD	TBD
Convolutional Conditional Variational Autoencoder (with labels in reconstruction loss)	TBD	TBD
Convolutional Conditional Variational Autoencoder (without labels in reconstruction loss)	TBD	TBD

Generative Adversarial Networks (GANs)

Title	Dataset	Description	Notebooks
Fully Connected GAN on MNIST	TBD	TBD
Fully Connected Wasserstein GAN on MNIST	TBD	TBD
Convolutional GAN on MNIST	TBD	TBD
Convolutional GAN on MNIST with Label Smoothing	TBD	TBD
Convolutional Wasserstein GAN on MNIST	TBD	TBD
Deep Convolutional GAN (DCGAN) on Cats and Dogs Images	TBD	TBD
Deep Convolutional GAN (DCGAN) on CelebA Face Images	TBD	TBD

Graph Neural Networks (GNNs)

Title	Dataset	Description	Notebooks
Most Basic Graph Neural Network with Gaussian Filter on MNIST	TBD	TBD
Basic Graph Neural Network with Edge Prediction on MNIST	TBD	TBD
Basic Graph Neural Network with Spectral Graph Convolution on MNIST	TBD	TBD

Recurrent Neural Networks (RNNs)

Many-to-one: Sentiment Analysis / Classification

Title	Dataset	Description	Notebooks
A simple single-layer RNN (IMDB)	TBD	TBD
A simple single-layer RNN with packed sequences to ignore padding characters (IMDB)	TBD	TBD
RNN with LSTM cells (IMDB)	TBD	TBD
RNN with LSTM cells (IMDB) and pre-trained GloVe word vectors	TBD	TBD
RNN with LSTM cells and Own Dataset in CSV Format (IMDB)	TBD	TBD
RNN with GRU cells (IMDB)	TBD	TBD
Multilayer bi-directional RNN (IMDB)	TBD	TBD
Bidirectional Multi-layer RNN with LSTM with Own Dataset in CSV Format (AG News)	TBD	TBD

Many-to-Many / Sequence-to-Sequence

Title	Dataset	Description	Notebooks
A simple character RNN to generate new text (Charles Dickens)	TBD	TBD

Model Evaluation

K-Fold Cross-Validation

Title	Dataset	Description	Notebooks
Baseline CNN	MNIST	A simple baseline with traditional train/validation/test splits
K-fold with pl_cross	MNIST	A 5-fold cross-validation run using the pl_cross library

Data Augmentation

Title	Dataset	Description	Notebooks
AutoAugment & TrivialAugment for Image Data	CIFAR-10	Trains a ResNet-18 using AutoAugment and TrivialAugment

Tips and Tricks

Title	Dataset	Description	Notebooks
Cyclical Learning Rate	TBD	TBD
Annealing with Increasing the Batch Size (w. CIFAR-10 & AlexNet)	TBD	TBD
Gradient Clipping (w. MLP on MNIST)	TBD	TBD

Transfer Learning

Title	Dataset	Description	Notebooks
Transfer Learning Example (VGG16 pre-trained on ImageNet for Cifar-10)	TBD	TBD

Visualization and Interpretation

Title	Dataset	Description	Notebooks
Vanilla Loss Gradient (wrt Inputs) Visualization (Based on a VGG16 Convolutional Neural Network for Kaggle's Cats and Dogs Images)	TBD	TBD
Guided Backpropagation (Based on a VGG16 Convolutional Neural Network for Kaggle's Cats and Dogs Images)	TBD	TBD

PyTorch Workflows and Mechanics

PyTorch Lightning Examples

Title	Dataset	Description	Notebooks
MLP in Lightning with TensorBoard -- continue training the last model	TBD	TBD
MLP in Lightning with TensorBoard -- checkpointing best model	TBD	TBD

Custom Datasets

Title	Dataset	Description	Notebooks
Custom Data Loader Example for PNG Files	TBD	TBD
Using PyTorch Dataset Loading Utilities for Custom Datasets -- CSV files converted to HDF5	TBD	TBD
Using PyTorch Dataset Loading Utilities for Custom Datasets -- Face Images from CelebA	TBD	TBD
Using PyTorch Dataset Loading Utilities for Custom Datasets -- Drawings from Quickdraw	TBD	TBD
Using PyTorch Dataset Loading Utilities for Custom Datasets -- Drawings from the Street View House Number (SVHN) Dataset	TBD	TBD
Using PyTorch Dataset Loading Utilities for Custom Datasets -- Asian Face Dataset (AFAD)	TBD	TBD
Using PyTorch Dataset Loading Utilities for Custom Datasets -- Dating Historical Color Images	TBD	TBD
Using PyTorch Dataset Loading Utilities for Custom Datasets -- Fashion MNIST	TBD	TBD

Training and Preprocessing

Title	Dataset	Description	Notebooks
PyTorch DataLoader State and Nested Iterations	Toy	Explains DataLoader behavior when in nested functions
Generating Validation Set Splits	TBD	TBD
Dataloading with Pinned Memory	TBD	TBD
Standardizing Images	TBD	TBD
Image Transformation Examples	TBD	TBD
Char-RNN with Own Text File	TBD	TBD
Sentiment Classification RNN with Own CSV File	TBD	TBD

Improving Memory Efficiency

Title	Dataset	Description	Notebooks
Gradient Checkpointing Demo (Network-in-Network trained on CIFAR-10)	TBD	TBD

Parallel Computing

Title	Description	Notebooks
Using Multiple GPUs with DataParallel -- VGG-16 Gender Classifier on CelebA	TBD
Distribute a Model Across Multiple GPUs with Pipeline Parallelism (VGG-16 Example)	TBD

Other

Title	Dataset	Description	Notebooks
PyTorch with and without Deterministic Behavior -- Runtime Benchmark	TBD	TBD
Sequential API and hooks	TBD	TBD
Weight Sharing Within a Layer	TBD	TBD
Plotting Live Training Performance in Jupyter Notebooks with just Matplotlib	TBD	TBD

Autograd

Title	Dataset	Description	Notebooks
Getting Gradients of an Intermediate Variable in PyTorch	TBD	TBD

TensorFlow Workflows and Mechanics

Custom Datasets

Title	Description	Notebooks
Chunking an Image Dataset for Minibatch Training using NumPy NPZ Archives	TBD
Storing an Image Dataset for Minibatch Training using HDF5	TBD
Using Input Pipelines to Read Data from TFRecords Files	TBD
Using Queue Runners to Feed Images Directly from Disk	TBD
Using TensorFlow's Dataset API	TBD

Training and Preprocessing

Title	Dataset	Description	Notebooks
Saving and Loading Trained Models -- from TensorFlow Checkpoint Files and NumPy NPZ Archives	TBD	TBD

Related Libraries

Title	Description	Notebooks
TorchMetrics	How do we use it, and what's the difference between .update() and .forward()?