Reading Group on Generative Models

Prompt: "Make an image with 10 people in the middle acting like explores and holding a red "Bristol" flag. Around them, I want a Llama, a Mamba, and a Transformer. Give it an Indiana Jones feel."

Past Papers

Some of the dates are probably incorrect because I still haven't learned how to read a calendar.

Title	Date	Topic
Nearly d-Linear Convergence Bounds for Diffusion Models via Stochastic Localization	25/08/2023	Diffusions
Score Approximation, Estimation and Distribution Recovery of Diffusion Models on Low-Dimensional Data	1/09/2023	Diffusions
Cold Diffusion: Inverting Arbitrary Image Transforms Without Noise	6/09/2023	Diffusions
Improving and generalizing flow-based generative models with minibatch optimal transport	11/09/2023	Flows
Simulation-free Schrödinger bridges via score and flow matching	15/09/2023	Flows
Trans-Dimensional Generative Modeling via Jump Diffusion Models	3/10/2023	Diffusion
Stochastic Interpolants: A Unifying Framework for Flows and Diffusions	5/10/2023	Diffusions, Flows
Convergence of denoising diffusion models under the manifold hypothesis	12/10/2023	Diffusions, Manifolds
Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion	17/10/2023	Diffusions, Flows
Building Normalizing Flows with Stochastic Interpolants and A Unifying Framework for Flows and Diffusions	19/10/2023	Diffusions
Optimal Transport in Systems and Control	24/10/2023	Optimal Transport
Particle Guidance: non-I.I.D. Diverse Sampling with Diffusion Models	27/10/2023	Diffusions
Generalization in diffusion models arises from geometry-adaptive harmonic representation	02/11/2023	Diffusions
ROBUST AND INTERPRETABLE BLIND IMAGE DENOISING VIA BIAS-FREE CONVOLUTIONAL NEURAL NETWORKS	06/11/2023	Diffusions
Diffusion Schrödinger Bridge Matching	14/11/2023	Diffusions
Chain of Log-Concave Markov Chains	21/11/2023	Log-Concave
Martingale posterior distributions	23/11/2023	Sampling
Multimeasurement Generative Models	05/12/2023	Generative Models
Deep Networks as Denoising Algorithms: Sample-Efficient Learning of Diffusion Models in High-Dimensional Graphical Models	11/01/2024	Diffusions
Analysis of learning a flow-based generative model from limited sample complexity	12/01/2024	Flows
Sampling with Mirrored Stein Operators	18/01/2024	Transformers
A mathematical perspective on Transformers	25/01/2024	Transformers
The emergence of clusters in self-attention dynamics	30/01/2024	Transformers
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding and The Illustrated BERT and Language Models are Few-Shot Learners and Tutorial 14 Transformers I	05/02/2024	Transformers
Dynamical Regimes of Diffusion Models	01/03/2024	Diffusions
Diffusive Gibbs Sampling	08/02/2024	Diffusions
Latent Attention for Linear Time Transformers	05/03/2024	Transformers
Implicit Diffusion: Efficient Optimization through Stochastic Sampling	07/03/2024	Diffusions
Efficiently Modeling Long Sequences with Structured State Spaces and How to Train Your HiPPO: State Space Models with Generalized Orthogonal Basis Projections	12/03/2024	Transformers
Formal Algorithms for Transformers	15/03/2024	Transformers
An Introduction to Transformers	19/03/2024	Transformers
Flow Matching for Generative Modelling	21/03/2024	Flows
RecurrentGemma: Moving Past Transformers for Efficient Open Language Models	16/04/2024	Transformers
Geometric Deep Learning Book - Chapter 1	19/04/2024	Geometric Deep Learning
The Kolmogorov–Arnold representation theorem revisited	26/04/2024	Theory of Neural Networks
Breaking the Curse of Dimensionality with Convex Neural Networks	30/04/2024	Theory of Neural Networks
Any-dimensional equivariant Neural Networks	02/05/2024	Theory of Neural Networks
On The Representation of Functions of Several Variables as a Superposition of Functions of a Smaller Number of Variables and Kolmogorov-Arnold Networks and On the Impact of the Activation Function on Deep Neural Networks Training	07/05/2024	Theory of Neural Networks
Continued On The Representation of Functions of Several Variables as a Superposition of Functions of a Smaller Number of Variables	09/05/2024	Theory of Neural Networks
Continued On The Representation of Functions of Several Variables as a Superposition of Functions of a Smaller Number of Variables	10/05/2024	Theory of Neural Networks
White-Box Transformers via Sparse Rate Reduction: Compression Is All There Is? and White-Box Transformers via Sparse Rate Reduction and Tutorial Lectures	21/05/2024	Transformers
Continued White-Box Transformers via Sparse Rate Reduction: Compression Is All There Is? and skimmed ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reduction and Segmentation of Multivariate Mixed Data via Lossy Data Coding and Compression	23/05/2024	Transformers
Scalable Optimization in the Modular Norm	28/05/2024	Stochastic Optimization
On a Neural Implementation of Brenier’s Polar Factorization	30/05/2024	Optimal Transport
On Amortizing Convex Conjugates for Optimal Transport	31/05/2024	Optimal Transport
Metric Flow Matching for Smooth Interpolations on the Data Manifold	06/06/2024	Flows
Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution	7/06/2024	Diffusions
Flow Map Matching	14/06/2024	Flows
Markovian Flow Matching: Accelerating MCMC with Continuous Normalizing Flows	18/06/2024	Flows
Continual Repeated Annealed Flow Transport Monte Carlo	20/06/2024	Flows
Differentiable Cost-Parameterized Monge Map Estimators	21/06/2024	Optimal Transport
Generative Modeling through the Semi-dual Formulation of Unbalanced Optimal Transport	25/06/2024	Optimal Transport
Stochastic Localization via Iterative Posterior Sampling	27/06/2024	Diffusions
Operator-informed score matching for Markov diffusion models	09/07/2024	Diffusions
Improving Diffusion Inverse Problem Solving with Decoupled Noise Annealing	11/07/2024	Diffusions

Additional Resources

Tutorials, videos, courses and notes that were shared at various times during the reading group.

• Attention is all you need - Lukasz Kaiser Masterclass
• Berkeley Deep Neural Networks Course and related video
• Stable Diffusions
• The Creativity of Text-to-Image Generation
• Deep Learning Foundations to Stable Diffusions - Jeremy Howard video
• The Illustrated Stable Diffusion
• Cross Attention augmented U-net
• The Illustrated GPT-2
• On the Expressivity Role of LayerNorm in Transformers' Attention
• The Annotated S4
• AlphaFold Meets Flow Matching for Generating Protein Ensembles
• Repo - Formal Algorithms for Transformers
• But what is a GPT? Visual intro to transformers - 3B1B
• Attention in Transformers, visually explained - 3B1B

How to start

1. Transformers
   • An Introduction to Transformers by Turner is perhaps the best way to start with attention, it is simple, to the point and gives you the "shape" of each array, which is helpful at first.
   • After Turner's paper, I would read A survey of transformers. There are typos but I found that this paper was helpful in giving a general overview: each paper implements them slightly differently and that made it very confusing for me.
   • You will likely still be confused after this paper, so I would recommend Formal Algorithm for Transformers.
   • I would then watch various of Andrey Karpathy's videos to consolidate understanding such as this and this and this.
2. Diffusions
   • To do

Suggesting new papers

You can suggest new papers here.