A Broad Review of ML Topics

This repository will be used to help me consolidate my knowledge regarding various ML topics.

This repository will include:

• Notes on various topics (Edit: Note that markdown on Github isn't formatting the same as my local IDE, PyCharm, which may cause readability issues!)
  • Classical (Non-DL) ML/Statistics
    • Linear Algebra and Calculus
    • Probability and Information Theory
    • Statistical Learning Theory
    • Statistical Testing and Metrics
    • Bayesian Statistics
    • Linear Regression & Regularization
    • Naive Bayes & Logistic Regression & GLMs
    • SVMs
    • Decision Trees
    • Ensemble Learning, Random Forests and Boosting
    • Dimensionality Reduction
    • Unsupervised Clustering
    • Gaussian Process
    • Causal Inference
    • ARIMA
  • DL (Note that some topics may bleed into each category)
    • Basics
    • Activation Functions
    • Initialization
    • Optimization and Regularization
    • Coding Practices
    • CNNs
    • RNNs
    • Attention & Transformers
    • Autoencoders
    • Diffusion Models
    • Flow-based Models
    • Generative Adversarial Networks
    • Graph Neural Networks
    • Meta-Learning
    • Self-Supervised Contrastive Learning
    • Computer Vision
    • Natural Language Processing
    • Reinforcement Learning
    • Audio
    • Video
    • Multimodal
    • Post Training
    • AI Safety
    • Hyperparameter Optimization
    • Computational Performance
    • Personal Projects
    • Misc
• Code implementations for various algorithms, which will mostly come from online resources/tutorials.
  • The first priority would be to fulfill learning goals.
  • If time permits, a stretch goal would be to refactor the code with a greater emphasis on OOP, e.g. John's repo
  • Code currently includes:
    • From scratch implementations, including BERT, GPT-2, Llama 2-3.2, DDPM, Real-NVP.
    • Post-training experiments, including (LoRA) fine-tuning and DPO.
    • Data and model parallelism, with and without JAX (+FLAX).
    • Experiments with TensorRT-LLM for model serving
• Interview Preparation

I shall try to be diligent in citing my sources. Due to visa-related time constraints, I do apologize for any lapses in citation. At the current moment (1/10/25), I have pulled most heavily from the following sources:

• UvA Deep Learning Tutorials by Lippe
• Notes / Code from classes I took at Duke.
• Hands-On Machine Learning by Aurélien Géron
• Dive into Deep Learning by Zhang, Lipton, Li and Smola
• Build a Large Language Model (From Scratch) by Raschka
• Hugging Face's Diffusion Course
• Deep Learning by Goodfellow, Bengio and Courville
• Lilian Weng's Blog
• Anthropic's Research
• System Design by Hello Interview
• Designing Machine Learning Systems by Huyen
• Machine Learning System Design Interview by Aminian and Xu
• Machine Learning System Design Interview by Pham
• System Design Interview by Xu