Machine Learning

Machine Learning

Basics

• Gradient Descent
• Regularization
• Cross-Validation
• Hyperparameter Tuning
• Principal Components Analysis

Models

• Linear Regression
• Generalized Linear Models
• Logistic Regression
• Naive Bayes
• Support Vector Machines
• Decision Trees
• Random Forest
• K-Means Clustering
  • Hierarchical Clustering
  • Density Clustering
• Gaussian Mixture Model

Deep Learning Models

• NN Engineering
• Multilayer Perceptrons (Feedforward Linear Neural Networks)
• Convolutional Neural Networks
  • R-CNN
  • UNet
  • ResNet
• Sequence Models
  • Recurrent Neural Networks
    • LSTM
    • GRU
    • Seq2Seq
  • Transformers
    • Vision Transformers
    • Decoder-Only Transformers
• Foundation Models (Mostly LLMs)
  • Foundations
    • Tokenization
    • Positional Encoding
    • LLM Architecture and Training Design Choices
    • Mixture-of-Experts
  • Finetuning
    • Instruct Fine-Tuning
    • LoRA
    • RLHF
  • Applications
    • Retrieval-Augmented Generation
    • “Agents”
    • Model Context Protocol (MCP)
  • Optimizations
    • Flash Attention
    • Sliding Window Attention
    • Ring Attention
    • Structured State Space Models / Mamba
  • Systems
    • Distributed Training
• Generative Modeling
  • Variational Autoencoders
  • GANs
    • Conditional GANs
  • Diffusion Models
• Graph Neural Networks

Primitives (from scratch)

Atomic notes, each with the equation and a short from-scratch PyTorch implementation. Full list: Primitives index.

• Normalization: GroupNorm, InstanceNorm, RMSNorm, QK-Norm, Adaptive LayerNorm, Normalization Comparison
• Activations: ReLU, GELU, Swish, SwiGLU, Softmax, Log Softmax
• Losses: Cross-Entropy Loss, Focal Loss, InfoNCE Loss, Triplet Loss, Knowledge Distillation Loss, Label Smoothing
• Attention: Scaled Dot-Product Attention, Multi-Head Attention, Grouped-Query Attention, Multi-Head Latent Attention, KV Cache Implementation, Rotary Positional Embeddings, Transformer Block
• MoE and adaptation: MoE Routing, MoE Load Balancing Loss, LoRA, QLoRA
• Generative: DDPM, DDIM Sampling, Noise Schedules, Classifier-Free Guidance, Flow Matching, Score Matching, VQ-VAE, VAE ELBO
• Optimization: AdamW, SGD with Momentum, RMSProp, Learning Rate Schedules, Gradient Clipping, Weight Initialization
• Framework internals: Autograd Engine, Manual Backprop Rules, Custom autograd Function, Broadcasting and Reduction, Tensor Layout and Strides, Straight-Through Estimator

Applications

Computer Vision

• NeRF
• Gaussian Splatting