Reference: Algorithm Subcategory Map

A single tree showing the family relationships between every supervised-learning algorithm covered in this article set. Each leaf points to the article that covers it. Linked from each article’s 📚 References toggle.

The map

Algorithms
│
├── Supervised (labelled y)
│   │
│   ├── Tree-based  ─── decisions encoded as a flowchart of yes/no splits
│   │   ├── Decision Tree ............. classical-ml-foundations-p05
│   │   ├── Random Forest (including vs. Boosting comparison) ... classical-ml-foundations-p06
│   │   ├── Gradient Boosting (XGBoost, LightGBM, CatBoost) ......... classical-ml-foundations-p04
│   │   └── Stacking / Ensemble ....... classical-ml-foundations-p12
│   │
│   ├── Linear  ─── one weighted sum per output
│   │   ├── Linear Regression ......... classical-ml-foundations-p02 (gradient descent), causal-* (backdoor)
│   │   ├── Logistic Regression ....... causal-inference-p01 (the email), causal-inference-p06 (propensity score)
│   │   └── Regularized (L1 Lasso / L2 Ridge) ..... classical-ml-foundations-p03
│   │
│   ├── Distance-based  ─── "find me the K nearest neighbours and vote"
│   │   └── KNN ...................... classical-ml-foundations-p08  (curse of dimensionality lives here)
│   │
│   ├── Probabilistic  ─── models P(feature | class)·P(class) as the joint, then inverts via Bayes' rule for the class posterior
│   │   └── Naive Bayes .............. classical-ml-foundations-p09
│   │
│   ├── Kernel  ─── linear-in-a-higher-space
│   │   └── SVM (Support Vector Machine) ..... classical-ml-foundations-p07
│   │
│   ├── Time-series  ─── y is itself, indexed by time, with autocorrelation
│   │   ├── ARIMA / SARIMA ........... arima-and-sarima-intuitively-when-classical-foreca
│   │   ├── Classical vs ML .......... classical-forecasting-vs-machine-learning-when-doe
│   │   ├── Prophet .................. prophet-vs-statistical-models-a-practical-forecast
│   │   ├── Gradient Boosting for time .... gradient-boosting-for-time-series-using-lightgbm-t
│   │   └── Cross-validation for time ...... time-series-cross-validation-why-k-fold-breaks-on
│   │
│   └── Deep Learning  ─── many stacked layers, learned representations
│       ├── Single Neuron / Perceptron ........ attention-from-scratch-p01
│       ├── Backpropagation ........... attention-from-scratch-p02
│       ├── CNN (image / spatial) ...... attention-from-scratch-p04
│       ├── RNN (sequence), LSTM, GRU .. attention-from-scratch-p05, attention-from-scratch-p06
│       ├── Attention mechanism ........ attention-from-scratch-p07, attention-from-scratch-p08
│       ├── The full Transformer ...... attention-from-scratch-p09
│       ├── Transfer learning .......... attention-from-scratch-p10
│       ├── Regularization (BatchNorm, Dropout) ........ attention-from-scratch-p11
│       └── Debug training ............. attention-from-scratch-p12
│
├── Unsupervised (no y)  ─── out of scope for this set
│   ├── Clustering (K-means, hierarchical, HDBSCAN)
│   └── Dimensionality reduction (PCA, t-SNE, UMAP)
│
└── Reinforcement Learning  ─── out of scope for this set

One-line definitions per subfamily

• Tree-based: splits the feature space with yes/no rules; predictions are averages (regression) or majority votes (classification) at the leaves. Interpretable, handles mixed types, doesn’t need scaling.
• Linear: one weighted sum per output. Fast, interpretable — fits when you need coefficient interpretability or have few informative features. Religious about scaling for feature comparison.
• Distance-based: predicts by looking up the K closest training rows and voting. Non-parametric; needs scale-normalized features; dies in high dimensions.
• Probabilistic: models P(class | features) via Bayes’ rule with a “naive” independence assumption. Cheap, fast, surprisingly strong on text.
• Kernel: linear classification in a high-dimensional space the data is lifted into via a kernel function. SVM is the flagship; the widest-margin intuition is the concept.
• Time-series: the order of rows matters. Standard CV breaks; special walk-forward CV applies. Roughly splits into statistical (ARIMA, Prophet) and ML gradient-boosting approaches.
• Deep Learning: many stacked layers learn a representation automatically. The choice between CNN (spatial), RNN (sequence), and Transformer (attention) is the first big fork.

The big forks (the call when deciding where to start)

1. Is y labelled? No → unsupervised (out of scope here). Yes → supervised.
2. If labelled, is y structured by time? Yes → time-series subfamily. No → flat supervised.
3. If flat supervised, how much data, what shape are the features?
   • Small-ish, tabular, mixed types, non-linear interactions → tree (Random Forest or XGBoost) — this shape of data is where trees fit without extra preprocessing.
   • Few informative features, want interpretability → linear / regularized linear.
   • Many features, want to rank by importance still → gradient boosting.
   • Small-to-medium samples, high-dimensional features, a margin between classes matters → kernel / SVM.
   • Low-dimensional features, a meaningful distance metric, no training budget → KNN.
   • High-dimensional sparse counts (especially text), and the independence assumption is tolerable → naive Bayes.
   • Image / audio / very-long-text → deep learning (CNN for images, Transformer for text).
4. If deep, is the input spatial (img), sequential (text/time), or both? spatial → CNN; sequential → RNN/LSTM or Transformer. Transformers fit when sequences are short enough for quadratic attention and the compute is affordable; recurrent models still fit streaming inference, very long sequences under memory limits, or small data.

Sensible couplings: metric ↔ algorithm ↔ problem

Problem	Algorithm family	Default metric	Why
Binary classification, balanced	any	accuracy / F1	both classes matter equally
Binary classification, imbalanced	any	recall-at-fixed-precision + PR-AUC	accuracy lies; the rare class is the point
Probabilistic decision (insurance, loan)	calibrated	log-loss + calibration curve	downstream uses the probability itself
Regression, outliers present	linear/tree	MAE and MAPE	MAE is robust; MSE would chase outliers
Forecast across multiple series	time-series	MASE	scale-free, comparable across series
Ranking items (rec engine)	tree/deep	ROC-AUC / NDCG	order matters, not the predicted value

See evaluation-metrics.md for the full metric roster.

Cross-references into the articles

• The whole classical-ML walk: start at classical-ml-foundations-p01-can-you-explain-the-bias-variance-tradeoff-in-plai.md (the Archer and the Target).
• The whole attention walk: start at attention-from-scratch-a-deep-p01-neural-networks-without-the-calculus-what-s-actual.md.
• The whole causal-inference walk: start at causal-inference-in-python-p01-correlation-isn-t-causation-and-now-you-can-do-som.md.
• Picking the right metric after the algorithm: evaluation-metrics.md.
• Activation functions inside deep nets: activation-functions.md.

Further reading

• Hastie, Tibshirani & Friedman (2009). The Elements of Statistical Learning. — the textbook that organizes every supervised algorithm in this tree.
• Bishop (2006). Pattern Recognition and Machine Learning. — chap. 3–4 for linear, chap. 6–7 for kernels and SVM, chap. 14 for trees + ensembles.
• Goodfellow, Bengio & Courville (2016). Deep Learning. — the cannon for the deep-learning leaf.
• Efron & Hastie (2016). Computer Age Statistical Inference. — the bridge between classical stats and modern ML that this map glosses over.
• Kaggle — House Prices: Advanced Regression Techniques (tree-based on tabular).
• Kaggle — Spaceship Titanic (mixed tree/linear baseline on tabular).
• Kaggle — Store Sales – Time Series Forecasting (the time-series leaf on a real problem).