Machine learning from first principles — the classical-ML floor beneath this blog's LLM and AI content, for engineers who never built it. What ML is (learning rules from examples instead of writing them; model, features, training, generalization; supervised/unsupervised/reinforcement/self-supervised paradigms), data and features (why they, not the model, decide outcomes — quantity, quality, representativeness as the root of bias; feature engineering as the model's vocabulary; the data-centric lesson), regression and linear models (linear and logistic regression; the universal loop of model-loss-gradient-descent that scales to LLMs; the learning rate; sigmoid to softmax), trees and ensembles (decision trees, random forests, gradient boosting, and why boosted trees still beat neural nets on tabular data), overfitting and generalization (ML's central problem; the bias-variance trade-off; the train/test split, leakage and contamination; regularization as Occam's razor), evaluating models (why accuracy lies on imbalanced data; the confusion matrix, precision/recall/F1, regression metrics, segment-level evaluation for fairness, and metrics as proxies — Goodhart's law), unsupervised learning (clustering, dimensionality reduction as the root of embeddings, anomaly detection, and evaluating without labels), and the ML workflow (framing, data prep, baselines, honest iteration, deployment and concept drift — the bridge to MLOps/LLMOps, and why classical ML isn't obsolete). Grounded in Wikipedia (machine learning, supervised learning, feature engineering, linear/logistic regression, decision trees, random forest, overfitting, bias-variance, precision and recall, cross-validation, cluster analysis, dimensionality reduction).
Most of this blog's AI content sits on top of machine learning without ever defining it. This series fixes that from the ground up: before transformers and LLMs, before fine-tuning and RAG, there is the plain idea of a program that improves at a task by learning patterns from data instead of being explicitly told the rules. Understanding that idea — and its three core flavors — is the foundation everything else in modern AI is built on.
Before transformers and LLMs there is the plain idea of a program that learns patterns from data instead of being told the rules. What machine learning is, the vocabulary (features, model, training, generalization), and the three paradigms — supervised, unsupervised, reinforcement — the whole AI stack is built on.
The glamorous part of machine learning is the model; the decisive part is the data and the features. A mediocre model on great features beats a great model on poor ones, almost every time — which is why experienced practitioners spend most of their effort here and almost none picking algorithms. This post is about the unglamorous, high-leverage work of turning raw data into the features a model can actually learn from.
A mediocre model on great features beats a great model on poor ones. The data properties that govern what's possible (quantity, quality, representativeness — the root of bias) and feature engineering as the model's vocabulary — the highest-leverage, least-glamorous work in ML.
Before neural networks, the two models that still quietly run more of the world's predictions than anything else are linear and logistic regression. They are simple enough to understand completely, fast enough to train on anything, and interpretable enough to explain to a regulator — and understanding them teaches the mechanics (fitting, loss, gradient descent) that every more complex model shares. This post builds both from scratch and shows why "just use a linear model" is often the right answer.
Linear and logistic regression still run more of the world's predictions than anything else — and they teach the universal loop (model, loss, gradient descent) that scales all the way to training an LLM. Fitting a line, the learning rate, and the sigmoid that becomes the softmax.
If you have a spreadsheet and a prediction to make, the state of the art is probably not a neural network — it's a gradient-boosted ensemble of decision trees. Decision trees are intuitive, handle messy real-world data gracefully, and when combined into ensembles become the most consistently winning approach to tabular machine learning. This post builds from a single tree to the ensembles that quietly power a huge fraction of production ML.
If you have a spreadsheet and a prediction to make, the state of the art is probably not a neural network — it's a gradient-boosted ensemble of trees. From a single decision tree to random forests and boosting, and the useful heuristic: tabular → trees, unstructured → deep learning.
If there is one idea that separates people who understand machine learning from people who can run a library, it is this: the goal is never to do well on the data you have, but on the data you don't. A model that nails its training data and fails in production hasn't learned — it has memorized. Overfitting is the central failure mode of ML, and the techniques to fight it (the train/test split, regularization, the bias-variance trade-off) are the discipline's core craft.
The goal is never to do well on the data you have, but on the data you don't. Overfitting vs underfitting, the bias-variance trade-off, the train/test split (why training accuracy is meaningless and leakage/contamination fools you), and regularization as Occam's razor for models.
The most dangerous number in machine learning is accuracy, because it's the one everyone reaches for and the one that most often misleads. A fraud model that's 99.9% accurate by never catching any fraud is useless and common. Choosing the right evaluation metric — and understanding what each one hides — is the difference between a model that looks good and a model that works. This post is about measuring models honestly.
The most dangerous number in ML is accuracy — a 99.9%-accurate fraud model that catches nothing is common. The confusion matrix, precision vs recall (a business decision), F1/ROC, regression metrics, segment-level evaluation (how bias is caught), and why every offline metric is a proxy (Goodhart's law).
Most data in the world is unlabeled, and labeling it is expensive — so some of the most useful machine learning finds structure in data with no answer key at all. Unsupervised learning groups similar things, compresses high-dimensional data, and flags the anomalies, all without being told what "right" looks like. This post covers the main unsupervised techniques and, just as importantly, how you tell whether they worked when there's no label to check against.
Most data is unlabeled, and some of the most useful ML finds structure without an answer key. Clustering, dimensionality reduction (the root of embeddings and semantic search), anomaly detection, and the hard problem of evaluating when there's no label to check against.
Knowing the algorithms is maybe 20% of doing machine learning. The other 80% is the workflow around them: framing the problem, preparing the data, iterating honestly, and getting a model into production where it keeps working. This closing post assembles the series into the end-to-end process real ML follows — and names the places where projects actually fail, which are almost never the algorithm.
Algorithms are ~20% of ML; the other 80% is the workflow — framing the problem, preparing data, iterating honestly, and keeping a model working in production where it decays (concept drift). Why classical ML isn't obsolete, and why these foundations are universal to deep learning and LLMs.
This series is part of a larger body of work by Pratik Dhanave, an Agentic AI Architect writing about production AI systems, distributed systems, and cloud-native engineering. Explore all course series, browse every post, or find topics via the tag index.