Topic 16.12
Classic Machine Learning in One Sitting: scikit-learn
In one line
Machine learning fits a rule to example data. You split the data into train and test sets, fit a model on the training part, predict and score on the test part, and watch for overfitting. You can build a nearest-neighbour classifier and a polynomial fit from scratch with NumPy before using scikit-learn.
Think of it like this
Learning to recognise ripe mangoes. You squeeze many mangoes and are told "ripe" or "not ripe". After enough examples you build a feel for colour and softness. To check you really learned, someone hands you new mangoes you have never touched. Machine learning is that: learn a rule from labelled examples, then test on new ones.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Feature
- One input column describing an example.
- Label (target)
- The answer you want the model to predict.
- Model
- A rule with adjustable numbers.
- Fit (train)
- Adjust the model's numbers to match the training data.
- Train/test split
- Separating data into examples to learn from and examples to check on.
- Overfitting
- Matching the noise in the training data, so new data is predicted badly.
- Pipeline
- Preprocessing steps and a model chained into one object.
- Data leakage
- Information from the test data sneaking into training, making scores look too good.
Step by step
01A classifier from scratch: nearest neighbour
The rule: to label a new point, find the closest training point and copy its label. This one needs no scikit-learn, so it runs live here. The first example makes two clusters with a seeded generator, shuffles and splits them 45/15, predicts every test point and prints the accuracy (the fraction right). Nothing is learned beyond remembering the training points, yet it works when classes are well separated.
02The five steps every ML script has
Load data, split it, fit, predict, score. In scikit-learn each is one line. Keep them in this order and keep the test set untouched until the end.
03Overfitting in numbers
The second example fits polynomials of degree 1, 3 and 8 to ten noisy points from a straight line, then measures error on new points from the same line. Training error shrinks with degree. Test error grows for degree 8: the curve wiggles through the noise. The simplest model that fits is often best.
04The same thing in scikit-learn
Not runnable here (scikit-learn is slow to load in the browser, so the course keeps this block as a reference). On your machine it is the real code you would write. The numbers below were checked on the classroom data.
05Scale before you measure distance
The last runnable example classifies a point whose first feature looks like class 1 but whose second feature has numbers in the hundreds. With raw features the big numbers decide and class 0 wins. After scaling with statistics from the training data, class 1 wins. A pipeline does this scaling automatically and consistently.
06How this connects to the rest of the phase
Embeddings (Topic 16.8) are features learned by a model. Evals (Topic 16.11) are train/test thinking applied to prompts. Gradient descent (Topic 16.13) is how many models, including neural networks, fit. The discipline of keeping a held-out set is the same everywhere.
Try it yourself
- 1
Move the clusters closer
In the first example change the second cluster's centre from 3 to 1. Predict whether accuracy goes up or down, then run it.
- 2
Try degree 2
Add 2 to the degree list in the overfitting example. Where is the best test error?
Code & diagrams
Expected output
train rows: 45 test rows: 15
test accuracy: 1.0
a point near (3, 3) is predicted as class 1Exact numbers depend on the seed. The pattern is what matters: degree 8 fits training points almost perfectly and does worse on new ones.
Expected output
degree 1: train error 0.1059 test error 0.0111
degree 3: train error 0.0633 test error 0.0304
degree 8: train error 0.0011 test error 0.1971Expected output
raw features -> 0
scaled features -> 1A score on training data is only a sanity check. In real work, call score on a held-out test set.
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X = [[1, 100], [2, 110], [3, 90], [8, 400], [9, 420], [10, 390]]
y = [0, 0, 0, 1, 1, 1]
model = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=3))
model.fit(X, y)
print("predict:", model.predict([[2, 105], [9, 410]]).tolist())
print("score on the training data:", model.score(X, y))Output
predict: [0, 1]
score on the training data: 1.0Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Predict with the wrong number of features
Give a trained model a row with fewer features than it was trained on.
import numpy as np
weights = np.array([0.5, -1.0, 2.0]) # the model learned 3 features
new_row = np.array([1.0, 2.0]) # but this row has only 2
print(weights @ new_row)Myth vs fact
Myth
A model with 100 percent training accuracy is perfect.
Fact
It may have memorised the training data. Only a held-out test set tells you how it generalises.
Myth
More complex models are always better.
Fact
Past a point, complexity fits noise. Start simple and add complexity only when test results justify it.
Myth
Machine learning needs neural networks.
Fact
Many problems on tables are solved by simple models. Neural networks matter most for images, audio and language.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Cross-validation repeats the split several ways (for example 5 folds) and averages scores, giving a steadier estimate than a single split, especially with small data. Hyperparameters (like the number of neighbours) are tuned with it, never on the final test set.
- ▸
Time-ordered data needs a time-ordered split, not a random shuffle: training on the future and testing on the past leaks information.
- ▸
Accuracy can mislead on imbalanced data (99 percent of rows are one class). Use precision, recall, F1 or a confusion matrix, and pick the metric from the cost of each kind of mistake.
- ▸
Tune and compare models inside the training data with cross-validation, and touch the test set once, at the end. Reuse of the test set to choose between models quietly leaks it into your decisions. With imbalanced classes use stratified splits (in scikit-learn,
train_test_split(..., stratify=y)orStratifiedKFold) so each part keeps the class proportions. - ▸
Put every learned preprocessing step (scaling, encoding, imputing missing values) inside a
Pipeline, and pass the pipeline tocross_val_score. Then each fold refits the preprocessing on its own training part, which is the correct, leak-free procedure. Fitting a scaler on all data first and cross-validating afterwards is the classic leak.
Remember this
- 1
The vocabulary: features are the input columns (colour, softness). The label (or target) is the answer to predict (ripe or not, or a price). A model is the rule with adjustable numbers. Fitting (training) adjusts those numbers to match the examples. Predicting applies the rule to new rows. Classification predicts a category; regression predicts a number.
- 2
Train/test split. Hold back part of the data (often 20 to 30 percent) that the model never sees while fitting. Score on it to estimate how the model behaves on new data. Scoring on the training data only tells you how well it memorised. Shuffle with a fixed seed before splitting so the result is repeatable (Topic 14.4).
- 3
Overfitting means the model matches the training data's noise, not the true pattern, so it looks great on training data and poor on new data. The polynomial example shows it with numbers: as the degree grows, training error keeps falling while test error rises. The opposite problem, underfitting, is a model too simple to capture the pattern. Fighting overfitting is much of practical ML. The tools are more data, simpler models, regularisation (a penalty that discourages extreme adjustable numbers) and cross-validation (testing several different splits and averaging the scores).
- 4
scikit-learn gives one interface to many models:
model.fit(X, y),model.predict(X_new),model.score(X, y). X is a 2-D array (rows are examples, columns are features) and y is a 1-D array. Preprocessing steps and a model can be chained into a pipeline, so the same scaling learned from the training data is applied to new data. That also prevents data leakage: letting information from the test set influence training, for example by scaling with the mean of all the data. - 5
Scale features for models based on distance, like nearest neighbours. In the last example one feature has numbers in the hundreds and one in single digits, so the distance is decided almost only by the big one. Standardising (subtract the mean, divide by the standard deviation, which measures how spread out the numbers are; both computed from the training data; Topic 14.4) puts features on a common footing.
- 6
Classic ML is still the right tool for tabular data: sales numbers, sensor readings, customer records. It is fast, cheap and explainable, and often beats a neural network on small tables. It is the base for deep learning too: the same train/test discipline applies. A full Hectal course on machine learning is planned.
Explain it without notes
Why do we split data into training and test sets?
What is overfitting, and how would you detect and reduce it?
What is data leakage? Give an example involving scaling.
A fraud model reports 99 percent accuracy. Is it good? What would you look at instead, and how would you evaluate it properly?
Practice
Write train_test_split_idx(n, test_fraction, seed) returning shuffled train and test index lists.
Write accuracy(y_true, y_pred) with a length check and 0.0 for empty input.
Write predict_1nn(X_train, y_train, point) that returns the label of the nearest training row.
Trade-offs
- ↔
Simple versus complex models. Simple models are fast, explainable and hard to overfit, but may underfit. Complex ones capture more and need more data and care.
- ↔
One split versus cross-validation. One split is quick and noisy. Cross-validation is steadier and costs several fits.
Done when you can
Done when you can name features, label, fit, predict and score.
Done when you can split data reproducibly with a seed.
Done when you can show overfitting with train and test errors.
Done when you can explain why scaling and pipelines matter.
Done when you know a full Hectal course on machine learning is planned.