Handbooks / Machine Learning Basics / Chapter 3

Model Evaluation

7 pages · ~4 min✓ Reviewed

Builds on Linear and Logistic Regression. Next up: Decision Trees and Ensembles.

Machine Learning BasicsPage 1

Confusion Matrix and Core Metrics

1. The confusion matrix

Actual \ Predicted	Positive	Negative
Positive	TP	FN
Negative	FP	TN

2x2 table of actual vs predicted
FP = Type I error (false alarm)
FN = Type II error (miss)

2. Core metrics

Metric	Formula	Use when
Accuracy	(TP+TN)/(TP+TN+FP+FN)	classes balanced
Precision	TP/(TP+FP)	false alarms costly (spam filter)
Recall (sensitivity, TPR)	TP/(TP+FN)	misses costly (cancer screening)

Accuracy: share of all predictions that are correct
Precision: of predicted positives, how many were right
Recall: of real positives, how many were found

3. Worked example

TP=40, FP=10, FN=20, TN=930
Accuracy = 970/1000 = 97%
Precision = 40/50 = 80%
Recall = 40/60 = 66.7%

High accuracy can hide misses

97% accuracy, yet 20 of 60 real positives were missed. Check recall too.

Machine Learning BasicsPage 2

F1 Score and the Threshold Trade-off

1. F1 and the Threshold

F1 = 2PR/(P+R), the harmonic mean of precision and recall
It punishes a gap: P=1.0, R=0.1 gives F1 ≈ 0.18, not 0.55
Lower the threshold: recall up, precision down

Raise thresholddefault 0.5
Fewer positivespredicted
Precision up
Recall down

2. F-beta and Averaging

F-beta = (1+β²)PR/(β²P+R)
β=2 weights recall more
β=0.5 weights precision more

Average	How
Macro	Classes count equally
Micro	Pools all counts; = accuracy for single-label multiclass
Weighted	Averages by class support

3. In sklearn

python

from sklearn import metrics as m
m.precision_recall_fscore_support(y_true, y_pred)
m.classification_report(y_true, y_pred)

both report P, R, F1 and support per class

Remember

F1 ignores TN, so state the threshold with any F1 you report.

Machine Learning BasicsPage 3

ROC Curve and AUC

1. The ROC Curve

ROC plots TPR (y) against FPR = FP/(FP+TN) (x)
The threshold sweeps from 1 to 0, tracing one point per cutoff

2. Reading AUC

AUC	Meaning
1.0	perfect ranking
0.5	random guessing (the diagonal)
below 0.5	labels are inverted

AUC = area under the ROC curve
= P(random positive is scored above a random negative)
Threshold-independent: measures ranking only
Says nothing about calibration or the best cutoff

3. Imbalance and sklearn

Huge TN count keeps FPR tiny, so ROC/AUC looks optimistic
On heavy imbalance use the precision-recall curve and PR-AUC (average precision)

python

from sklearn.metrics import roc_auc_score, average_precision_score
roc_auc_score(y_true, y_score)
average_precision_score(y_true, y_score)

y_score = probabilities or scores

Scores, not labels

roc_auc_score needs probabilities or scores. Hard 0/1 labels collapse the curve to one point.

Machine Learning BasicsPage 4

Regression Metrics: MAE, RMSE, R2

1. Error Metrics

Metric	Formula	Notes
MAE	`mean(\|y−ŷ\|)`	target's own units; robust to outliers
MSE	`mean((y−ŷ)²)`	squared units; squaring punishes big errors
RMSE	`√MSE`	own units; RMSE ≥ MAE; big gap = outliers

2. R² Score

R² = 1 − SS_res/SS_tot: share of variance explained
R² = 0: same as predicting the mean
Negative on test data: worse than the mean
Extra features never lower training R²
Use adjusted R² to penalize useless features

3. Choosing One

Pick	When
MAE	all errors cost the same
RMSE	big misses are disproportionately bad

Example: y=[3,5,7], ŷ=[2,5,10] gives errors −1, 0, +3

python

y = [3, 5, 7]; p = [2, 5, 10]
e = [a - b for a, b in zip(y, p)]
mae = sum(map(abs, e)) / 3
rmse = (sum(x * x for x in e) / 3) ** .5
print(round(mae, 2), round(rmse, 2))

output

1.33 1.83

Read the gap

RMSE well above MAE means a few large misses. R² can go negative on test data.

Machine Learning BasicsPage 5

Class Imbalance and How to Handle It

1. Why Accuracy Fails

Always predict majority: 99% accuracy on 1% positives, zero recall
Judge with recall, precision, F1 and PR-AUC
Stratified splits keep the class ratio in every fold

2. The Fixes

Fix	How	Data changed?
Stratify	`train_test_split(..., stratify=y)`, `StratifiedKFold`	No
Class weights	`class_weight='balanced'`, XGBoost `scale_pos_weight`	No
Resample	Over, under, or SMOTE (interpolates new minority points)	Train only
Threshold	Tune on validation set, not 0.5; cheapest fix	No

3. Leakage

Resampling before the split leaks synthetic copies into test
Scores get inflated
Resample inside the CV pipeline

python

from imblearn.pipeline import Pipeline
from imblearn.over_sampling import SMOTE
pipe = Pipeline([("smote", SMOTE()), ("clf", clf)])
cross_val_score(pipe, X, y, cv=StratifiedKFold(5),
                scoring="average_precision")

SMOTE refits on each training fold only

Order matters

Split first, resample train only, score on the untouched test set.

Machine Learning BasicsPage 6

Metrics Cheat Sheet

1. Classification Formulas

Metric	Formula
Accuracy	`(TP+TN)/N`
Precision	`TP/(TP+FP)`
Recall	`TP/(TP+FN)`
F1	`2PR/(P+R)`
FPR	`FP/(FP+TN)`

2. Pick the Metric

Situation	Use
False alarms hurt	Precision
Misses hurt	Recall
Need balance	F1
Ranking quality	AUC

Imbalanced data: ignore accuracy, use PR-AUC or F1
Stratify splits, use class weights, tune the threshold

MAE: typical error
RMSE: punishes outliers
R²: variance explained, can be negative

3. Never Do

Resample before splitting
Feed hard labels to roc_auc_score
Skip reporting the threshold used

Remember

Resample only the training data, give AUC scores, and always state the threshold.

Machine Learning BasicsPage 7

Check yourself

TP=40, FP=10, FN=20, TN=930. Which is higher, precision or recall?

Precision = 40/50 = 80%
Recall = 40/60 = 66.7%
Accuracy of 97% hides the 20 misses

roc_auc_score(y, model.predict(X)) gives a poor AUC. What is the bug?

predict returns hard 0/1 labels, so ranking is lost
Pass predict_proba(X)[:, 1] instead

SMOTE is applied to all data, then split and cross-validated. Scores look great. Why is that wrong?

Synthetic copies of test points leak into training
Resample only inside each training fold

y=[3,5,7], ŷ=[2,5,10]. Why is RMSE above MAE?

Errors 1, 0, 3: MAE 1.33, RMSE 1.83
Squaring weights the 3-miss more heavily