Handbooks / Machine Learning Basics / Chapter 2
Linear and Logistic Regression
7 pages · ~5 min✓ Reviewed
Builds on ML Foundations. Next up: Model Evaluation.
Regression Mental Model
1. Linear regression
- Predicts a number: ŷ = w·x + b
- A weighted sum of features plus a bias
- Learning = choose w and b that minimize a loss over the training data
- Residual = y − ŷ; the fit line has the smallest Σ residual²
2. Logistic regression
- z = w·x + blinear score
- p = σ(z)probability
- p ≥ 0.5 ?threshold
- Class0 or 1
| Linear | Logistic | |
|---|---|---|
| Predicts | a number | a class probability |
| Formula | ŷ = w·x + b | p = σ(w·x + b) |
| Last step | none | threshold (default 0.5) |
3. Both are linear models
- Boundary is where p = 0.5, i.e. w·x + b = 0
- A straight line in 2D, a hyperplane in higher dimensions
- Class flips as w·x + b changes sign
Remember
The sigmoid bends the probability, not the boundary. Logistic regression still separates classes with a flat line or plane.
Least Squares and Gradient Descent
1. Least Squares and the Normal Equation
- Least squares minimizes Σ(y − ŷ)², the sum of squared residuals
- Closed form: w = (XᵀX)⁻¹Xᵀy
- Cost about O(n·d²+d³): fine for small d
- Fails when XᵀX is singular (collinear features)
2. Gradient Descent
- Predict ŷcurrent w
- Gradient∂L/∂w
- Updatew ← w − α·∂L/∂w
- Loss falling?repeat until it stops
| α (learning rate) | Effect |
|---|---|
| Too large | diverges |
| Too small | crawls |
| Just right + standardized features | converges fast |
3. Batch, SGD, Mini-batch
| Variant | Rows per step |
|---|---|
| Batch | all rows |
| SGD | one row |
| Mini-batch | a small chunk (default in practice) |
Remember
Always standardize features before gradient descent. Use the normal equation only for small d.
Sigmoid and Logistic Regression
1. Sigmoid and logit
- σ(z) = 1 / (1 + e^−z) squashes any real z into (0, 1)
- σ(0) = 0.5
- Logit = ln(p / (1 − p)) = w·x + b: linear in log-odds
2. Reading a weight
- w = change in log-odds per one-unit increase in a feature
- e^w = odds ratio for that one-unit increase
| w | e^w | Effect |
|---|---|---|
| 0 | 1 | none |
| 0.69 | 2 | odds double |
| −0.69 | 0.5 | odds halve |
3. Training and threshold
- Log loss = −(1/n)·Σ[y·ln p + (1−y)·ln(1−p)]
- Gradient w.r.t. w: Xᵀ(p − y)/n, same form as linear regression
| Threshold | Favors | Cost |
|---|---|---|
| Below 0.5 | recall: positives caught | more false positives |
| Above 0.5 | precision: positives correct | more misses |
Separable data
Weights grow without bound unless regularized.
Loss Functions: MSE and Log Loss
1. MSE and Its Cousins
- MSE = (1/n)·Σ(y − ŷ)²
- Convex for linear regression
- Big errors cost quadratically; outliers dominate
| MSE | MAE | Huber | |
|---|---|---|---|
| Penalty | squared | absolute | squared small, linear large |
| Outliers | sensitive | robust | robust |
2. Log Loss
- Log loss = binary cross-entropy: −(1/n)·Σ[y·ln p + (1−y)·ln(1−p)]
- y=1 keeps
ln p; y=0 keepsln(1−p) - Confident and wrong costs a lot: p = 0.01, y = 1 → −ln 0.01 ≈ 4.6
python
from math import log def log_loss(ys, ps, e=1e-15): ps = [min(max(p, e), 1 - e) for p in ps] return -sum(y*log(p) + (1-y)*log(1-p) for y, p in zip(ys, ps))/len(ys) print(round(log_loss([1], [0.01]), 2))
clipping to [ε, 1−ε] avoids ln 0 = −∞
output
4.61- Interview Point: why not MSE for classification? Non-convex with sigmoid, weak gradients.
- Which loss for outliers? MAE or Huber.
Regularization: L1 and L2
1. Penalized Loss
- Penalty on weights added to the loss to fight overfitting
L = loss + λ·penalty
| L2 (Ridge) | L1 (Lasso) | |
|---|---|---|
| Penalty | λ·Σw² | λ·Σ|w| |
| Effect | shrinks smoothly, rarely exactly 0 | drives some weights exactly 0 |
| Constraint shape | circle: misses the axes | diamond: hits the axes (sparse) |
| Bonus | none | built-in feature selection |
2. Tuning λ
- Larger λ → more bias, less variance
- Choose λ by cross-validation
- scikit-learn logistic regression uses
C = 1/λ: small C = strong penalty
python
from sklearn.linear_model import Ridge, Lasso, LogisticRegression Ridge(alpha=1.0) # L2, alpha = λ Lasso(alpha=0.1) # L1 LogisticRegression(C=0.1) # C = 1/λ
alpha is λ; C is its inverse
3. Rules and Elastic Net
Elastic Net
Mixes L1 + L2. Sparse like L1, and handles correlated features better than L1 alone.
Regression Cheat Sheet
1. Models and Losses
| Linear | Logistic | |
|---|---|---|
| Predicts | ŷ = Xw + b | p = σ(Xw + b) |
| Loss | MSE | log loss |
| Outliers dominate | MAE or Huber | n/a |
2. Fitting
- Both models:
w ← w − α·Xᵀ(ŷ − y)/n, with ŷ = p for logistic - Normal equation
w = (XᵀX)⁻¹Xᵀy: small d only
yesno
3. Regularize and Check
| Penalty | Effect | |
|---|---|---|
| L2 (Ridge) | λ·Σw² | shrinks |
| L1 (Lasso) | λ·Σ|w| | selects |
| Elastic Net | both | both |
- Standardize features
- Check the learning-rate curve
- Hold out a validation set
- Pick the threshold from precision vs recall
Tune λ
Choose λ by cross-validation. Larger λ means more bias, less variance.
Check yourself
A logistic model outputs p = 0.01 for a row whose true label is 1. What is that row's log loss, and why does it matter?
- −ln 0.01 ≈ 4.6
- A hedged p = 0.4 costs only about 0.9
- Confident wrong predictions dominate the loss
Gradient descent loss rises every step, then becomes NaN. What is wrong and what fixes it?
- Learning rate α is too large, so the steps overshoot and diverge
- Lower α and standardize the features
Two models use the same λ. One ends with several weights exactly 0. Which penalty was it?
- L1: its diamond constraint hits the axes
- L2 shrinks weights but rarely zeroes them
Unregularized logistic regression is fit on perfectly separable data. What happens to the weights?
- They grow without bound, because larger weights keep lowering the loss
- Add regularization such as L2, or lower C