Neural Network Basics

• output = activation(w·x + b)
• Weights scale inputs; bias shifts the decision threshold
• One perceptron = one linear boundary
• XOR is not linearly separable, so it fails

• Many neurons, same input: y = f(W x + b)
• W is shaped (out, in)
• Params = in × out + out (weights + biases)

• Linear layers only: W2(W1 x) = one linear map
• Each layer needs a nonlinear activation
• Nonlinearity is what lets a stack solve XOR

• Why can one perceptron not solve XOR? It draws only one linear boundary.
• Params in Linear(784, 128)? 784 × 128 + 128 = 100,480
• Why do N linear layers collapse? A product of matrices is a single matrix.

• Subtract max(x) before exp
• Result is unchanged, exp never overflows

import math
def softmax(xs):
    m = max(xs)
    e = [math.exp(x - m) for x in xs]
    return [v / sum(e) for v in e]

definition only, call it with your logits

• Feed input layer by layer: a = f(W a_prev + b)
• Cache each layer's input a_prev and pre-activation z
• Backward pass reuses the cache instead of recomputing

• Chain rule from loss to each weight: dL/dW = dL/da · da/dz · dz/dW
• Vanishing gradients: repeated factors below 1 shrink the signal
• Sigmoid derivative peaks at 0.25; ReLU (slope 1 for x > 0) eases it

pred = model(x)
loss = loss_fn(pred, y)
optimizer.zero_grad()
loss.backward()
optimizer.step()

call-site fragment; model, x, y defined elsewhere

• Why cache z? Backward needs f'(z) and a_prev.
• Why does sigmoid vanish? Derivative is at most 0.25, so deep products shrink.

• Gradient descent: w ← w − lr · ∂L/∂w
• Momentum keeps a velocity of past gradients
• Smooths noisy steps, speeds convergence

• Too high: diverges or oscillates
• Too low: crawls

• Epoch = one full pass over the training set
• Iteration = one batch update
• Iterations per epoch = N / batch_size
• Shuffle training data every epoch
• Never shuffle or augment the validation set

• Dropout is a regularizer
• Typical p: 0.2–0.5 for dense layers
• Early stopping: val loss rises while train loss falls

model.train()  # dropout on
...
model.eval()  # dropout off
with torch.no_grad():
    preds = model(x_val)

call-site fragment

• Why shuffle each epoch? Stops order-dependent learning; gradients are less correlated, more representative.
• Why rescale by 1/(1−p)? Keeps expected activation equal in training and eval.

pred = model(x)
loss = loss_fn(pred, y)
opt.zero_grad()
loss.backward()
opt.step()

call-site fragment: model, x, y, opt defined elsewhere

• Overfit one small batch first
• Loss NaN: lr too high, or bad softmax/log
• Train/val gap: more regularization