Learning Outcomes
SLO | After this lecture you will be able to... | CLO | Bloom |
|---|---|---|---|
SLO 27.1 | Explain the structure of a multilayer network and compute its forward pass with sigmoid neurons | CLO3 | C3 Apply |
SLO 27.2 | Apply back-propagation to compute output and hidden deltas and update the weights once | CLO3 | C3 Apply |
SLO 27.3 | Explain how training settings (epochs, learning rate, hidden neurons, random start) affect learning | CLO3 | C2 Understand |
1. Layers and Smooth Neurons
§1.1 From step to sigmoid
Slide L27-S06
The step function of the perceptron is flat (slope 0), so it cannot tell a weight which way to move. The sigmoid (logistic function) σ(z) = 1 / (1 + e^(-z)) is a smooth S-curve from 0 to 1 with slope σ(1 - σ) - largest (0.25) at z = 0. Picture: a light switch (step) against a dimmer (sigmoid). Other activations: tanh (-1 to 1) and ReLU = max(0, z).
z step sigmoid slope s(1-s) -4 0 0.018 0.018 -2 0 0.119 0.105 -1 0 0.269 0.197 0 1 0.500 0.250 1 1 0.731 0.197 2 1 0.881 0.105 4 1 0.982 0.018 |
§1.2 A 2-2-1 network and its forward pass
Slide L27-S07
A multilayer perceptron (MLP) has an input layer, one or more hidden layers (not seen from outside; they build new features) and an output layer. The forward pass computes layer by layer: h = σ(W·x + b), then y = σ(v·h + c). Error for one example: E = ½ (t - y)².
h1: z = 0.2*1 + (-0.3)*0 + 0.1 = 0.300 h1 = sigmoid(0.300) = 0.574 h2: z = 0.4*1 + 0.1*0 + (-0.2) = 0.200 h2 = sigmoid(0.200) = 0.550 y: z = 0.5*0.574 + (-0.4)*0.550 + 0.1 = 0.167 y = sigmoid(0.167) = 0.542 E = 1/2 (1 - 0.542)^2 = 0.1050 |
Parameters of this network: 2 x 2 weights + 2 biases + 2 weights + 1 bias = 9.
2. Back-propagation
§2.1 Learning = walking downhill on the error
Slide L27-S09
Gradient descent moves every weight a small step against the slope of the error: w <- w - lr x slope (P11b used it for ANFIS). Back-propagation (Rumelhart, Hinton and Williams, 1986) finds all the slopes with the chain rule: first the output delta, then the hidden deltas, which receive a share of the output's error through the weights v.
output delta: d_o = (y - t) · y · (1 - y) hidden delta: d_h = d_o · v · h · (1 - h) update: weight <- weight - lr · delta · input bias <- bias - lr · delta |
§2.2 One backward pass by hand
Slide L27-S10
output delta d_o = (y - t) y (1 - y) = (0.542 - 1)(0.542)(0.458) = -0.1138 hidden deltas d_h = d_o * v * h(1 - h) = -0.0139, 0.0113 new output weights v = 0.5327, -0.3687 bias c = 0.1569 new hidden weights h1 = (0.2070, -0.3000) b 0.1070; h2 = (0.3944, 0.1000) b -0.2056 output 0.5417 -> 0.5654; error 0.1050 -> 0.0945 |
d_o is negative because the output is too low, so the weights on active paths go up. The weights from x2 do not change because x2 = 0. After one update the error falls from 0.1050 to 0.0945.
3. Training a Network
§3.1 Back-propagation learns XOR
Slide L27-S12
epoch 1: error 0.5183 epoch 500: error 0.0135 epoch 1000: error 0.0041 epoch 2000: error 0.0016 epoch 5000: error 0.0006 outputs: [0.017 0.981 0.984 0.015] -> [0 1 1 0] target [0 1 1 0] |
§3.2 Random starts, local minima and settings
Slide L27-S13
seed 0: error 0.0006 learned seed 1: error 0.2509 STUCK seed 2: error 0.0006 learned seed 3: error 0.0006 learned seed 4: error 0.3342 STUCK seed 5: error 0.0006 learned seed 6: error 0.2508 STUCK seed 7: error 0.0006 learned seed 8: error 0.0006 learned seed 9: error 0.0006 learned 7 of 10 starts learned XOR |
Three of ten random starts get stuck in a local minimum (a valley that is not the lowest). Fixes: several random starts, more hidden neurons, better optimisers (adam, lbfgs), momentum.
Hyperparameter (we choose) | Too small | Too large |
|---|---|---|
Learning rate | Very slow | Jumps about |
Epochs | Underfits | May overfit |
Hidden neurons | Cannot bend | Memorises noise |
§3.3 The same in scikit-learn
Slide L27-S14
from sklearn.neural_network import MLPClassifier m = MLPClassifier(hidden_layer_sizes=(4,), activation="logistic", solver="lbfgs", random_state=0, max_iter=2000).fit(X, T) |
XOR predictions: [0 1 1 0] weight shapes: (2, 4) (4, 1) moons, 400 points: 1 hidden: train 0.86 test 0.90 2 hidden: train 0.86 test 0.91 5 hidden: train 0.94 test 0.95 20 hidden: train 0.97 test 0.92 |
Choose the number of hidden neurons by test accuracy: 5 is best here; 20 fits the training data better but the test data worse. P03 and P05 used MLPClassifier, P04 MLPRegressor - the same algorithm.
4. Common Mistakes
Wrong idea | Correct idea |
|---|---|
Back-propagation predicts by running backwards. | Prediction is forward; only the error goes back. |
Hidden neurons get targets. | They get a share of the output delta through v. |
w <- w + lr · delta · input. | Minus: downhill. |
One run always finds the best weights. | It can stick in a local minimum. |
More hidden neurons is always better. | Too many can overfit. |
Word Bank
Word | Simple meaning | Example |
|---|---|---|
Sigmoid | Smooth S-shaped activation, 0 to 1 | σ(0) = 0.5 |
Hidden layer | Neurons between input and output | h1, h2 |
Forward pass | Computing the output layer by layer | y = 0.542 |
Delta | How much the error changes with a neuron's input | d_o = -0.1138 |
Back-propagation | Sending deltas backwards to update all weights | Rumelhart et al., 1986 |
Local minimum | A valley that is not the lowest | 3 of 10 XOR starts |
Hyperparameter | A setting we choose | learning rate, hidden size |
Self-Check Questions
- Why can the step function not be used with back-propagation? (see §1.1)
- Forward pass for x = (0, 0) in the S07 network. (see §1.2)
- Write the output and hidden delta formulas. (see §2.1)
- Why did the x2 weights not change in §2.2? (see §2.2)
- What is a local minimum and how can it be avoided? (see §3.2)
- How many parameters does a 3-5-2 network have? (see §3.3)
Reading: NN Toolbox Guide Ch 3, 5 (multilayer networks, back-propagation); Russell & Norvig Ch 21.2-21.4; P03-P05 handouts.