Learning Outcomes
SLO | After this lecture you will be able to... | CLO | Bloom |
|---|---|---|---|
SLO 26.1 | Explain the parts of an artificial neuron and compute its output with a step activation | CLO3 | C2 Understand |
SLO 26.2 | Apply the perceptron learning rule to train a neuron on a small data set (AND, OR) | CLO3 | C3 Apply |
SLO 26.3 | Explain linear separability and why a single perceptron cannot learn XOR | CLO3 | C2 Understand |
1. From the Brain to an Artificial Neuron
§1.1 Biological and artificial neurons
Slide L26-S06
A biological neuron receives signals through its dendrites, adds them in the cell body and fires along its axon if the total passes a threshold; synapses (connections) can be strong or weak. McCulloch and Pitts (1943) turned this into a simple mathematical neuron; Frank Rosenblatt (1958) added a way to learn the weights - the perceptron.
Brain | Artificial neuron | Meaning |
|---|---|---|
Dendrites | Inputs x1, x2, ... | The facts the neuron sees |
Synapse strength | Weights w1, w2, ... | How important each input is (sign = direction) |
Cell body adds | z = w·x + b | Weighted sum plus bias |
Fires or not | Step: y = 1 if z >= 0 else 0 | The decision |
The bias b moves the cut-off: a negative bias makes the neuron harder to fire. It is a learned weight on a constant input 1.
§1.2 One neuron by hand
Slide L26-S07
Should a farmer run the tubewell tonight? Inputs are 0/1: soil is dry, hot day, rain forecast. Rain argues against pumping, so its weight is negative.
soil is dry (0/1) x = 1 w = +0.6 w*x = +0.6 hot day (0/1) x = 1 w = +0.4 w*x = +0.4 rain forecast (0/1) x = 0 w = -0.9 w*x = +0.0 bias b = -0.5 z = w.x + b = +0.5 -> step(z) = 1 (1 = run the pump) same day with rain forecast: z = -0.4 -> 0 |
2. The Perceptron Learning Rule
§2.1 The rule
Slide L26-S09
error = t - y (t = target, y = output; error is 0, +1 or -1) w <- w + lr · error · x b <- b + lr · error |
t | y | t - y | What happens |
|---|---|---|---|
1 | 1 | 0 | Nothing |
0 | 0 | 0 | Nothing |
1 | 0 | +1 | Add lr·x to w and lr to b (raise z) |
0 | 1 | -1 | Subtract lr·x from w and lr from b (lower z) |
Learning rate (lr): the step size. Epoch: one pass over all training examples. Stop when a whole epoch has no mistakes. Convergence theorem: if one straight line can separate the classes, the rule always finds one in a finite number of steps.
§2.2 Worked example: AND
Slide L26-S10
Learning rate 1, start w = (0, 0), b = 0, examples in the order (0,0), (0,1), (1,0), (1,1). Epoch 1 is shown in full, then the weights at the end of each epoch.
x=(0, 0) t=0 y=1 -> w=(+0, +0) b=-1 x=(0, 1) t=0 y=0 -> w=(+0, +0) b=-1 x=(1, 0) t=0 y=0 -> w=(+0, +0) b=-1 x=(1, 1) t=1 y=0 -> w=(+1, +1) b=+0 epoch 1: 2 mistakes w = (1, 1), b = 0 epoch 2: 3 mistakes w = (2, 1), b = -1 epoch 3: 3 mistakes w = (2, 1), b = -2 epoch 4: 2 mistakes w = (2, 2), b = -2 epoch 5: 1 mistake w = (2, 1), b = -3 epoch 6: 0 mistakes w = (2, 1), b = -3 learned: w = (2, 1), b = -3 after 6 epochs |
Epoch 1, first example: (0, 0) gives z = 0, so y = 1 but t = 0. Error -1, so b becomes -1. The weights do not change because both inputs are 0 (lr · error · 0 = 0). Check the result: (1, 1): 2 + 1 - 3 = 0 -> 1; (1, 0): -1 -> 0; (0, 1): -2 -> 0.
§2.3 A perceptron for the 12 students
Slide L26-S11
weights [1.32 0.69] bias 0.00 training accuracy 1.00 epochs 7 new student 8 h, 60 % -> pass new student 4 h, 85 % -> fail new student 11 h, 60 % -> pass new student 7 h, 72 % -> pass |
Features are scaled first (StandardScaler): a perceptron adds the features, so big numbers would dominate. The perceptron gives study hours the larger weight, so it predicts 8 h, 60 % as pass - the L25 tree said fail. Both are 100 % on 12 students; only more data and a test set can tell which to trust. A perceptron's boundary is always a straight line w1·x1 + w2·x2 + b = 0.
3. What One Perceptron Cannot Do
§3.1 Linear separability and XOR
Slide L26-S13
Linearly separable: one straight line can put all the 1s on one side and all the 0s on the other. AND and OR are; XOR (1 when exactly one input is 1 - like a staircase light with two switches) is not.
AND: target [0, 0, 0, 1] output [0, 0, 0, 1] w = (2, 1) b = -3 learned in 6 epochs OR : target [0, 1, 1, 1] output [0, 1, 1, 1] w = (1, 1) b = -1 learned in 4 epochs XOR: target [0, 1, 1, 0] output [1, 1, 0, 0] w = (-1, 0) b = 0 still 4 mistakes after 20 epochs |
Minsky and Papert's book Perceptrons (1969) proved this limit. Research money for neural networks fell sharply for more than a decade (an 'AI winter').
§3.2 The way out: more than one layer
Slide L26-S14
x1 | x2 | OR | NAND | XOR = AND(OR, NAND) |
|---|---|---|---|---|
0 | 0 | 0 | 1 | 0 |
0 | 1 | 1 | 1 | 1 |
1 | 0 | 1 | 1 | 1 |
1 | 1 | 1 | 0 | 0 |
OR, NAND and AND are each one perceptron (NAND: w = (-1, -1), b = 1.5). Two neurons in a hidden layer (OR, NAND) feeding one AND neuron compute XOR. The perceptron rule cannot train hidden neurons (we do not know their targets, and the step has no slope); smooth activations and back-propagation (Rumelhart, Hinton and Williams, 1986) solve this - Lecture 27. P02 built a perceptron; P03's MLPClassifier is a multilayer network.
4. Common Mistakes
Wrong idea | Correct idea |
|---|---|
Weights change after every example. | Only after a mistake. |
A weight changes even when its input is 0. | lr · error · 0 = 0. |
A perceptron can learn any pattern. | Only linearly separable ones. |
The bias is an input value. | It is a learned weight on a constant input 1. |
The learned weights are unique. | Many weight sets work. |
Word Bank
Word | Simple meaning | Example |
|---|---|---|
Neuron | A unit that weighs inputs, adds them and decides | Tubewell neuron |
Weight | How important an input is | rain -0.9 |
Bias | Shifts the cut-off | b = -0.5 |
Activation function | Turns the sum into an output | step: 1 if z >= 0 |
Learning rate | Size of each correction | lr = 1 |
Epoch | One pass over all training examples | 4 examples of AND |
Linearly separable | One straight line can split the classes | AND, OR |
Hidden layer | Neurons between input and output | OR and NAND neurons for XOR |
Self-Check Questions
- Tubewell neuron: output for (0, 0, 1)? (see §1.2)
- Why is the rain weight negative? (see §1.2)
- w = (2, 1), b = -3, x = (0, 1), t = 1, lr = 1. New w and b? (see §2.1)
- What is an epoch? (see §2.1)
- Why must features be scaled for a perceptron? (see §2.3)
- Draw the four XOR points and explain why no line separates them. (see §3.1)
- Give the weights of an OR neuron and a NAND neuron. (see §3.2)
Reading: NN Toolbox Guide Ch 1-3 (neuron model, perceptrons); Russell & Norvig Ch 21.1; Practical 02 handout (perceptron from scratch).