Recap of Lecture 25 and Homework Check
LAST LECTURE · 5 MIN
Today: a tree asks one question at a time. A neuron weighs ALL its inputs at once and adds them up - like a judge giving marks.
Today's Plan (50 minutes)
ROADMAP
0-5
Recap of Lecture 25 and homework check
S02
5-8
Plan and learning outcomes
S03-S04
8-16
Module 1: From the brain to an artificial neuron
S05-S07
16-27
Module 2: The perceptron learning rule
S08-S11
27-35
Module 3: What one perceptron cannot do - XOR
S12-S14
35-43
Class activity: neurons and weight updates by hand
S15
43-48
Common mistakes and recap quiz
S16-S17
48-50
Homework and next lecture
S18
Learning Outcomes (SLOs)
BY THE END OF THIS LECTURE YOU WILL BE ABLE TO
SLO 26.1
Explain the parts of an artificial neuron and compute its output with a step activation
→ CLO3 · C2 Understand
SLO 26.2
Apply the perceptron learning rule to train a neuron on a small data set (AND, OR)
→ CLO3 · C3 Apply
SLO 26.3
Explain linear separability and why a single perceptron cannot learn XOR
→ CLO3 · C2 Understand
- Course Learning Outcome(s) served
- CLO3: Apply various techniques and algorithms of AI used in general problem solving, optimization problems, constraint satisfaction problems, and game programming.
Slide 5
MODULE 1
From the Brain to an Artificial Neuron
Inputs, weights, a sum and a decision
Biological and Artificial Neurons
MODULE 1 · THE MODEL
THE PERCEPTRON'S OUTPUT
Bias b: moves the cut-off - a negative bias makes the neuron harder to fire.
- z = w1·x1 + w2·x2 + w3·x3 + b
- y = step(z) = 1 if z >= 0 else 0
McCulloch and Pitts (1943) first model; Rosenblatt (1958) added learning - the perceptron.
One Neuron by Hand: Run the Tubewell Tonight?
MODULE 1 · WORKED EXAMPLE
REAL OUTPUT: L26/NEURON.PY
- soil is dry (0/1) x = 1 w = +0.6 w*x = +0.6
- hot day (0/1) x = 1 w = +0.4 w*x = +0.4
- rain forecast (0/1) x = 0 w = -0.9 w*x = +0.0
- bias b = -0.5
- z = w.x + b = +0.5 -> step(z) = 1 (1 = run the pump)
- same day with rain forecast: z = -0.4 -> 0
BY HAND
- z = 0.6·1 + 0.4·1 + (-0.9)·0 + (-0.5) = +0.5 -> 1 (run)
- z = 0.6·1 + 0.4·1 + (-0.9)·1 + (-0.5) = -0.4 -> 0 (wait)
Question: we chose these weights ourselves. Can the neuron learn them from past nights instead?
Slide 8
MODULE 2
The Perceptron Learning Rule
Learn from every mistake
The Learning Rule
MODULE 2 · ROSENBLATT (1958)
PERCEPTRON RULE
- error = t - y
- w <- w + lr · error · x
- b <- b + lr · error
Learning rate
Epoch
Stop
Step size lr. We use lr = 1 so the numbers stay whole.
One pass over all the training examples.
When a whole epoch has no mistakes.
Convergence theorem: if one straight line can separate the classes, the rule is guaranteed to find one.
Training a Perceptron on AND
MODULE 2 · WORKED EXAMPLE
REAL OUTPUT: L26/TRAIN_AND.PY (LR = 1, START W = 0, B = 0)
- x=(0, 0) t=0 y=1 -> w=(+0, +0) b=-1
- x=(0, 1) t=0 y=0 -> w=(+0, +0) b=-1
- x=(1, 0) t=0 y=0 -> w=(+0, +0) b=-1
- x=(1, 1) t=1 y=0 -> w=(+1, +1) b=+0
- epoch 1: 2 mistakes w = (1, 1), b = 0
- epoch 2: 3 mistakes w = (2, 1), b = -1
- epoch 3: 3 mistakes w = (2, 1), b = -2
- epoch 4: 2 mistakes w = (2, 2), b = -2
- epoch 5: 1 mistake w = (2, 1), b = -3
- epoch 6: 0 mistakes w = (2, 1), b = -3
- learned: w = (2, 1), b = -3 after 6 epochs
CHECK
- check w = (2, 1), b = -3:
- (1, 1): 2 + 1 - 3 = 0 -> 1
- (1, 0): 2 + 0 - 3 = -1 -> 0
- (0, 1): 0 + 1 - 3 = -2 -> 0
Epoch 1 by hand: (0, 0) gives z = 0 so y = 1, but t = 0: error -1, so b = -1 (w stays, because x = 0).
A Perceptron for the 12 Students
MODULE 2 · REAL DATA
CODE: L26/STUDENTS_PERC.PY
REAL OUTPUT
- from sklearn.linear_model import Perceptron
- from sklearn.preprocessing import StandardScaler
- sc = StandardScaler().fit(X)
- m = Perceptron(random_state=0).fit(sc.transform(X), y)
- weights [1.32 0.69] bias 0.00
- training accuracy 1.00 epochs 7
- new student 8 h, 60 % -> pass
- new student 4 h, 85 % -> fail
- new student 11 h, 60 % -> pass
- new student 7 h, 72 % -> pass
Boundary: always a straight line w1·x1 + w2·x2 + b = 0 - the tree's boundary was a step on one feature.
Lesson: both models are 100 % on 12 students but disagree on new ones. More data, and a test set, decide which to trust.
Slide 12
MODULE 3
What One Perceptron Cannot Do
Linear separability and XOR
AND and OR Yes - XOR No
MODULE 3 · LINEAR SEPARABILITY
REAL OUTPUT: L26/GATES.PY (LR 1, AT MOST 20 EPOCHS)
- AND: target [0, 0, 0, 1] output [0, 0, 0, 1] w = (2, 1) b = -3 learned in 6 epochs
- OR : target [0, 1, 1, 1] output [0, 1, 1, 1] w = (1, 1) b = -1 learned in 4 epochs
- XOR: target [0, 1, 1, 0] output [1, 1, 0, 0] w = (-1, 0) b = 0 still 4 mistakes after 20 epochs
The Way Out: More Than One Layer
MODULE 3 · PREVIEW OF LECTURE 27
TWO LAYERS OF PERCEPTRONS
- hidden layer:
- h1 = OR(x1, x2)
- h2 = NAND(x1, x2)
- output layer:
- y = AND(h1, h2) = XOR
Hidden layer
Smooth activation
Back-propagation
Neurons between input and output build new features.
Sigmoid instead of step, so errors have a slope.
Send the error backwards to every weight (L27).
You have used it: P02 built a perceptron; P03's MLPClassifier is exactly this - many layers of neurons.
Activity: Neurons and Weight Updates by Hand
CLASS ACTIVITY · PAIRS · 8 MIN
Pairs, 6 min + 2 min check. Step function: 1 if z >= 0, else 0. Tubewell neuron: w = (0.6, 0.4, -0.9), b = -0.5.
Tubewell neuron, inputs (dry, hot, rain): (1, 0, 0), (0, 1, 0), (1, 1, 1). Output of each?
Check that w = (-1, -1), b = 1.5 computes NAND.
w = (0, 0), b = 0, lr = 1, example x = (1, 1), t = 1. What changes?
Is the data 'pass if hours + attendance/10 >= 14' linearly separable? Why?
w = (1, 1), b = -1, lr = 1, example x = (0, 1), t = 0. New w and b?
Why can the perceptron rule not train the hidden neurons of the XOR network?
Common Mistakes
WATCH OUT
WRONG IDEA
CORRECT IDEA
✗ The perceptron changes weights after every example.
✓ Only when it makes a MISTAKE (t - y is not 0).
✗ If x = 0 its weight still changes.
✓ lr · error · 0 = 0: only weights of active inputs change.
✗ A perceptron can learn any pattern.
✓ Only linearly separable ones - not XOR.
✗ The bias is just another input value.
✓ It is a learned weight on a constant input 1.
✗ One correct epoch means the weights are unique.
✓ Many weight sets work; start and order change them.
Recap Quiz
CHECK YOUR UNDERSTANDING · 3 MIN
- Q1. w = (2, 1), b = -3, x = (1, 1). The step output is:
- (a) 0 (b) 1 (c) 3 (d) -3
- Q2. In the perceptron rule, when y = t the weights:
- (a) Increase (b) Decrease (c) Do not change (d) Reset to 0
- Q3. Which gate can a single perceptron NOT learn?
- (a) AND (b) OR (c) NAND (d) XOR
- Q4. Who showed the XOR limit of perceptrons (1969)?
- (a) Minsky and Papert (b) Rosenblatt (c) Quinlan (d) Turing
Homework and Next Lecture
BEFORE THE NEXT CLASS
- Task 1: Train a perceptron for NOT (one input x, t = 1 for x = 0 and t = 0 for x = 1) with lr = 1, w = 0, b = 0. Show each update.
- Task 2: Train OR by hand (lr = 1, start w = (0, 0), b = 0, order (0,0), (0,1), (1,0), (1,1)). Show the weights after each example until an epoch has no mistakes.
- Task 3: Should a farmer sow cotton? Inputs (price high, rain expected, seed available) = (1, 0, 1), w = (0.5, 0.3, 0.4), b = -0.6. Compute z and the output.
NEXT LECTURE
- Lecture 27:
- Multilayer Networks and Back-propagation