Recap of Lecture 23 and Homework Check
LAST LECTURE · 5 MIN
Today: the fan rules came from an expert. Could a computer LEARN good rules by itself - from examples, or by trial and error?
Today's Plan (50 minutes)
ROADMAP
0-5
Recap of Lecture 23 and homework check
S02
5-8
Plan and learning outcomes
S03-S04
8-18
Module 1: What is machine learning? Three kinds of learning
S05-S07
18-27
Module 2: Supervised and unsupervised learning
S08-S11
27-35
Module 3: Reinforcement learning and Q-learning
S12-S15
35-43
Class activity: which kind of learning?
S16
43-48
Common mistakes and recap quiz
S17-S18
48-50
Homework and next lecture
S19
Learning Outcomes (SLOs)
BY THE END OF THIS LECTURE YOU WILL BE ABLE TO
SLO 24.1
Explain machine learning and differentiate supervised, unsupervised and reinforcement learning with examples
→ CLO3 · C2 Understand
SLO 24.2
Apply k-nearest neighbours to labelled data and one k-means step to unlabelled data
→ CLO3 · C3 Apply
SLO 24.3
Apply the Q-learning update rule to a small reinforcement learning problem
→ CLO3 · C3 Apply
- Course Learning Outcome(s) served
- CLO3: Apply various techniques and algorithms of AI used in general problem solving, optimization problems, constraint satisfaction problems, and game programming.
Slide 5
MODULE 1
What Is Machine Learning?
Learning the rules from data
Traditional Programming vs Machine Learning
MODULE 1 · THE IDEA
TWO WAYS TO GET A PROGRAM
- Traditional: rules + data -> answers
- Machine learning: data + answers -> rules (a model)
- model + new data -> answers
Learning means: performance on the task gets better with experience - not 'the computer becomes intelligent'.
Three Kinds of Learning
MODULE 1 · THE BIG MAP
Supervised
Unsupervised
Reinforcement
- Examples WITH answers (labels).
- Like practising with a solved past paper.
- Predict pass/fail, wheat yield.
- Examples WITHOUT answers.
- Find groups or patterns by itself.
- Group customers by spending.
- Actions and REWARDS.
- Learn by trial and error.
- A robot learns to walk; AlphaGo.
Slide 8
MODULE 2
Supervised and Unsupervised Learning
With answers and without
Supervised Learning: Learning from Labelled Examples
MODULE 2 · SUPERVISED
REAL OUTPUT: L24/SUPERVISED.PY (K-NN, K = 3)
- trained on 8 students, tested on 4
- 9 h, 75 % true pass predicted pass
- 6 h, 58 % true fail predicted fail
- 10 h, 85 % true pass predicted pass
- 5 h, 65 % true fail predicted fail
- test accuracy = 1.00
- new student 6 h, 72 % -> fail
- new student 11 h, 60 % -> fail
SUPERVISED WORKFLOW
- 1 collect labelled data 2 split train / test
- 3 train the model 4 test on unseen data
- 5 predict new cases
k-NN: find the k closest known examples and take a majority vote - 'tell me who your friends are'.
Unsupervised Learning: Finding Groups
MODULE 2 · UNSUPERVISED
REAL OUTPUT: L24/UNSUPERVISED.PY
- group 0: 6 students
- centre 9.2 h, 85.5 %
- group 1: 6 students
- centre 3.8 h, 59.7 %
- labels:
- 1 1 1 1 1 1 0 0 0 0 0 0
K-MEANS (P06A)
Note: k-means finds the two groups but cannot name them. People decide what a group means.
- repeat until no change:
- assign each point to its nearest centre
- move each centre to the mean of its points
Where You Have Already Used Them
MODULE 2 · LINKS TO THE PRACTICALS
Overfitting
Feature scaling
Memorising the training data. Always measure on a test set the model has not seen.
Attendance (50-95) swamps hours (2-12) in a distance. Scale first (StandardScaler, P04b).
Slide 12
MODULE 3
Reinforcement Learning
Learning by trial, error and reward
The Reinforcement Learning Loop
MODULE 3 · AGENT AND REWARDS
State s
Action a
Reward r
Policy
The situation now: the rider is at cell 2.
Move left or right.
-1 per move, +10 at the goal.
Best action in each state - what is learned.
THE RL LOOP
- loop:
- agent sees state s and chooses action a
- environment returns reward r and new state s'
- agent updates what it knows
Goal: the largest TOTAL reward over time, not only the next reward.
Q-Learning: The Update Rule
MODULE 3 · Q-LEARNING
Q-LEARNING UPDATE (WATKINS, 1989)
Q(s, a) <- Q(s, a) + α · [ r + γ · max Q(s', a') - Q(s, a) ]
REAL OUTPUT: L24/QUPDATE.PY - ONE UPDATE BY HAND
Q = 0.0 + 0.5 x (-1 + 0.9 x 4.0 - 0.0) = 1.30
Q-Learning on a Delivery Road
MODULE 3 · WORKED EXAMPLE
- Early trips: wander - 10 and 13 steps.
- Later trips: 4 steps, the shortest.
- Q values: 8.00 = -1 + 0.9 x 10.00.
- Policy: always move right.
REAL OUTPUT: L24/QLEARN.PY
- steps per episode: 10 13 6 4 4 6 6 4 4 5 ... 4 5 4 4 7
- learned Q table left right
- cell 0 2.71 4.58
- cell 1 0.95 6.20
- cell 2 1.67 8.00
- cell 3 3.45 10.00
- policy: 0:right 1:right 2:right 3:right
Key idea: the reward at the goal flows back, one cell at a time, through γ.
Real uses: AlphaGo (2016) beat the world Go champion; robots learning to walk; traffic-light timing.
Activity: Which Kind of Learning?
CLASS ACTIVITY · PAIRS · 8 MIN
Pairs, 6 min + 2 min check. Students (hours, attendance %) - fail: (2, 55) (3, 60) (4, 50) (5, 65) (3, 70) (6, 58); pass: (8, 80) (9, 75) (7, 90) (10, 85) (12, 88) (9, 95)
Supervised, unsupervised or RL? (a) wheat yield from past records (b) group mobile users (c) a robot learns to walk (d) spam filter
Q-learning: Q = 2, r = -1, γ = 0.9, best next = 6.2, α = 0.5. New Q?
Classification or regression? (a) electricity bill in rupees (b) is a mango ripe?
What feedback does RL get that unsupervised learning does not?
k-NN, k = 3: new student 7 h, 72 %. Nearest three and the vote? (use the 12 students)
Who writes the rules in (a) the L23 fan controller (b) a k-NN model?
Common Mistakes
WATCH OUT
WRONG IDEA
CORRECT IDEA
✗ Unsupervised learning has a teacher.
✓ No labels: it finds groups by itself.
✗ Clustering tells us what the groups mean.
✓ People name the groups afterwards.
✗ 100 % training accuracy = a good model.
✓ Measure on unseen test data (overfitting).
✗ RL is told the correct action.
✓ It only gets rewards and learns by trial.
✗ Regression means 'going backwards'.
✓ Regression predicts a number.
Recap Quiz
CHECK YOUR UNDERSTANDING · 3 MIN
- Q1. Learning from examples that include the correct answers is:
- (a) Supervised (b) Unsupervised (c) Reinforcement (d) Fuzzy
- Q2. Predicting tomorrow's temperature in °C is:
- (a) Regression (b) Classification (c) Clustering (d) Reinforcement
- Q3. k-means is an example of:
- (a) Unsupervised learning (b) Supervised learning (c) Reinforcement learning (d) A fuzzy system
- Q4. Q = 0, α = 0.5, r = 10, goal reached (no future). New Q:
- (a) 5 (b) 10 (c) 0.5 (d) 9
Homework and Next Lecture
BEFORE THE NEXT CLASS
- Task 1: k-NN with the 12 students: new student 8 h, 60 %. Find the nearest 1 and 3 students (distances) and the predictions. Is the answer fair? Why?
- Task 2: Road of S15, all Q = 0, α = 0.5, γ = 0.9. (i) Rider at cell 2 moves right (r = -1). (ii) From cell 3 moves right to the goal (r = 10). (iii) Again from cell 2 right. Q after each?
- Task 3: k-means, k = 2, on spending 2, 3, 4, 10, 11, 12 (thousand Rs) with starting centres 3 and 4. Show each iteration.
NEXT LECTURE
- Lecture 25:
- Decision Trees