1. Objectives
- Build and read a 10 x 10 confusion matrix and find the most confused pair of digits.
- Compute precision and recall per class with classification_report.
- Display the wrongly recognised images and explain the errors.
- Test the network on noisy images and improve it by training with noisy copies (data augmentation).
2. Word Bank
Word | Simple meaning | Real-world example |
|---|---|---|
Confusion matrix | Table of true class (rows) against predicted class (columns) | Row 8, column 1: an 8 read as a 1 |
Diagonal | Cells where true = predicted (correct answers) | cm[3, 3] |
Precision | Of all images PREDICTED as a digit, how many really are it | Precision of 1 |
Recall | Of all images that REALLY are a digit, how many were found | Recall of 8 |
Noise | Random changes to the pixels, like dust or a bad scan | Smudged cheque |
Data augmentation | Adding changed copies of training images so the network learns to ignore the changes | Noisy copies |
3. Background
Accuracy is not the whole story
In P05a the network read 98% of the test digits correctly. Which 2% did it get wrong? A bank reading cheques must know whether it confuses 1 and 7 (a costly mistake). The confusion matrix answers this: row = true digit, column = predicted digit. Correct answers lie on the diagonal; every other non-zero cell is a particular kind of mistake (as in P03, but now with 10 classes).
Precision and recall
For one digit, precision asks 'when the network says 8, is it right?' and recall asks 'of all real 8s, how many did it find?'. A digit can have high precision but low recall.
Real scans are noisy
Old forms, smudged ink and cheap scanners add noise. A network trained only on clean images may fail on dirty ones. Training on extra noisy copies (data augmentation) makes it more robust (strong).
Data files for this lab
No data file is needed - the programs make their own data.
4. Time Plan (50 minutes)
Minutes | Activity | Part |
|---|---|---|
0-5 | Background: which digits does the network confuse? | 1-3 |
5-10 | Guided warm-up: confusion matrix and report | Part A |
10-13 | Predict the output | Part B |
13-17 | Bug hunt | Part C |
17-47 | Graded tasks 1-3 | Part D |
47-50 | Wrap-up; post-lab questions set as homework | Part E |
Part A - Guided Warm-up (type, run, observe)
A1. The confusion matrix
Same network and split as P05a. np.fill_diagonal sets the correct cells to 0 so we can find the biggest mistake.
import numpy as np from sklearn.datasets import load_digits from sklearn.model_selection import train_test_split from sklearn.neural_network import MLPClassifier digits = load_digits() X, y = digits.data / 16, digits.target X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y) net = MLPClassifier(hidden_layer_sizes=(32,), max_iter=1000, random_state=1).fit(X_tr, y_tr) pred = net.predict(X_te) from sklearn.metrics import confusion_matrix cm = confusion_matrix(y_te, pred) print(cm) off = cm.copy(); np.fill_diagonal(off, 0) t, p = np.unravel_index(off.argmax(), off.shape) print("wrong:", off.sum(), "of", len(y_te), "| most confused: true", t, "read as", p, "-", off[t, p], "times") |
Output [[44 0 0 0 1 0 0 0 0 0] [ 0 45 0 0 0 0 0 0 1 0] [ 0 1 43 0 0 0 0 0 0 0] [ 0 0 0 46 0 0 0 0 0 0] [ 0 0 0 0 45 0 0 0 0 0] [ 0 0 0 0 0 45 0 0 0 1] [ 0 0 0 0 0 0 44 0 1 0] [ 0 0 0 0 1 0 0 44 0 0] [ 0 2 0 0 0 0 1 0 40 0] [ 0 0 0 0 0 0 0 0 0 45]] wrong: 9 of 450 | most confused: true 8 read as 1 - 2 times |
A2. Precision and recall for every digit
classification_report gives precision, recall and f1 (their balance) for each class.
import numpy as np from sklearn.datasets import load_digits from sklearn.model_selection import train_test_split from sklearn.neural_network import MLPClassifier digits = load_digits() X, y = digits.data / 16, digits.target X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y) net = MLPClassifier(hidden_layer_sizes=(32,), max_iter=1000, random_state=1).fit(X_tr, y_tr) pred = net.predict(X_te) from sklearn.metrics import classification_report print(classification_report(y_te, pred, digits=3)) |
Output precision recall f1-score support 0 1.000 0.978 0.989 45 1 0.938 0.978 0.957 46 2 1.000 0.977 0.989 44 3 1.000 1.000 1.000 46 4 0.957 1.000 0.978 45 5 1.000 0.978 0.989 46 6 0.978 0.978 0.978 45 7 1.000 0.978 0.989 45 8 0.952 0.930 0.941 43 9 0.978 1.000 0.989 45 accuracy 0.980 450 macro avg 0.980 0.980 0.980 450 weighted avg 0.980 0.980 0.980 450 |
Part B - Predict the Output
Write your prediction BEFORE running the code. Then run it and correct yourself in a different colour.
B1.
from sklearn.metrics import confusion_matrix print(confusion_matrix([0, 1, 1, 2, 2, 2], [0, 1, 2, 2, 2, 1])) |
My prediction |
B2.
tp, fp, fn = 40, 2, 5 # for digit 8 print(round(tp / (tp + fp), 3), round(tp / (tp + fn), 3)) |
My prediction |
B3.
import numpy as np img = np.array([0.0, 0.5, 1.0]) noisy = np.clip(img + np.array([-0.2, 0.3, 0.4]), 0, 1) print(noisy) |
My prediction |
Part C - Bug Hunt
Each program has ONE bug. Some crash with an error message; others run but give a wrong answer (silent logic error - the dangerous kind). Find the bug, explain it and write the fix.
C1.
import numpy as np from sklearn.datasets import load_digits from sklearn.model_selection import train_test_split from sklearn.neural_network import MLPClassifier digits = load_digits() X, y = digits.data / 16, digits.target X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y) net = MLPClassifier(hidden_layer_sizes=(32,), max_iter=1000, random_state=1).fit(X_tr, y_tr) pred = net.predict(X_te) from sklearn.metrics import confusion_matrix cm = confusion_matrix(pred, y_te) # rows should be the TRUE digits print("real 8s read as 1:", cm[8, 1]) |
It runs and prints real 8s read as 1: 1 |
Bug, why it happens, and the fixed line(s) |
C2.
import numpy as np from sklearn.datasets import load_digits from sklearn.model_selection import train_test_split from sklearn.neural_network import MLPClassifier digits = load_digits() X, y = digits.data / 16, digits.target X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y) net = MLPClassifier(hidden_layer_sizes=(32,), max_iter=1000, random_state=1).fit(X_tr, y_tr) pred = net.predict(X_te) import matplotlib.pyplot as plt plt.imshow(X_te[0], cmap="gray_r") # show the first test digit plt.savefig("one.png") |
Bug, why it happens, and the fixed line(s) |
C3.
import numpy as np from sklearn.datasets import load_digits from sklearn.model_selection import train_test_split from sklearn.neural_network import MLPClassifier digits = load_digits() X, y = digits.data / 16, digits.target X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y) net = MLPClassifier(hidden_layer_sizes=(32,), max_iter=1000, random_state=1).fit(X_tr, y_tr) pred = net.predict(X_te) rng = np.random.default_rng(0) noisy = X_te + rng.normal(0, 0.3, X_te.shape) # add scanner noise print("noisy pixel range:", round(noisy.min(), 2), "to", round(noisy.max(), 2)) print("accuracy on noisy images:", round(net.score(noisy, y_te), 3)) |
It runs and prints noisy pixel range: -1.35 to 1.98 accuracy on noisy images: 0.8 |
Bug, why it happens, and the fixed line(s) |
Part D - Graded Tasks (10 marks)
Task | Level | Title | Marks |
|---|---|---|---|
T1 | Easy | Show every mistake | 3 |
T2 | Medium | How strong is the network against noise? | 3 |
T3 | Challenge | Data augmentation: train with noisy copies | 4 |
Total | 10 |
Task 1 (Easy, 3 marks): Show every mistake
Using the A1 network, find all wrongly recognised test images (pred != y_te). Print how many there are and draw them in one row with the title 'true T / pred P' (save task1.png). Look at the pictures: are the mistakes understandable for a human?
Sample run (your output must match):
Output wrong images: 9 | indexes: [ 25 62 200 229 239 242 255 313 329] Saved task1.png (title = true / predicted) |
Your code (or attach the file) |
Task 2 (Medium, 3 marks): How strong is the network against noise?
Add Gaussian noise with standard deviation 0.0, 0.1, 0.2 and 0.3 to the TEST images (use np.random.default_rng(0) for each level and clip to 0-1). Print the accuracy for each level and say how much accuracy is lost at 0.3.
Sample run (your output must match):
Output noise 0.0: accuracy 0.980 noise 0.1: accuracy 0.958 noise 0.2: accuracy 0.918 noise 0.3: accuracy 0.827 |
Your code (or attach the file) |
Task 3 (Challenge, 4 marks): Data augmentation: train with noisy copies
Make a noisy copy of the TRAINING images (noise 0.2, np.random.default_rng(1), clipped) and train a new network on the clean + noisy images together (twice as many rows; repeat y_tr). Compare the old and new networks on test noise 0.0, 0.2 and 0.3 in one table. Explain the result and one real use.
Sample run (your output must match):
Output training rows: 1347 -> 2694 noise clean-trained augmented 0.0 0.980 0.978 0.2 0.918 0.942 0.3 0.827 0.840 |
Your code (or attach the file) |
Part E - Post-Lab Questions (finish as homework if time runs out)
E1. In a confusion matrix, what do the rows, the columns and the diagonal mean?
Your answer |
E2. Differentiate precision and recall with a digit example.
Your answer |
E3. What is data augmentation and why did it help with noisy images?
Your answer |
Marking Rubric (applied to Part D)
Criterion | Full | Partial | None |
|---|---|---|---|
Code runs without errors | Runs fully | Small fixable error | Does not run |
Correct confusion matrix / noise handling | All correct | One slip | Several wrong |
Results printed and images drawn | All present | Some missing | None |
Explanations (mistakes, robustness, augmentation) | Clear and correct | Partly correct | Missing |
Submission
- Make a folder RollNo_P05b with task1.py, task2.py, task3.py (or P05b.ipynb) and task1.png.
- Upload the zipped folder to the LMS / Google Classroom by the end of the day.
- Copy Task 2 and Task 3 code and output into your Lab Book (Practical 05b page).
- Use random_state and the random generators exactly as given so your numbers match.