Nexus Learn

Practical 5B: Pattern Recognition II: Confusion Matrix, Errors and Noisy Images

Lab Practical CS-511 Artificial Intelligence 8 min read Updated 8 Nov 2026
On this page
  1. 1. Objectives
  2. 2. Word Bank
  3. 3. Background
  4. 4. Time Plan (50 minutes)
  5. Part A - Guided Warm-up (type, run, observe)
  6. Part B - Predict the Output
  7. Part C - Bug Hunt
  8. Part D - Graded Tasks (10 marks)
  9. Part E - Post-Lab Questions (finish as homework if time runs out)
  10. Marking Rubric (applied to Part D)
  11. Submission

1. Objectives

  1. Build and read a 10 x 10 confusion matrix and find the most confused pair of digits.
  2. Compute precision and recall per class with classification_report.
  3. Display the wrongly recognised images and explain the errors.
  4. Test the network on noisy images and improve it by training with noisy copies (data augmentation).

2. Word Bank

Word

Simple meaning

Real-world example

Confusion matrix

Table of true class (rows) against predicted class (columns)

Row 8, column 1: an 8 read as a 1

Diagonal

Cells where true = predicted (correct answers)

cm[3, 3]

Precision

Of all images PREDICTED as a digit, how many really are it

Precision of 1

Recall

Of all images that REALLY are a digit, how many were found

Recall of 8

Noise

Random changes to the pixels, like dust or a bad scan

Smudged cheque

Data augmentation

Adding changed copies of training images so the network learns to ignore the changes

Noisy copies

3. Background

Accuracy is not the whole story

In P05a the network read 98% of the test digits correctly. Which 2% did it get wrong? A bank reading cheques must know whether it confuses 1 and 7 (a costly mistake). The confusion matrix answers this: row = true digit, column = predicted digit. Correct answers lie on the diagonal; every other non-zero cell is a particular kind of mistake (as in P03, but now with 10 classes).

Precision and recall

For one digit, precision asks 'when the network says 8, is it right?' and recall asks 'of all real 8s, how many did it find?'. A digit can have high precision but low recall.

Real scans are noisy

Old forms, smudged ink and cheap scanners add noise. A network trained only on clean images may fail on dirty ones. Training on extra noisy copies (data augmentation) makes it more robust (strong).

Data files for this lab

No data file is needed - the programs make their own data.

4. Time Plan (50 minutes)

Minutes

Activity

Part

0-5

Background: which digits does the network confuse?

1-3

5-10

Guided warm-up: confusion matrix and report

Part A

10-13

Predict the output

Part B

13-17

Bug hunt

Part C

17-47

Graded tasks 1-3

Part D

47-50

Wrap-up; post-lab questions set as homework

Part E

Part A - Guided Warm-up (type, run, observe)

A1. The confusion matrix

Same network and split as P05a. np.fill_diagonal sets the correct cells to 0 so we can find the biggest mistake.

import numpy as np

from sklearn.datasets import load_digits

from sklearn.model_selection import train_test_split

from sklearn.neural_network import MLPClassifier

digits = load_digits()

X, y = digits.data / 16, digits.target

X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)

net = MLPClassifier(hidden_layer_sizes=(32,), max_iter=1000, random_state=1).fit(X_tr, y_tr)

pred = net.predict(X_te)

from sklearn.metrics import confusion_matrix

cm = confusion_matrix(y_te, pred)

print(cm)

off = cm.copy(); np.fill_diagonal(off, 0)

t, p = np.unravel_index(off.argmax(), off.shape)

print("wrong:", off.sum(), "of", len(y_te), "| most confused: true", t, "read as", p, "-", off[t, p], "times")

Output

[[44 0 0 0 1 0 0 0 0 0]

[ 0 45 0 0 0 0 0 0 1 0]

[ 0 1 43 0 0 0 0 0 0 0]

[ 0 0 0 46 0 0 0 0 0 0]

[ 0 0 0 0 45 0 0 0 0 0]

[ 0 0 0 0 0 45 0 0 0 1]

[ 0 0 0 0 0 0 44 0 1 0]

[ 0 0 0 0 1 0 0 44 0 0]

[ 0 2 0 0 0 0 1 0 40 0]

[ 0 0 0 0 0 0 0 0 0 45]]

wrong: 9 of 450 | most confused: true 8 read as 1 - 2 times

A2. Precision and recall for every digit

classification_report gives precision, recall and f1 (their balance) for each class.

import numpy as np

from sklearn.datasets import load_digits

from sklearn.model_selection import train_test_split

from sklearn.neural_network import MLPClassifier

digits = load_digits()

X, y = digits.data / 16, digits.target

X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)

net = MLPClassifier(hidden_layer_sizes=(32,), max_iter=1000, random_state=1).fit(X_tr, y_tr)

pred = net.predict(X_te)

from sklearn.metrics import classification_report

print(classification_report(y_te, pred, digits=3))

Output

precision recall f1-score support

0 1.000 0.978 0.989 45

1 0.938 0.978 0.957 46

2 1.000 0.977 0.989 44

3 1.000 1.000 1.000 46

4 0.957 1.000 0.978 45

5 1.000 0.978 0.989 46

6 0.978 0.978 0.978 45

7 1.000 0.978 0.989 45

8 0.952 0.930 0.941 43

9 0.978 1.000 0.989 45

accuracy 0.980 450

macro avg 0.980 0.980 0.980 450

weighted avg 0.980 0.980 0.980 450

Part B - Predict the Output

Write your prediction BEFORE running the code. Then run it and correct yourself in a different colour.

B1.

from sklearn.metrics import confusion_matrix

print(confusion_matrix([0, 1, 1, 2, 2, 2], [0, 1, 2, 2, 2, 1]))

My prediction

B2.

tp, fp, fn = 40, 2, 5 # for digit 8

print(round(tp / (tp + fp), 3), round(tp / (tp + fn), 3))

My prediction

B3.

import numpy as np

img = np.array([0.0, 0.5, 1.0])

noisy = np.clip(img + np.array([-0.2, 0.3, 0.4]), 0, 1)

print(noisy)

My prediction

Part C - Bug Hunt

Each program has ONE bug. Some crash with an error message; others run but give a wrong answer (silent logic error - the dangerous kind). Find the bug, explain it and write the fix.

C1.

import numpy as np

from sklearn.datasets import load_digits

from sklearn.model_selection import train_test_split

from sklearn.neural_network import MLPClassifier

digits = load_digits()

X, y = digits.data / 16, digits.target

X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)

net = MLPClassifier(hidden_layer_sizes=(32,), max_iter=1000, random_state=1).fit(X_tr, y_tr)

pred = net.predict(X_te)

from sklearn.metrics import confusion_matrix

cm = confusion_matrix(pred, y_te) # rows should be the TRUE digits

print("real 8s read as 1:", cm[8, 1])

It runs and prints

real 8s read as 1: 1

Bug, why it happens, and the fixed line(s)

C2.

import numpy as np

from sklearn.datasets import load_digits

from sklearn.model_selection import train_test_split

from sklearn.neural_network import MLPClassifier

digits = load_digits()

X, y = digits.data / 16, digits.target

X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)

net = MLPClassifier(hidden_layer_sizes=(32,), max_iter=1000, random_state=1).fit(X_tr, y_tr)

pred = net.predict(X_te)

import matplotlib.pyplot as plt

plt.imshow(X_te[0], cmap="gray_r") # show the first test digit

plt.savefig("one.png")

Bug, why it happens, and the fixed line(s)

C3.

import numpy as np

from sklearn.datasets import load_digits

from sklearn.model_selection import train_test_split

from sklearn.neural_network import MLPClassifier

digits = load_digits()

X, y = digits.data / 16, digits.target

X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)

net = MLPClassifier(hidden_layer_sizes=(32,), max_iter=1000, random_state=1).fit(X_tr, y_tr)

pred = net.predict(X_te)

rng = np.random.default_rng(0)

noisy = X_te + rng.normal(0, 0.3, X_te.shape) # add scanner noise

print("noisy pixel range:", round(noisy.min(), 2), "to", round(noisy.max(), 2))

print("accuracy on noisy images:", round(net.score(noisy, y_te), 3))

It runs and prints

noisy pixel range: -1.35 to 1.98

accuracy on noisy images: 0.8

Bug, why it happens, and the fixed line(s)

Part D - Graded Tasks (10 marks)

Task

Level

Title

Marks

T1

Easy

Show every mistake

3

T2

Medium

How strong is the network against noise?

3

T3

Challenge

Data augmentation: train with noisy copies

4

Total

10

Task 1 (Easy, 3 marks): Show every mistake

Using the A1 network, find all wrongly recognised test images (pred != y_te). Print how many there are and draw them in one row with the title 'true T / pred P' (save task1.png). Look at the pictures: are the mistakes understandable for a human?

Sample run (your output must match):

Output

wrong images: 9 | indexes: [ 25 62 200 229 239 242 255 313 329]

Saved task1.png (title = true / predicted)

Your code (or attach the file)

Task 2 (Medium, 3 marks): How strong is the network against noise?

Add Gaussian noise with standard deviation 0.0, 0.1, 0.2 and 0.3 to the TEST images (use np.random.default_rng(0) for each level and clip to 0-1). Print the accuracy for each level and say how much accuracy is lost at 0.3.

Sample run (your output must match):

Output

noise 0.0: accuracy 0.980

noise 0.1: accuracy 0.958

noise 0.2: accuracy 0.918

noise 0.3: accuracy 0.827

Your code (or attach the file)

Task 3 (Challenge, 4 marks): Data augmentation: train with noisy copies

Make a noisy copy of the TRAINING images (noise 0.2, np.random.default_rng(1), clipped) and train a new network on the clean + noisy images together (twice as many rows; repeat y_tr). Compare the old and new networks on test noise 0.0, 0.2 and 0.3 in one table. Explain the result and one real use.

Sample run (your output must match):

Output

training rows: 1347 -> 2694

noise clean-trained augmented

0.0 0.980 0.978

0.2 0.918 0.942

0.3 0.827 0.840

Your code (or attach the file)

Part E - Post-Lab Questions (finish as homework if time runs out)

E1. In a confusion matrix, what do the rows, the columns and the diagonal mean?

Your answer

E2. Differentiate precision and recall with a digit example.

Your answer

E3. What is data augmentation and why did it help with noisy images?

Your answer

Marking Rubric (applied to Part D)

Criterion

Full

Partial

None

Code runs without errors

Runs fully

Small fixable error

Does not run

Correct confusion matrix / noise handling

All correct

One slip

Several wrong

Results printed and images drawn

All present

Some missing

None

Explanations (mistakes, robustness, augmentation)

Clear and correct

Partly correct

Missing

Submission

  • Make a folder RollNo_P05b with task1.py, task2.py, task3.py (or P05b.ipynb) and task1.png.
  • Upload the zipped folder to the LMS / Google Classroom by the end of the day.
  • Copy Task 2 and Task 3 code and output into your Lab Book (Practical 05b page).
  • Use random_state and the random generators exactly as given so your numbers match.

← All CS-511 material