Nexus Learn

Lecture 25: Decision Trees

Lecture Slides CS-511 Artificial Intelligence 7 min read Updated 8 Nov 2026

↓ Download slides (PowerPoint)

Recap of Lecture 24 and Homework Check

LAST LECTURE · 5 MIN

Today: k-NN answers but cannot explain WHY. Can a model give its reasons, like a doctor asking question after question?

Today's Plan (50 minutes)

ROADMAP

0-5

Recap of Lecture 24 and homework check

S02

5-8

Plan and learning outcomes

S03-S04

8-15

Module 1: What is a decision tree?

S05-S07

15-27

Module 2: Choosing the best question - entropy and information gain

S08-S11

27-35

Module 3: Trees in Python, overfitting and pruning

S12-S14

35-43

Class activity: entropy, gain and reading a tree

S15

43-48

Common mistakes and recap quiz

S16-S17

48-50

Homework and next lecture

S18

Learning Outcomes (SLOs)

BY THE END OF THIS LECTURE YOU WILL BE ABLE TO

SLO 25.1

Explain the parts of a decision tree and use a tree to classify a new example

→ CLO3 · C2 Understand

SLO 25.2

Calculate entropy and information gain to choose the best splitting attribute (ID3)

→ CLO3 · C3 Apply

SLO 25.3

Build a decision tree with scikit-learn and control overfitting with the tree depth

→ CLO3 · C3 Apply

  • Course Learning Outcome(s) served
  • CLO3: Apply various techniques and algorithms of AI used in general problem solving, optimization problems, constraint satisfaction problems, and game programming.

Slide 5

MODULE 1

What Is a Decision Tree?

Twenty questions, learned from data

A Tree of Questions

MODULE 1 · PARTS OF A TREE

CLASSIFYING A NEW DAY

Not used: temperature - the tree keeps only useful questions.

  • New day: rain, hot, high humidity, strong wind
  • Outlook? rain -> Wind? strong -> no (match off)

Trees Are Everywhere - and They Are Rules

MODULE 1 · WHY TREES ARE POPULAR

Bank loan

Hospital triage

Mango export

Job? -> own house? -> approve / reject

Fever? -> breathing problem? -> emergency / wait

Colour? -> size? -> grade A / B

THE TREE AS FIVE IF-THEN RULES

  • IF outlook = overcast THEN play
  • IF outlook = sunny AND humidity = high THEN no play
  • IF outlook = sunny AND humidity = normal THEN play
  • IF outlook = rain AND wind = strong THEN no play
  • IF outlook = rain AND wind = weak THEN play

One path = one rule. Each root-to-leaf path is an IF-THEN rule.

Two families: ID3 (Quinlan, 1986) uses entropy; CART (Breiman, 1984) uses Gini - scikit-learn uses CART.

Slide 8

MODULE 2

Choosing the Best Question

Entropy and information gain

Entropy: How Mixed Is a Group?

MODULE 2 · ENTROPY

ENTROPY (SHANNON, 1948)

REAL OUTPUT: L25/ENTROPY_DEMO.PY

H(S) = - Σ p_i · log2(p_i)

  • 14 yes, 0 no entropy = 0.000 bits
  • 7 yes, 7 no entropy = 1.000 bits
  • 9 yes, 5 no entropy = 0.940 bits
  • 3 yes, 1 no entropy = 0.811 bits
  • Gini of 9 yes, 5 no = 0.459

BY HAND

  • 9 yes, 5 no: p(yes) = 9/14 = 0.643
  • p(no) = 5/14 = 0.357
  • H = -0.643·log2(0.643) - 0.357·log2(0.357)
  • = 0.410 + 0.530 = 0.940 bits

Goal of a split: make the groups below as PURE as possible - entropy as LOW as possible.

Gini index (CART): 1 - Σ p_i² = 0.459 for 9 yes, 5 no. Also 0 when pure.

Information Gain: Which Question First?

MODULE 2 · INFORMATION GAIN

INFORMATION GAIN

Gain(S, A) = H(S) - Σ (|S_v| / |S|) · H(S_v)

REAL OUTPUT: L25/GAINS.PY

BY HAND

  • Root: 9 yes, 5 no -> entropy 0.940
  • outlook = sunny 2 yes 3 no entropy 0.971
  • outlook = overcast 4 yes 0 no entropy 0.000
  • outlook = rain 3 yes 2 no entropy 0.971
  • Gain(outlook ) = 0.247
  • Gain(temperature) = 0.029
  • Gain(humidity ) = 0.152
  • Gain(wind ) = 0.048
  • Gain(outlook) = 0.940 - ( 5/14·0.971
  • + 4/14·0.000
  • + 5/14·0.971 )
  • = 0.940 - 0.694 = 0.247

Winner: Outlook (0.247) - it becomes the root.

Why outlook? 'overcast' alone gives a pure group: 4 days, all played.

Growing the Whole Tree (ID3)

MODULE 2 · THE ALGORITHM

ID3 (QUINLAN, 1986)

REAL OUTPUT: L25/GROW.PY

  • ID3(examples, attributes):
  • if all examples have the same class: return a leaf
  • if no attributes left: return a majority-class leaf
  • A = attribute with the highest information gain
  • for each value v of A:
  • add branch A = v -> ID3(examples with A = v,
  • attributes - A)
  • Gains inside outlook = sunny (5 days):
  • temperature 0.571, humidity 0.971, wind 0.020
  • outlook = overcast: yes
  • outlook = rain:
  • wind = strong: no
  • wind = weak: yes
  • outlook = sunny:
  • humidity = high: no
  • humidity = normal: yes

Pure node

No attributes left

Depth limit

All one class - make a leaf.

Leaf with the majority class.

Stop early to avoid overfitting.

ID3 is greedy (L13): it takes the best question now and never goes back.

Slide 12

MODULE 3

Decision Trees in Python

scikit-learn, numbers and overfitting

A Tree for the 12 Students

MODULE 3 · SCIKIT-LEARN

CODE: L25/STUDENTS_TREE.PY

REAL OUTPUT

  • from sklearn.tree import DecisionTreeClassifier, export_text
  • m = DecisionTreeClassifier(criterion="entropy", random_state=0)
  • m.fit(X, y)
  • print(export_text(m, feature_names=["hours", "attendance"]))
  • m.predict([[8, 60]])
  • |--- attendance <= 72.50
  • | |--- class: fail
  • |--- attendance > 72.50
  • | |--- class: pass
  • new student 8 h, 60 % -> fail
  • new student 4 h, 85 % -> pass
  • new student 11 h, 60 % -> fail
  • Numbers: the tree asks 'attendance <= 72.5?'
  • One question splits all 12 perfectly.
  • No scaling needed - one feature per question.

Discuss: the tree ignored study hours completely. With only 12 students, a very simple rule fits - is it fair?

Overfitting and Pruning

MODULE 3 · HOW DEEP?

REAL OUTPUT: L25/OVERFIT.PY (300 LOAN APPLICANTS)

  • Depth 3: test 0.83 - the best.
  • No limit: train 1.00 but test 0.74.
  • Overfitting: the deep tree memorised noisy cases.
  • Pruning: max_depth, min_samples_leaf, ccp_alpha.
  • depth leaves train test
  • 1 2 0.74 0.79
  • 2 4 0.80 0.74
  • 3 8 0.86 0.83
  • 5 24 0.92 0.76
  • 8 41 0.98 0.76
  • None 50 1.00 0.74

Preview: a random forest = many trees voting - more accurate, less readable.

Activity: Entropy, Gain and Reading a Tree

CLASS ACTIVITY · PAIRS · 8 MIN

Pairs, 6 min + 2 min check. Calculators allowed. Use the cricket tree of S06 and log2 values from the board.

Entropy of a group with 5 yes, 5 no? With 8 yes, 0 no?

Classify: rain, hot, high humidity, strong wind.

Gini index of 9 yes, 5 no.

From the overfit table (S14), which depth would you choose and why?

Humidity splits the 14 days into high (3 yes, 4 no) and normal (6 yes, 1 no). Entropy of each and Gain(humidity)?

Why does a decision tree not need feature scaling, but k-NN does?

Common Mistakes

WATCH OUT

WRONG IDEA

CORRECT IDEA

✗ High entropy is good.

✓ We want LOW entropy (pure groups) after a split.

✗ Gain = entropy of the biggest branch.

✓ Use the WEIGHTED average of all branches.

✗ The deepest tree is the best tree.

✓ Deep trees overfit; check test accuracy.

✗ Every attribute must appear in the tree.

✓ Useless attributes (temperature) are left out.

✗ A tree finds the best tree overall.

✓ ID3 is greedy: best question at each step only.

Recap Quiz

CHECK YOUR UNDERSTANDING · 3 MIN

  • Q1. The entropy of a pure node is:
  • (a) 0 (b) 1 (c) 0.5 (d) 0.940
  • Q2. ID3 chooses the attribute with the:
  • (a) Lowest gain (b) Highest information gain (c) Most values (d) Highest entropy
  • Q3. A leaf node holds:
  • (a) A question (b) The entropy (c) The predicted class (d) The training data
  • Q4. Train 1.00, test 0.74 for a very deep tree shows:
  • (a) Underfitting (b) Feature scaling (c) Pruning (d) Overfitting

Homework and Next Lecture

BEFORE THE NEXT CLASS

  • Task 1: Entropy of: 6 yes 2 no; 4 yes 4 no; 1 yes 7 no.
  • Task 2: 8 loan applicants (job, own house -> A approved / R rejected): govt yes A, govt yes A, private yes A, private yes A, govt no A, private no R, private no R, private no R. Root entropy, Gain(job), Gain(own house), and the full ID3 tree.
  • Task 3: Use the cricket tree to classify: (a) sunny, cool, high, strong (b) overcast, hot, high, strong (c) rain, mild, normal, weak.

NEXT LECTURE

  • Lecture 26:
  • Neural Networks and the Perceptron

← All CS-511 material