Learning Outcomes
SLO | After this lecture you will be able to... | CLO | Bloom |
|---|---|---|---|
SLO 25.1 | Explain the parts of a decision tree and use a tree to classify a new example | CLO3 | C2 Understand |
SLO 25.2 | Calculate entropy and information gain to choose the best splitting attribute (ID3) | CLO3 | C3 Apply |
SLO 25.3 | Build a decision tree with scikit-learn and control overfitting with the tree depth | CLO3 | C3 Apply |
1. What Is a Decision Tree?
§1.1 A tree of questions
Slide L25-S06
A decision tree is a supervised learning model that classifies an example by asking a chain of questions about its features. It is learned from labelled examples. Running example: 14 Sundays of a friendly cricket match in Burewala - was it played?
Day | Outlook | Temperature | Humidity | Wind | Played? |
|---|---|---|---|---|---|
1 | sunny | hot | high | weak | no |
2 | sunny | hot | high | strong | no |
3 | overcast | hot | high | weak | yes |
4 | rain | mild | high | weak | yes |
5 | rain | cool | normal | weak | yes |
6 | rain | cool | normal | strong | no |
7 | overcast | cool | normal | strong | yes |
8 | sunny | mild | high | weak | no |
9 | sunny | cool | normal | weak | yes |
10 | rain | mild | normal | weak | yes |
11 | sunny | mild | normal | strong | yes |
12 | overcast | mild | high | strong | yes |
13 | overcast | hot | normal | weak | yes |
14 | rain | mild | high | strong | no |
Part | Meaning | In the cricket tree |
|---|---|---|
Root node | The first question | Outlook? |
Internal node | A later question | Humidity?, Wind? |
Branch | One answer to a question | sunny / overcast / rain |
Leaf | The final answer (class) | yes / no |
To classify a new day, start at the root and follow the matching branches to a leaf. Rain, hot, high humidity, strong wind: Outlook? rain -> Wind? strong -> no. Temperature is not used: the tree keeps only useful questions.
§1.2 Trees are rules
Slide L25-S07
Every root-to-leaf path is one IF-THEN rule, like the expert systems of Lecture 19 - but learned from data, not written by an expert. Uses: bank loans, hospital triage, mango export grading, predicting which mobile customers will leave.
IF outlook = overcast THEN play IF outlook = sunny AND humidity = high THEN no play IF outlook = sunny AND humidity = normal THEN play IF outlook = rain AND wind = strong THEN no play IF outlook = rain AND wind = weak THEN play |
Two families: ID3 (Ross Quinlan, 1986) chooses questions with entropy; CART (Breiman and others, 1984) uses the Gini index and is the method in scikit-learn.
2. Choosing the Best Question
§2.1 Entropy
Slide L25-S09
Entropy measures how mixed (uncertain) a group is, in bits: H(S) = - Σ p_i log2 p_i. A pure group has entropy 0; a 50/50 group of two classes has entropy 1.
14 yes, 0 no entropy = 0.000 bits 7 yes, 7 no entropy = 1.000 bits 9 yes, 5 no entropy = 0.940 bits 3 yes, 1 no entropy = 0.811 bits Gini of 9 yes, 5 no = 0.459 |
9 yes, 5 no: p(yes) = 9/14 = 0.643, p(no) = 5/14 = 0.357 H = -0.643 x log2(0.643) - 0.357 x log2(0.357) = 0.410 + 0.530 = 0.940 bits |
Gini index (CART): 1 - Σ p_i². For 9 yes, 5 no it is 0.459; also 0 when pure.
§2.2 Information gain
Slide L25-S10
Gain(S, A) = H(S) - Σ over values v of A ( |S_v| / |S| ) x H(S_v) |
Root: 9 yes, 5 no -> entropy 0.940 outlook = sunny 2 yes 3 no entropy 0.971 outlook = overcast 4 yes 0 no entropy 0.000 outlook = rain 3 yes 2 no entropy 0.971 Gain(outlook ) = 0.247 Gain(temperature) = 0.029 Gain(humidity ) = 0.152 Gain(wind ) = 0.048 |
By hand: Gain(outlook) = 0.940 - (5/14 x 0.971 + 4/14 x 0 + 5/14 x 0.971) = 0.940 - 0.694 = 0.247. Outlook has the highest gain, so it is the root - 'overcast' alone gives a pure group of 4 days.
Exam tip: use the WEIGHTED average of the branch entropies - each branch is weighted by its share of the examples. |
§2.3 Growing the whole tree: ID3
Slide L25-S11
ID3(examples, attributes): if all examples have the same class: return a leaf if no attributes left: return a majority-class leaf A = attribute with the highest information gain for each value v of A: add branch A = v -> ID3(examples with A = v, attributes - A) |
Gains inside outlook = sunny (5 days): temperature 0.571, humidity 0.971, wind 0.020 outlook = overcast: yes outlook = rain: wind = strong: no wind = weak: yes outlook = sunny: humidity = high: no humidity = normal: yes |
ID3 is recursive (it repeats itself on smaller groups) and greedy (Lecture 13): it chooses the best question now and never goes back.
3. Decision Trees in Python
§3.1 A tree for the 12 students
Slide L25-S13
For number features the tree asks 'feature <= threshold?', trying thresholds halfway between sorted values. Trees need no feature scaling, because each question looks at one feature only.
from sklearn.tree import DecisionTreeClassifier, export_text m = DecisionTreeClassifier(criterion="entropy", random_state=0).fit(X, y) print(export_text(m, feature_names=["hours", "attendance"])) |
|--- attendance <= 72.50 | |--- class: fail |--- attendance > 72.50 | |--- class: pass new student 8 h, 60 % -> fail new student 4 h, 85 % -> pass new student 11 h, 60 % -> fail |
§3.2 Overfitting and pruning
Slide L25-S14
300 loan applicants (monthly income and existing debt, thousand Rs), 15 % of the labels noisy, 70 % for training and 30 % for testing:
depth leaves train test 1 2 0.74 0.79 2 4 0.80 0.74 3 8 0.86 0.83 5 24 0.92 0.76 8 41 0.98 0.76 None 50 1.00 0.74 |
Depth 3 gives the best test accuracy (0.83). With no limit the tree reaches 100 % on the training data but only 0.74 on the test data - it has memorised the noise (overfitting). Pruning (cutting back branches) keeps the tree small: max_depth, min_samples_leaf or ccp_alpha.
Strengths | Weaknesses |
|---|---|
Easy to read and explain | Overfits if grown too deep |
No feature scaling | Unstable: small data change, new tree |
Fast; handles categories | One feature per question (boxy boundaries) |
4. Common Mistakes
Wrong idea | Correct idea |
|---|---|
High entropy is good. | We want low entropy (pure groups) after a split. |
Gain = entropy of the biggest branch. | Use the weighted average of all branches. |
The deepest tree is the best. | Deep trees overfit; check test accuracy. |
Every attribute must appear. | Useless attributes are left out. |
ID3 finds the best tree overall. | It is greedy: best question at each step. |
Word Bank
Word | Simple meaning | Example |
|---|---|---|
Decision tree | A model that asks questions to reach an answer | Outlook? -> Wind? -> no |
Root / leaf | First question / final answer | Outlook? / yes |
Entropy | How mixed a group is (0 = pure) | 9 yes 5 no = 0.940 |
Information gain | Drop in entropy after a split | Gain(outlook) = 0.247 |
Gini index | Another impurity measure (CART) | 0.459 |
Greedy | Takes the best step now, never goes back | ID3 |
Overfitting | Memorising the training data | train 1.00, test 0.74 |
Pruning | Cutting back the tree | max_depth = 3 |
Self-Check Questions
- Classify: sunny, mild, normal humidity, weak wind. (see §1.1)
- Write the tree's rule for rain days. (see §1.2)
- Entropy of 2 yes, 6 no? (see §2.1)
- Wind splits the 14 days into weak (6 yes, 2 no) and strong (3 yes, 3 no). Gain(wind)? (see §2.2)
- Why is ID3 called greedy? (see §2.3)
- Why does a tree not need feature scaling? (see §3.1)
- Which depth would you choose from the overfit table, and why? (see §3.2)
Reading: Govers Ch 8; Russell & Norvig Ch 19.3 (learning decision trees); scikit-learn user guide 'Decision Trees'.