Learning Outcomes
SLO | After this lecture you will be able to... | CLO | Bloom |
|---|---|---|---|
SLO 20.1 | Compute joint, marginal and conditional probabilities from a table and test independence | CLO3 | C3 Apply |
SLO 20.2 | Apply Bayes' rule to update a belief from evidence and explain the effect of the prior | CLO3 | C3 Apply |
SLO 20.3 | Apply a Naive Bayes classifier to a small example and explain its independence assumption | CLO3 | C3 Apply |
Note: all medical and SMS data in this lecture are made up for teaching; they are realistic, not official figures. |
1. Probability Basics
§1.1 Why AI needs probability
Slide L20-S06
The world is uncertain: causes are hidden (a doctor sees fever, not the virus), sensors are noisy, and rules have too many exceptions to list. Probability gives a consistent number from 0 (impossible) to 1 (certain). P(not A) = 1 - P(A). A random variable takes values with probabilities, e.g. Weather in {sunny, rain}.
§1.2 Joint, marginal and conditional probability
Slide L20-S07
fever no fever total dengue 45 5 50 no dengue 190 760 950 total 235 765 1000 P(dengue) = 0.050 P(fever) = 0.235 P(dengue AND fever) = 0.045 P(dengue | fever) = 0.045 / 0.235 = 0.191 P(fever | dengue) = 0.045 / 0.050 = 0.900 P(dengue) x P(fever) = 0.0118, not 0.045 Independent? no |
Kind | Meaning | Formula | Value |
|---|---|---|---|
Joint | Both together | P(D and F) | 0.045 |
Marginal | One variable alone (row/column total) | P(D) | 0.05 |
Conditional | A when B is known | P(D | F) = P(D and F) / P(F) | 0.191 |
Careful: P(fever | dengue) = 0.90 but P(dengue | fever) = 0.19 - the order matters. |
§1.3 Rules and independence
Slide L20-S08
Rule | Formula |
|---|---|
Product rule | P(A and B) = P(A | B) P(B) = P(B | A) P(A) |
Sum rule (marginalisation) | P(A) = P(A and B) + P(A and not B) |
Independence | P(A | B) = P(A), i.e. P(A and B) = P(A) P(B) |
2. Bayes' Rule
§2.1 The formula
Slide L20-S10
P(H | E) = P(E | H) x P(H) / P(E) P(E) = P(E | H) P(H) + P(E | not H) P(not H) |
Name | Symbol | Meaning |
|---|---|---|
Prior | P(H) | Belief before the evidence |
Likelihood | P(E | H) | How often the evidence appears if H is true |
Evidence | P(E) | How often the evidence appears overall |
Posterior | P(H | E) | Belief after the evidence |
§2.2 The dengue test
Slide L20-S11
Prior P(D) = 0.02 Sensitivity P(+|D) = 0.9 False positive P(+|not D) = 0.05 P(+) = 0.9 x 0.02 + 0.05 x 0.98 = 0.067 P(D|+) = 0.9 x 0.02 / 0.067 = 0.269 Second positive test: 0.269 -> 0.869 Effect of the prior (same test): prior 0.001 -> P(D|+) = 0.018 prior 0.02 -> P(D|+) = 0.269 prior 0.2 -> P(D|+) = 0.818 prior 0.5 -> P(D|+) = 0.947 |
Frequency view: of 1,000 people, 20 have dengue (18 test positive) and 980 are healthy (49 test positive). 67 positives, 18 really ill: 18 / 67 = 0.27.
§2.3 The prior and repeated tests
Slide L20-S12
The same test means very different things in different groups: P(D | +) is 0.018 in a random survey (prior 0.001) but 0.947 in an outbreak ward (prior 0.5). Ignoring the prior is the base-rate fallacy. Bayes is sequential: the posterior becomes the next prior (0.02 -> 0.269 -> 0.869 after two positive tests).
3. Naive Bayes
§3.1 The classifier
Slide L20-S14
score(C) = P(C) x P(e1 | C) x ... x P(en | C); choose the class with the larger score. The naive assumption: the clues are independent once the class is known. Laplace smoothing: P(w | C) = (count + 1) / (words in C + vocabulary size), so an unseen word never gives probability 0.
§3.2 Worked example: SMS spam filter
Slide L20-S15
spam = ["win free prize now", "free mobile package offer", "claim your prize now", "free recharge win"] ham = ["class at nine tomorrow", "send me the notes", "free period after lunch", "see you at the bus stop", "prize distribution ceremony tomorrow", "call me now"] msg = "free prize now" vocab = sorted(set(" ".join(spam + ham).split())) cs, ch = Counter(" ".join(spam).split()), Counter(" ".join(ham).split()) ns, nh, V = sum(cs.values()), sum(ch.values()), len(vocab) prior_s, prior_h = len(spam) / 10, len(ham) / 10 print(f"Training: {len(spam)} spam, {len(ham)} ham; vocabulary {V} words; spam words {ns}, ham words {nh}") ps, ph = prior_s, prior_h for w in msg.split(): a, b = (cs[w] + 1) / (ns + V), (ch[w] + 1) / (nh + V) print(f" P('{w}'|spam) = ({cs[w]}+1)/({ns}+{V}) = {a:.4f} P('{w}'|ham) = ({ch[w]}+1)/({nh}+{V}) = {b:.4f}") ps *= a; ph *= b print(f"score spam = {ps:.3e} score ham = {ph:.3e}") print(f"P(spam | '{msg}') = {ps / (ps + ph):.3f}") |
Training: 4 spam, 6 ham; vocabulary 28 words; spam words 15, ham words 25 P('free'|spam) = (3+1)/(15+28) = 0.0930 P('free'|ham) = (1+1)/(25+28) = 0.0377 P('prize'|spam) = (2+1)/(15+28) = 0.0698 P('prize'|ham) = (1+1)/(25+28) = 0.0377 P('now'|spam) = (2+1)/(15+28) = 0.0698 P('now'|ham) = (1+1)/(25+28) = 0.0377 score spam = 1.811e-04 score ham = 3.224e-05 P(spam | 'free prize now') = 0.849 scikit-learn MultinomialNB: P(spam) = 0.849 'see you after class': P(spam) = 0.088 'win a free recharge': P(spam) = 0.937 |
§3.3 Probability vs certainty factors
Slide L20-S16
Certainty factors (L19) | Probability | |
|---|---|---|
Scale | -1 to +1 | 0 to 1 |
Combining | a + b(1 - a), heuristic | Exact rules |
Numbers from | Expert judgement | Data or experts |
Problem | Can be inconsistent | Needs many numbers -> Bayesian networks (L21) |
4. Common Mistakes
Wrong idea | Correct idea |
|---|---|
P(A | B) = P(B | A). | 0.19 vs 0.90 in the clinic. |
A positive 90% test means 90% ill. | Depends on the prior. |
P(A and B) = P(A)P(B) always. | Only if independent. |
Naive Bayes believes words are independent. | It assumes it to simplify. |
An unseen word gets probability 0. | Use Laplace smoothing. |
Word Bank
Word | Simple meaning | Example |
|---|---|---|
Probability | Number from 0 to 1 for how likely | P(rain) = 0.3 |
Conditional probability | Probability when something is known | P(dengue | fever) |
Independent | One does not change the other | Two coin tosses |
Prior | Belief before evidence | 2% have dengue |
Posterior | Belief after evidence | 27% after a + test |
Base-rate fallacy | Forgetting the prior | Thinking + test = 90% ill |
Laplace smoothing | Add 1 to every count | (0+1)/(15+28) |
Self-Check Questions
- From the clinic table find P(fever | no dengue). (see §1.2)
- State the product and sum rules. (see §1.3)
- Name the four parts of Bayes' rule. (see §2.1)
- Repeat the dengue calculation with a prior of 0.1. (see §2.2)
- Why does the prior change the answer so much? (see §2.3)
- Why is Laplace smoothing needed? (see §3.1)
Reading: Russell & Norvig (4e) 12.1-12.6 (quantifying uncertainty, Bayes' rule, naive Bayes).