Learning Outcomes
SLO | After this lecture you will be able to... | CLO | Bloom |
|---|---|---|---|
SLO 30.1 | Explain what natural language processing is, name its main tasks and applications, and explain why language is hard for computers | CLO1 | C2 Understand |
SLO 30.2 | Turn short texts into numbers by hand: tokens, stop words, a bag of words and TF-IDF | CLO1 | C3 Apply |
SLO 30.3 | Explain how naive Bayes classifies the sentiment of a review, and how modern language models such as ChatGPT go beyond word counts | CLO1 | C2 Understand |
1. What Is NLP?
§1.1 Language tasks all around us
Slide L30-S06
Natural language processing (NLP) is the part of AI that makes computers read, understand and produce human language. A natural language is one that people speak - Urdu, English, Punjabi - not a programming language.
Task | Input | Output | Example |
|---|---|---|---|
Text classification / sentiment | A message or review | Spam / not spam; positive / negative | Gmail, Daraz reviews |
Machine translation | Urdu sentence | English sentence | Google Translate |
Speech recognition | Voice | Text | Urdu voice typing |
Information extraction | A document | Names, dates, amounts | Reading a bill |
Question answering / chatbots | A question | An answer | ChatGPT, Gemini |
§1.2 Why language is hard for computers
Slide L30-S07
The main difficulty is ambiguity (one sentence, more than one meaning). People use context and world knowledge; a computer must learn them from data.
Problem | Example | Why it is hard |
|---|---|---|
Word with two meanings | bank (money / river); aam (mango / common) | Needs context |
Sentence structure | I saw a man with a telescope | Who has the telescope? |
Negation | The biryani was not bad | 'bad', but the meaning is good |
Sarcasm | Wah, kya service hai - 2 ghante intezar! | Positive words, angry meaning |
Spelling variety | acha / achha / achaa | Same word, different tokens |
Code-mixing | Service was bohat slow | Two languages in one sentence |
2. Turning Text into Numbers
§2.1 Tokens and stop words
Slide L30-S09
Cleaning: lower case and remove punctuation. Tokenisation: split the text into tokens (usually words). Stop words are very common words that carry little meaning (the, is, and; Urdu: ka, ki, hai, aur) and are often removed.
Sentence : The new mobile is very good, but the battery is NOT good! Tokens : ['the', 'new', 'mobile', 'is', 'very', 'good', 'but', 'the', 'battery', 'is', 'not', 'good'] (12) Our stop words removed: ['new', 'mobile', 'good', 'battery', 'not', 'good'] scikit-learn list removed: ['new', 'mobile', 'good', 'battery', 'good'] ('not' is in its list!) Roman Urdu, same meaning: [['khana', 'bohat', 'acha', 'tha'], ['khana', 'bht', 'achha', 'tha']] |
Warning: scikit-learn's ready-made English stop-word list contains 'not', so 'battery is NOT good' becomes 'battery good'. For sentiment analysis, keep negation words. Roman Urdu has no fixed spelling, so real systems first normalise spellings (bht -> bohat).
§2.2 Bag of words
Slide L30-S10
Make a vocabulary of every different word, then describe each text by how many times each word appears. Word order is lost: 'dog bites man' and 'man bites dog' give the same bag. df (document frequency) = number of texts that contain the word.
Reviews (stop words 'was', 'and' removed): R1: food good service good R2: food cold service slow R3: good biryani good price Bag of words (counts) word R1 R2 R3 df idf biryani 0 0 1 1 0.477 cold 0 1 0 1 0.477 food 1 1 0 2 0.176 good 2 0 2 2 0.176 price 0 0 1 1 0.477 service 1 1 0 2 0.176 slow 0 1 0 1 0.477 |
§2.3 TF-IDF
Slide L30-S11
TF = count of the word in this text IDF = log10( N / df ) N = number of texts, df = texts containing the word TF-IDF = TF x IDF |
TF-IDF of R1 (TF x IDF): food 1 x 0.176 = 0.176 good 2 x 0.176 = 0.352 service 1 x 0.176 = 0.176 TF-IDF of R3: biryani 0.477, good 0.352, price 0.477 |
A word that appears in every text gets IDF = log10(1) = 0 - it cannot tell texts apart. A rare word gets a high IDF. Example: 'good' is in 2 of 3 reviews, IDF = log10(3/2) = 0.176, and appears twice in R1, so its TF-IDF there is 0.352; 'biryani' is in one review only, IDF = 0.477.
3. Sentiment Analysis and Language Models
§3.1 Sentiment analysis with naive Bayes
Slide L30-S13
Sentiment analysis decides whether a text is positive or negative. We use the naive Bayes method of Lecture 20: P(word | class) = (count + 1) / (words in the class + vocabulary size); multiply the word probabilities for each class and compare. Made-up practice reviews, 8 positive and 8 negative, in English and Roman Urdu:
Training: 8 positive, 8 negative reviews; vocabulary 41; positive words 36, negative words 34 P('khana'|pos) = (1+1)/(36+41) = 0.0260 P('khana'|neg) = (1+1)/(34+41) = 0.0267 P('acha'|pos) = (2+1)/(36+41) = 0.0390 P('acha'|neg) = (0+1)/(34+41) = 0.0133 P('tha'|pos) = (1+1)/(36+41) = 0.0260 P('tha'|neg) = (1+1)/(34+41) = 0.0267 P(positive | 'khana acha tha') = 0.735 scikit-learn, single words (41 features): khana acha tha P(positive) = 0.735 -> positive service was slow and food cold P(positive) = 0.243 -> negative food was not good P(positive) = 0.458 -> negative biryani was not bad P(positive) = 0.070 -> negative scikit-learn, words + word pairs (92 features): khana acha tha P(positive) = 0.844 -> positive service was slow and food cold P(positive) = 0.238 -> negative food was not good P(positive) = 0.211 -> negative biryani was not bad P(positive) = 0.024 -> negative |
Adding word pairs (bigrams such as 'not good') makes 'food was not good' clearly negative (0.458 -> 0.211). But 'biryani was not bad' - a good review - is still called negative, because 'not bad' never appeared in training. Word counting has limits.
§3.2 From word counts to ChatGPT
Slide L30-S14
Era | Idea | Example |
|---|---|---|
1966 Rules | Hand-written patterns | ELIZA chatbot (MIT) |
1990s Statistics | Word counts, naive Bayes, n-grams | Spam filters |
2013 Word embeddings | Each word = a list of numbers; similar words are close | word2vec: king - man + woman = queen |
2014-16 RNN / LSTM | Read words in order (Lecture 28) | Google Translate 2016 |
2017- Transformers | Attention over all words at once | ChatGPT, Gemini, Claude |
A language model predicts the next word ('Pakistan ne match ...' -> 'jeet liya'). Trained on huge amounts of text, it learns grammar, facts and expressions such as 'not bad'. Urdu is a low-resource language (less text online), so Urdu NLP is an open research area.
4. Common Mistakes
Wrong idea | Correct idea |
|---|---|
Computers understand words like we do. | They work with numbers made from words. |
Always remove every stop word. | Keep negation words ('not', 'nahi') for sentiment. |
Bag of words keeps word order. | Order is lost. |
A high count always means an important word. | Words in every text get IDF 0. |
ChatGPT looks words up in a dictionary. | It is a language model that predicts the next word. |
Word Bank
Word | Simple meaning | Example |
|---|---|---|
NLP | AI for human language | Google Translate |
Token | One piece of text, usually a word | khana | acha | tha |
Stop word | Very common word with little meaning | the, is, ka, hai |
Vocabulary | All different words in a collection | 7 words |
Bag of words | Word counts, order ignored | R1 = 0 0 1 2 0 1 0 |
TF-IDF | Count weighted by how rare the word is | biryani 0.477 |
Sentiment | Feeling of a text: positive or negative | angry review |
Ambiguity | More than one possible meaning | bank |
Language model | Predicts the next word | ChatGPT |
Self-Check Questions
- Name four NLP tasks with a Pakistani example of each. (see §1.1)
- Give three kinds of ambiguity with examples. (see §1.2)
- Why should 'not' be kept for sentiment analysis? (see §2.1)
- Make a bag of words for 'pani pani pani' and 'pani nahi'. (see §2.2)
- 4 texts; a word is in 2 of them. Find its IDF. (see §2.3)
- Why did word pairs help with 'food was not good'? (see §3.1)
- What does a language model predict? (see §3.2)
Reading: Russell & Norvig Ch 23 (natural language processing) and Ch 24 (deep learning for NLP); Lecture 20 (naive Bayes); Lecture 28 (RNNs).