Nexus Learn

Lecture 28: Recurrent Neural Networks

Lecture Slides CS-511 Artificial Intelligence 6 min read Updated 8 Nov 2026

↓ Download slides (PowerPoint)

Recap of Lecture 27 and Homework Check

LAST LECTURE · 5 MIN

Today: an MLP sees one input and forgets it. How can a network remember yesterday's electricity use, or the start of a sentence?

Today's Plan (50 minutes)

ROADMAP

0-5

Recap of Lecture 27 and homework check

S02

5-8

Plan and learning outcomes

S03-S04

8-16

Module 1: Sequences and the recurrent neuron

S05-S07

16-27

Module 2: Memory, training through time, and LSTM

S08-S11

27-35

Module 3: An RNN on real electricity data

S12-S14

35-43

Class activity: RNN steps by hand

S15

43-48

Common mistakes and recap quiz

S16-S17

48-50

Homework and next lecture

S18

Learning Outcomes (SLOs)

BY THE END OF THIS LECTURE YOU WILL BE ABLE TO

SLO 28.1

Explain why sequence data needs memory and describe the structure of a recurrent neural network

→ CLO3 · C2 Understand

SLO 28.2

Compute the hidden state of a simple RNN over a short input sequence

→ CLO3 · C3 Apply

SLO 28.3

Explain the vanishing gradient problem and how LSTM gates address it

→ CLO3 · C2 Understand

  • Course Learning Outcome(s) served
  • CLO3: Apply various techniques and algorithms of AI used in general problem solving, optimization problems, constraint satisfaction problems, and game programming.

Slide 5

MODULE 1

Sequences and the Recurrent Neuron

When order matters

Data That Comes in Order

MODULE 1 · SEQUENCES

Electricity

Language

Speech

Units day after day (P06b).

Words in a sentence; translation.

Sound samples -> Urdu text.

REAL OUTPUT: L28/SENTENCE.PY

  • mehnat ki pass final h = [0.397 0.731]
  • pass ki mehnat final h = [0.659 0.342]
  • mehnat nahi ki final h = [0.099 0.606]
  • same W and U for every word: 8 weights

Same words, different order: different final memory - the RNN 'notices' order.

The Recurrent Neuron and Unrolling

MODULE 1 · STRUCTURE

SIMPLE RNN (ELMAN, 1990)

  • h_t = tanh( W·x_t + U·h_(t-1) + b )
  • y_t = V·h_t + c

Unrolled: one copy per time step - but the SAME W, U, V at every step. Any length, fixed weights.

Picture: reading a novel - a running summary updated page by page.

Slide 8

MODULE 2

Memory, Training and LSTM

How long can an RNN remember?

Watching the Memory Fade

MODULE 2 · ONE NEURON BY HAND

REAL OUTPUT: L28/MEMORY.PY (W = 0.8, INPUT 1, 0, 0, 0, 0)

  • u = 0.5: h = 0.664, 0.320, 0.159, 0.079, 0.040
  • u = 0.9: h = 0.664, 0.535, 0.448, 0.382, 0.331
  • by hand (u = 0.5): h1 = tanh(0.8*1 + 0.5*0) = 0.664; h2 = tanh(0.8*0 + 0.5*0.664) = 0.320

BY HAND (U = 0.5)

  • h1 = tanh(0.8·1 + 0.5·0) = tanh(0.8) = 0.664
  • h2 = tanh(0.8·0 + 0.5·0.664) = tanh(0.332) = 0.320
  • h3 = tanh(0.8·0 + 0.5·0.320) = tanh(0.160) = 0.159

Key idea: the recurrent weight u controls how long the network remembers an event.

Training Through Time and the Vanishing Gradient

MODULE 2 · TRAINING

REAL OUTPUT: L28/VANISH.PY (ERROR SIGNAL AFTER K STEPS BACK = U^K)

  • steps: 1 5 10 20
  • u = 0.5: 0.5 0.0312 0.000977 9.54e-07
  • u = 0.9: 0.9 0.59 0.349 0.122
  • u = 1.5: 1.5 7.59 57.7 3.33e+03

Vanishing

u < 1: signal shrinks to almost 0 - long links are not learned.

BACK-PROPAGATION THROUGH TIME

Exploding

  • BPTT: 1 unroll the network over the sequence
  • 2 back-propagate through every copy
  • 3 add up the updates for the shared W, U, V

u > 1: signal blows up - fix with gradient clipping.

Picture: a whisper passed along 20 students - it fades away (vanishing) or becomes a shout (exploding).

Described by Hochreiter (1991) and Bengio, Simard and Frasconi (1994).

LSTM: Gates That Decide What to Remember

MODULE 2 · LSTM AND GRU

Forget gate

Input gate

Output gate

What old memory to drop. 0 = forget, 1 = keep.

What new information to write into memory.

What part of memory to use for the output now.

Cell state: a conveyor belt that carries memory forward by ADDING, not multiplying - so the error signal survives many steps.

Slide 12

MODULE 3

An RNN on Real Data

Forecasting tomorrow's electricity

An RNN Forecasts Electricity Use

MODULE 3 · REAL DATA

REAL OUTPUT: L28/FORECAST.PY

  • days: 730, test: last 120 days
  • hidden neurons: 50
  • MAE naive (tomorrow = today): 1.76 units
  • MAE linear on 7 lags : 1.56 units
  • MAE RNN (trained output) : 1.64 units

THE RECURRENT LOOP

Honest result: RNN 1.64 beats naive 1.76, but 7 lags (1.56) is as good on this simple series. Always compare with a baseline.

  • for xt in x: # one day at a time
  • h = np.tanh(W * xt + U @ h) # memory update
  • # then: units tomorrow = V · [h, today, 1] (V by least squares)

Only V is trained here (an echo state network); full RNNs train W and U with BPTT in Keras or PyTorch.

Where Recurrent Networks Are Used

MODULE 3 · APPLICATIONS

Since 2017: Transformers (attention over all words at once) replaced RNNs in most language tasks - ChatGPT, Gemini and Claude are Transformers.

RNNs and LSTMs are still used on small devices and for streaming sensor data. Lecture 30 continues with NLP.

Activity: RNN Steps by Hand

CLASS ACTIVITY · PAIRS · 8 MIN

Pairs, 6 min + 2 min check. One-neuron RNN h_t = tanh(w·x_t + u·h_(t-1)), h_0 = 0. tanh(0.8) = 0.664, tanh(1.132) = 0.812.

w = 0.8, u = 0.5, inputs 1, 1. Compute h1 and h2.

Why can an RNN read a sentence of any length with a fixed number of weights?

In the formula, which weight carries the memory: W, U or V?

RNN with 3 inputs, 4 hidden neurons, 1 output: how many weights and biases?

Error signal factor u = 0.5 after 10 steps back? u = 1.5?

Which LSTM gate decides to drop old memory when a new sentence starts?

Common Mistakes

WATCH OUT

WRONG IDEA

CORRECT IDEA

✗ Each time step has its own weights.

✓ The SAME W, U, V are shared by all steps.

✗ An RNN remembers everything forever.

✓ Simple RNN memory fades; LSTM keeps it longer.

✗ Vanishing gradient = the output becomes 0.

✓ The ERROR SIGNAL becomes tiny, so early steps do not learn.

✗ A fancier model always beats a simple one.

✓ RNN 1.64 vs 7-lag linear 1.56: compare with baselines.

✗ ChatGPT is an RNN.

✓ It is a Transformer (attention, 2017).

Recap Quiz

CHECK YOUR UNDERSTANDING · 3 MIN

  • Q1. The recurrent weights U connect:
  • (a) Input to hidden (b) Hidden to output (c) Hidden at t-1 to hidden at t (d) Output to input
  • Q2. w = 0.8, u = 0.5, input 1 then 0, h0 = 0. h2 is:
  • (a) 0.664 (b) 0.320 (c) 0.800 (d) 0
  • Q3. Training an RNN by unrolling it and back-propagating is called:
  • (a) BPTT (b) ID3 (c) Q-learning (d) Resolution
  • Q4. LSTM was proposed to reduce the:
  • (a) Learning rate (b) Number of inputs (c) Overfitting of trees (d) Vanishing gradient problem

Homework and Next Lecture

BEFORE THE NEXT CLASS

  • Task 1: One-neuron RNN with w = 0.6, u = 0.7, h0 = 0 reads the inputs 1, 0, 1. Compute h1, h2, h3 (use a calculator for tanh).
  • Task 2: Error signal factor u = 0.8: compute 0.8^5, 0.8^10 and 0.8^30. What does this say about learning links 30 steps apart?
  • Task 3: The forecast RNN has 1 input, 50 hidden neurons and 1 output. Count W, U, b, V and c. Why is U the biggest?

NEXT LECTURE

  • Lecture 29:
  • Genetic Algorithms

← All CS-511 material