Nexus Learn

Lecture 5: Benchmarks and Metrics

Lecture Notes CS-704 Advanced Computer Architecture 9 min read Updated 8 Nov 2026

↓ Download notes (Word)

On this page
  1. Learning Outcomes
  2. §1 What Is a Benchmark?
  3. §2 Misleading Metrics
  4. §3 Summarizing Results
  5. §4 Common Mistakes
  6. Word Bank
  7. Self-Check Questions
  8. Further Reading

Learning Outcomes

SLO

After this lecture you can…

CLO

Bloom

SLO 5.1

Classify benchmarks (real programs, kernels, toy, synthetic, suites) and explain why suites of real programs are preferred.

CLO2

C2 Understand

SLO 5.2

Calculate MIPS and show with an example why MIPS, MFLOPS and peak performance can mislead.

CLO2

C3 Apply

SLO 5.3

Summarize results with SPECratios and the geometric mean, and choose the right mean for a situation.

CLO2

C3 Apply

§1 What Is a Benchmark?

§1.1 Why we need benchmarks

Slide L05-S06

A benchmark (a standard test program used to compare computers) is run on every machine so that results can be compared. The best test is your own workload (the programs you really run every day), but a buyer cannot run all of their programs on every computer in the market. So the industry agrees on standard programs.

Real-world examples: phone review websites show AnTuTu or Geekbench scores to compare phones. A car company prints a fuel average of 20 km per litre measured on a test track, but in Lahore traffic you get about 13. A test must look like real use, or its number is misleading.

§1.2 Kinds of benchmarks

Slide L05-S07

Kind

What it is

Example

Trust

Real applications

Programs people actually run

gcc compiler, xz compressor, a web browser

Highest

Kernels

Small key loops taken from real programs

Linpack, Livermore Loops

Medium

Toy benchmarks

Tiny 10-100 line programs

Quicksort, Sieve of Eratosthenes, Puzzle

Low

Synthetic benchmarks

Invented code that tries to copy an 'average' instruction mix

Dhrystone, Whetstone

Lowest

Synthetic benchmarks failed because compiler writers learned to optimize the benchmark itself: some compilers removed most of the Dhrystone code because its results were never used. The score went up; real programs did not get faster. Kernels are useful to find one weakness of a machine, but they miss the rest of a real program.

§1.3 Benchmark suites

Slide L05-S08

A suite (collection of benchmarks) reduces the effect of any one program: a machine may be lucky on one program, but rarely on twenty.

Suite

Measures

Notes

SPEC CPU

Processor, memory and compiler on desktop and server machines

SPEC89, 92, 95, 2000, 2006, 2017 and SPEC CPU 2026 (released May 2026, 52 benchmarks; free for accredited academic institutions). SPEC CPU 2017 (43 benchmarks) is retired in November 2026. The textbook uses SPEC2006.

TPC

Database and transaction servers

TPC-C: online order processing (like Daraz); TPC-H: decision support (business reports)

EEMBC

Embedded processors

CoreMark: processors in cars, routers, washing machines

MLPerf

AI systems

Time to train a model, or speed of running (inference) a trained model

SPEC suites come in two forms: SPECspeed (time to finish one copy of each program - response time) and SPECrate (many copies at once - throughput). This matches the two meanings of performance from Lecture 4.

§2 Misleading Metrics

§2.1 MIPS

Slide L05-S10

MIPS = Instruction count / (Execution time x 10^6) = Clock rate / (CPI x 10^6)

Example: 2 GHz, CPI 1.25 -> MIPS = 2000 / 1.25 = 1600

MIPS (million instructions per second) looks attractive because bigger is better and it is easy to understand. It has three problems:

  • Ignores the ISA: it ignores what an instruction does, so you cannot compare computers with different instruction sets (one CISC instruction can do the work of several RISC instructions);
  • Changes per program: the same computer gives a different MIPS for every program, so a computer has no single MIPS value;
  • Can move the wrong way: MIPS can rise while the program becomes slower (§2.2).

§2.2 Worked example: higher MIPS, slower program

Slide L05-S11

A 4 GHz computer has instruction classes A, B and C with CPI 1, 2 and 3. Two compilers produce different code for the same program.

A (CPI 1)

B (CPI 2)

C (CPI 3)

Cycles

Time

MIPS

Compiler 1

5 billion

1 billion

1 billion

10 billion

2.5 s

2800

Compiler 2

10 billion

1 billion

1 billion

15 billion

3.75 s

3200

Compiler 1: cycles = 5x1 + 1x2 + 1x3 = 10 billion time = 10e9 / 4e9 = 2.5 s
MIPS = 7e9 / (2.5 x 10^6) = 2800
Compiler 2: cycles = 10x1 + 1x2 + 1x3 = 15 billion time = 15e9 / 4e9 = 3.75 s
MIPS = 12e9 / (3.75 x 10^6) = 3200

Compiler 2 has the higher MIPS but takes 3.75 / 2.5 = 1.5 times longer. MIPS rewarded it for adding cheap instructions. Always compare time.

§2.3 MFLOPS and peak performance

Slide L05-S12

MFLOPS (millions of floating-point operations per second) only counts floating-point work, so it says nothing about a compiler or a database, and it counts a slow divide or square root the same as a fast add.

Peak performance is the highest rate the hardware could ever reach. A 3 GHz processor that can start 4 instructions per cycle has a peak of 3000 × 4 = 12,000 MIPS. A real program with CPI 1.2 gets 3000 / 1.2 = 2,500 MIPS, only about 21% of the peak. Peak is a 'guaranteed not to exceed' number - like a speedometer that shows 220 km/h when your real average speed from Faisalabad to Lahore is about 90 km/h.

§3 Summarizing Results

§3.1 Arithmetic and weighted arithmetic mean of times

Slide L05-S14

Arithmetic mean AM = (1/n) x sum(Time_i)
Weighted mean WAM = sum(Weight_i x Time_i), weights add up to 1

If the workload really is 'run each program once', total time (or the arithmetic mean of times) is the true answer. If the programs run different amounts, use weights: a bank that spends 70% of its day on transactions and 30% on reports uses WAM = 0.7 × T1 + 0.3 × T2. The weakness: different users have different weights, and one long program dominates the mean.

§3.2 The trap: averaging normalized times

Slide L05-S15

Normalized time = time on this machine / time on a reference machine.

P1

P2

Arithmetic mean

Geometric mean

Time on A

1 s

1000 s

Time on B

10 s

100 s

B normalized to A

10

0.1

5.05 ('B 5 times slower')

1.0

A normalized to B

0.1

10

5.05 ('A 5 times slower')

1.0

Both conclusions cannot be true: the arithmetic mean of ratios depends on which machine is the reference. The geometric mean gives 1.0 either way. For the workload 'run each program once', total time decides: A = 1001 s, B = 110 s, so B is 9.1 times faster.

§3.3 SPECratio and the geometric mean

Slide L05-S16

SPECratio = Time on reference machine / Time on your machine (bigger is better)
GM = (r1 x r2 x ... x rn)^(1/n) (calculator: x^(1/n))

Program

Ref time

A time

A ratio

B time

B ratio

Prog 1

1000 s

100 s

10

125 s

8

Prog 2

1500 s

250 s

6

150 s

10

Prog 3

800 s

80 s

10

100 s

8

Geometric mean

600^(1/3) = 8.43

640^(1/3) = 8.62

B is 8.62 / 8.43 = 1.02 times faster by SPEC's measure. (These times are made up for practice; real SPEC reference times are published with each suite.) A SPECratio of 10 means '10 times faster than the reference machine'.

§3.4 Why the geometric mean, and its limit

Slide L05-S17

Property: GM(ratios of B) / GM(ratios of A) = GM(time of A / time of B). The reference time cancels, so the comparison is the same whichever reference machine SPEC chooses. Check: GM of (100/125, 250/150, 80/100) = (0.8 × 1.667 × 0.8)^(1/3) = 1.02, the same as 8.62 / 8.43.

Limit: the geometric mean does not predict total time. Total times are A = 430 s and B = 375 s, so for 'run each program once' B is 1.15 times faster, not 1.02. (The arithmetic means of the ratios are both 8.67 - a third, different answer.)

Situation

Use

You know the real workload

Total time, or the weighted arithmetic mean of times

A fair single score for many unknown users

Geometric mean of SPECratios

§4 Common Mistakes

Slide L05-S19

Wrong idea

Correct idea

A higher MIPS rating means a faster computer.

MIPS ignores the ISA and the work per instruction; compare time.

Peak performance tells real speed.

Peak is a never-exceed limit; real programs reach only a fraction of it.

Synthetic benchmarks represent real work.

They are easy to cheat; real programs in a suite are trusted.

Average normalized times with an arithmetic mean.

That answer depends on the reference machine; use the geometric mean of ratios.

Word Bank

Word

Simple meaning

Real-world example

Benchmark

Standard program used to compare computers

The same exam paper given to every student

Workload

The set of programs a user really runs

Your real daily driving in city traffic

Kernel

Small key loop taken from a real program

Testing only a car's brakes

Synthetic benchmark

Invented program copying an average mix

A practice exam written to look like the real one

Suite

A collection of benchmarks

A full exam with many questions, not one

MIPS

Million instructions per second

Counting steps per minute without asking where you walked

Peak performance

Highest rate the hardware could ever reach

Top speed on the speedometer

Normalize

Divide by a reference value

Marks out of 100 turned into a percentage

SPECratio

Reference time / your time

How many times faster than the reference machine

Geometric mean

nth root of the product of n numbers

Average growth rate of savings over several years

Self-Check Questions

  • Why is your own workload the best benchmark, and why is it rarely used? (§1.1)
  • Name the four kinds of benchmarks in order of trust, with one example each. (§1.2)
  • Why did synthetic benchmarks such as Dhrystone lose trust? (§1.2)
  • What does SPEC CPU measure, and what is the newest suite? (§1.3)
  • A 2 GHz processor runs a program with CPI 1.25. Find the MIPS. (§2.1)
  • Using the compiler example, explain how MIPS can rise while time also rises. (§2.2)
  • A 3 GHz processor can start 4 instructions per cycle. What is its peak MIPS? What does a program with CPI 1.5 achieve? (§2.3)
  • When is the weighted arithmetic mean of times the right summary? (§3.1)
  • SPECratios on three programs are 4, 9 and 6. Find the geometric mean. (§3.3)
  • Why does SPEC use the geometric mean, and what can it not tell you? (§3.4)

Answers: 1600 MIPS; peak 12,000 MIPS, achieved 3000 / 1.5 = 2,000 MIPS; GM = 216^(1/3) = 6.

Further Reading

  • Hennessy and Patterson (2012): Chapter 1, section 1.8 (measuring, reporting and summarizing performance).
  • SPEC website: www.spec.org - the SPEC CPU suites and thousands of published results.

← All CS-704 material