Learning Outcomes
SLO | After this lecture you can… | CLO | Bloom |
|---|---|---|---|
SLO 5.1 | Classify benchmarks (real programs, kernels, toy, synthetic, suites) and explain why suites of real programs are preferred. | CLO2 | C2 Understand |
SLO 5.2 | Calculate MIPS and show with an example why MIPS, MFLOPS and peak performance can mislead. | CLO2 | C3 Apply |
SLO 5.3 | Summarize results with SPECratios and the geometric mean, and choose the right mean for a situation. | CLO2 | C3 Apply |
§1 What Is a Benchmark?
§1.1 Why we need benchmarks
Slide L05-S06
A benchmark (a standard test program used to compare computers) is run on every machine so that results can be compared. The best test is your own workload (the programs you really run every day), but a buyer cannot run all of their programs on every computer in the market. So the industry agrees on standard programs.
Real-world examples: phone review websites show AnTuTu or Geekbench scores to compare phones. A car company prints a fuel average of 20 km per litre measured on a test track, but in Lahore traffic you get about 13. A test must look like real use, or its number is misleading.
§1.2 Kinds of benchmarks
Slide L05-S07
Kind | What it is | Example | Trust |
|---|---|---|---|
Real applications | Programs people actually run | gcc compiler, xz compressor, a web browser | Highest |
Kernels | Small key loops taken from real programs | Linpack, Livermore Loops | Medium |
Toy benchmarks | Tiny 10-100 line programs | Quicksort, Sieve of Eratosthenes, Puzzle | Low |
Synthetic benchmarks | Invented code that tries to copy an 'average' instruction mix | Dhrystone, Whetstone | Lowest |
Synthetic benchmarks failed because compiler writers learned to optimize the benchmark itself: some compilers removed most of the Dhrystone code because its results were never used. The score went up; real programs did not get faster. Kernels are useful to find one weakness of a machine, but they miss the rest of a real program.
§1.3 Benchmark suites
Slide L05-S08
A suite (collection of benchmarks) reduces the effect of any one program: a machine may be lucky on one program, but rarely on twenty.
Suite | Measures | Notes |
|---|---|---|
SPEC CPU | Processor, memory and compiler on desktop and server machines | SPEC89, 92, 95, 2000, 2006, 2017 and SPEC CPU 2026 (released May 2026, 52 benchmarks; free for accredited academic institutions). SPEC CPU 2017 (43 benchmarks) is retired in November 2026. The textbook uses SPEC2006. |
TPC | Database and transaction servers | TPC-C: online order processing (like Daraz); TPC-H: decision support (business reports) |
EEMBC | Embedded processors | CoreMark: processors in cars, routers, washing machines |
MLPerf | AI systems | Time to train a model, or speed of running (inference) a trained model |
SPEC suites come in two forms: SPECspeed (time to finish one copy of each program - response time) and SPECrate (many copies at once - throughput). This matches the two meanings of performance from Lecture 4.
§2 Misleading Metrics
§2.1 MIPS
Slide L05-S10
MIPS = Instruction count / (Execution time x 10^6) = Clock rate / (CPI x 10^6) |
MIPS (million instructions per second) looks attractive because bigger is better and it is easy to understand. It has three problems:
- Ignores the ISA: it ignores what an instruction does, so you cannot compare computers with different instruction sets (one CISC instruction can do the work of several RISC instructions);
- Changes per program: the same computer gives a different MIPS for every program, so a computer has no single MIPS value;
- Can move the wrong way: MIPS can rise while the program becomes slower (§2.2).
§2.2 Worked example: higher MIPS, slower program
Slide L05-S11
A 4 GHz computer has instruction classes A, B and C with CPI 1, 2 and 3. Two compilers produce different code for the same program.
A (CPI 1) | B (CPI 2) | C (CPI 3) | Cycles | Time | MIPS | |
|---|---|---|---|---|---|---|
Compiler 1 | 5 billion | 1 billion | 1 billion | 10 billion | 2.5 s | 2800 |
Compiler 2 | 10 billion | 1 billion | 1 billion | 15 billion | 3.75 s | 3200 |
Compiler 1: cycles = 5x1 + 1x2 + 1x3 = 10 billion time = 10e9 / 4e9 = 2.5 s |
Compiler 2 has the higher MIPS but takes 3.75 / 2.5 = 1.5 times longer. MIPS rewarded it for adding cheap instructions. Always compare time.
§2.3 MFLOPS and peak performance
Slide L05-S12
MFLOPS (millions of floating-point operations per second) only counts floating-point work, so it says nothing about a compiler or a database, and it counts a slow divide or square root the same as a fast add.
Peak performance is the highest rate the hardware could ever reach. A 3 GHz processor that can start 4 instructions per cycle has a peak of 3000 × 4 = 12,000 MIPS. A real program with CPI 1.2 gets 3000 / 1.2 = 2,500 MIPS, only about 21% of the peak. Peak is a 'guaranteed not to exceed' number - like a speedometer that shows 220 km/h when your real average speed from Faisalabad to Lahore is about 90 km/h.
§3 Summarizing Results
§3.1 Arithmetic and weighted arithmetic mean of times
Slide L05-S14
Arithmetic mean AM = (1/n) x sum(Time_i) |
If the workload really is 'run each program once', total time (or the arithmetic mean of times) is the true answer. If the programs run different amounts, use weights: a bank that spends 70% of its day on transactions and 30% on reports uses WAM = 0.7 × T1 + 0.3 × T2. The weakness: different users have different weights, and one long program dominates the mean.
§3.2 The trap: averaging normalized times
Slide L05-S15
Normalized time = time on this machine / time on a reference machine.
P1 | P2 | Arithmetic mean | Geometric mean | |
|---|---|---|---|---|
Time on A | 1 s | 1000 s | ||
Time on B | 10 s | 100 s | ||
B normalized to A | 10 | 0.1 | 5.05 ('B 5 times slower') | 1.0 |
A normalized to B | 0.1 | 10 | 5.05 ('A 5 times slower') | 1.0 |
Both conclusions cannot be true: the arithmetic mean of ratios depends on which machine is the reference. The geometric mean gives 1.0 either way. For the workload 'run each program once', total time decides: A = 1001 s, B = 110 s, so B is 9.1 times faster.
§3.3 SPECratio and the geometric mean
Slide L05-S16
SPECratio = Time on reference machine / Time on your machine (bigger is better) |
Program | Ref time | A time | A ratio | B time | B ratio |
|---|---|---|---|---|---|
Prog 1 | 1000 s | 100 s | 10 | 125 s | 8 |
Prog 2 | 1500 s | 250 s | 6 | 150 s | 10 |
Prog 3 | 800 s | 80 s | 10 | 100 s | 8 |
Geometric mean | 600^(1/3) = 8.43 | 640^(1/3) = 8.62 |
B is 8.62 / 8.43 = 1.02 times faster by SPEC's measure. (These times are made up for practice; real SPEC reference times are published with each suite.) A SPECratio of 10 means '10 times faster than the reference machine'.
§3.4 Why the geometric mean, and its limit
Slide L05-S17
Property: GM(ratios of B) / GM(ratios of A) = GM(time of A / time of B). The reference time cancels, so the comparison is the same whichever reference machine SPEC chooses. Check: GM of (100/125, 250/150, 80/100) = (0.8 × 1.667 × 0.8)^(1/3) = 1.02, the same as 8.62 / 8.43.
Limit: the geometric mean does not predict total time. Total times are A = 430 s and B = 375 s, so for 'run each program once' B is 1.15 times faster, not 1.02. (The arithmetic means of the ratios are both 8.67 - a third, different answer.)
Situation | Use |
|---|---|
You know the real workload | Total time, or the weighted arithmetic mean of times |
A fair single score for many unknown users | Geometric mean of SPECratios |
§4 Common Mistakes
Slide L05-S19
Wrong idea | Correct idea |
|---|---|
A higher MIPS rating means a faster computer. | MIPS ignores the ISA and the work per instruction; compare time. |
Peak performance tells real speed. | Peak is a never-exceed limit; real programs reach only a fraction of it. |
Synthetic benchmarks represent real work. | They are easy to cheat; real programs in a suite are trusted. |
Average normalized times with an arithmetic mean. | That answer depends on the reference machine; use the geometric mean of ratios. |
Word Bank
Word | Simple meaning | Real-world example |
|---|---|---|
Benchmark | Standard program used to compare computers | The same exam paper given to every student |
Workload | The set of programs a user really runs | Your real daily driving in city traffic |
Kernel | Small key loop taken from a real program | Testing only a car's brakes |
Synthetic benchmark | Invented program copying an average mix | A practice exam written to look like the real one |
Suite | A collection of benchmarks | A full exam with many questions, not one |
MIPS | Million instructions per second | Counting steps per minute without asking where you walked |
Peak performance | Highest rate the hardware could ever reach | Top speed on the speedometer |
Normalize | Divide by a reference value | Marks out of 100 turned into a percentage |
SPECratio | Reference time / your time | How many times faster than the reference machine |
Geometric mean | nth root of the product of n numbers | Average growth rate of savings over several years |
Self-Check Questions
- Why is your own workload the best benchmark, and why is it rarely used? (§1.1)
- Name the four kinds of benchmarks in order of trust, with one example each. (§1.2)
- Why did synthetic benchmarks such as Dhrystone lose trust? (§1.2)
- What does SPEC CPU measure, and what is the newest suite? (§1.3)
- A 2 GHz processor runs a program with CPI 1.25. Find the MIPS. (§2.1)
- Using the compiler example, explain how MIPS can rise while time also rises. (§2.2)
- A 3 GHz processor can start 4 instructions per cycle. What is its peak MIPS? What does a program with CPI 1.5 achieve? (§2.3)
- When is the weighted arithmetic mean of times the right summary? (§3.1)
- SPECratios on three programs are 4, 9 and 6. Find the geometric mean. (§3.3)
- Why does SPEC use the geometric mean, and what can it not tell you? (§3.4)
Answers: 1600 MIPS; peak 12,000 MIPS, achieved 3000 / 1.5 = 2,000 MIPS; GM = 216^(1/3) = 6.
Further Reading
- Hennessy and Patterson (2012): Chapter 1, section 1.8 (measuring, reporting and summarizing performance).
- SPEC website: www.spec.org - the SPEC CPU suites and thousands of published results.