Skip to main content

The central limit theorem

Sample chapter: shows what the notes can do

This chapter is a template. It shows prose, maths, numbered theorems, sidenotes, callouts and executed code in one place. Delete it once you have real notes.

1 Setup

Let X_1, X_2, \dots be independent and identically distributed with mean \mu and finite variance \sigma^2. Write the sample mean as

\bar{X}_n = \frac{1}{n} \sum_{i=1}^{n} X_i . \tag{1}

The law of large numbers says \bar{X}_n \to \mu. The central limit theorem describes the fluctuations around that limit.1

Definition 1 (Convergence in distribution) A sequence Y_n converges in distribution to Y, written Y_n \xrightarrow{d} Y, if F_{Y_n}(y) \to F_Y(y) at every point y where F_Y is continuous.

Theorem 1 (Central limit theorem) With \bar{X}_n as in Equation 1,

\sqrt{n}\,\frac{\bar{X}_n - \mu}{\sigma} \xrightarrow{d} \mathcal{N}(0, 1).

Proof. Sketch. Standardise so that \mu = 0 and \sigma = 1, and let \varphi be the characteristic function of X_1. Then \varphi(t) = 1 - t^2/2 + o(t^2), so

\varphi_{\sqrt{n}\bar{X}_n}(t) = \varphi\!\left(\tfrac{t}{\sqrt{n}}\right)^{n} = \left(1 - \frac{t^2}{2n} + o\!\left(\tfrac{1}{n}\right)\right)^{n} \to e^{-t^2/2},

which is the characteristic function of \mathcal{N}(0,1). Lévy’s continuity theorem finishes the argument.2

NoteWhat the theorem does not say

Theorem 1 needs finite variance. For heavy-tailed distributions such as the Cauchy, the sample mean does not settle down at all.

2 Seeing it numerically

The cell below draws sample means from an exponential distribution, which is strongly skewed, and standardises them as in Theorem 1.

import numpy as np

rng = np.random.default_rng(0)

def standardised_means(n, reps=20_000):
    """Standardised sample means of n Exponential(1) draws, for which mu = sigma = 1."""
    x = rng.exponential(scale=1.0, size=(reps, n))
    return np.sqrt(n) * (x.mean(axis=1) - 1.0)

for n in (1, 5, 30, 200):
    z = standardised_means(n)
    skew = ((z - z.mean()) ** 3).mean() / z.std() ** 3
    print(f"n = {n:>3}  mean = {z.mean():+.3f}  sd = {z.std():.3f}  skew = {skew:+.3f}")
n =   1  mean = -0.007  sd = 0.994  skew = +2.007
n =   5  mean = -0.006  sd = 1.005  skew = +0.906
n =  30  mean = +0.002  sd = 1.005  skew = +0.374
n = 200  mean = -0.010  sd = 1.006  skew = +0.139

The skewness shrinks towards zero as n grows, at the rate in Table 1. Figure 1 shows the same thing.

Quantity Exact value Rate
Mean of Z_n 0 none
Variance of Z_n 1 none
Skewness of Z_n 2/\sqrt{n} n^{-1/2}
Table 1: Moments of the standardised mean Z_n of Exponential(1) draws.
Plot code
import matplotlib.pyplot as plt

grid = np.linspace(-4, 4, 400)
normal_pdf = np.exp(-grid**2 / 2) / np.sqrt(2 * np.pi)

fig, axes = plt.subplots(1, 3, figsize=(7.4, 2.3), sharey=True)
for ax, n in zip(axes, (1, 5, 200)):
    ax.hist(standardised_means(n), bins=60, range=(-4, 4), density=True,
            color="#cfcfcf", edgecolor="none")
    ax.plot(grid, normal_pdf, color="#d0384e", linewidth=1.2)
    ax.set_title(f"n = {n}")
    ax.set_xlabel("standardised mean")
axes[0].set_ylabel("density")
fig.tight_layout()
plt.show()
Figure 1: Standardised sample means of Exponential(1) draws against the standard normal density.

3 Exercises

  1. Repeat the experiment with a Bernoulli(0.01) distribution. How large must n be before the histogram looks normal?
  2. Replace the exponential with a standard Cauchy. What happens, and which hypothesis of Theorem 1 fails?

Footnotes

  1. Footnotes are printed in the margin on a wide screen, level with where they are referenced. On a narrow screen, hover or tap the number to read the note in a card.↩︎

  2. A long note is cut off after nine lines with a fade, and shown in full while you hover it or its number. This one is padded to show that: the characteristic function always exists, unlike the moment generating function, which is why the proof is usually written with it. The price is a detour through complex numbers. A proof by moment generating functions is shorter to state but needs every moment to be finite, which rules out many distributions the theorem actually covers. Lindeberg’s replacement argument avoids transforms altogether.↩︎