The Law of Large Numbers Explained: Making Sense of Data Analysis and Sampling Techniques

1. Introduction

Probability theory is essential in many fields, including finance, statistics, and engineering. One of the fundamental concepts in probability theory is the law of large numbers, which states that the average of a large number of independent and identically distributed (iid) random variables tends to converge to a fixed value as the sample size increases.

The law of large numbers is vital for scientific endeavours and our understanding of natural phenomena since it allows us to accumulate data from which scientists can construct theories. The law of large numbers says that the more observations we gather, the more confident we can be about what we see.

This article will provide a technical overview of the law of large numbers, including its history, statement, proof, applications, and limitations.

2. History and Development of the Law of Large Numbers

The idea of the law of large numbers can be traced back to the 16th century when the Italian mathematician Gerolamo Cardano studied the probability of rolling a specific number with two dice. However, it was not until the 18th century that mathematicians such as Jakob Bernoulli and Pierre-Simon Laplace formally stated and proved the law of large numbers.

In the 19th century, the Russian mathematicians Pafnuty Chebyshev and Andrey Markov significantly contributed to developing the law of large numbers.

Chebyshev proved the weak law of large numbers, which states that the sample mean of a sequence of iid random variables converges in probability to the expected value. Markov later extended Chebyshev’s result to prove the strong law of large numbers, which states that the sample mean of a sequence of iid random variables converges almost surely to the expected value.

3. What Is the Law of Large Numbers?

The law of large numbers is a concept in probability theory that says that as you repeat an experiment many times, the average result will get closer and closer to the expected result.

For example, if you flip a coin, you expect it to land heads up 50% of the time and tails up 50% of the time. If you flip the coin just a few times, you might get all heads or all tails, but if you keep flipping it many times, the number of heads and tails will get closer and closer to 50%.

The idea behind the law of large numbers is that chance events tend to balance out over time. So, if you keep repeating an experiment, the results will eventually settle down to the expected value. This is why the law of large numbers is important in fields like statistics and finance, where we want to make predictions based on random events.

By understanding how the law of large numbers works, we can make more accurate predictions and reduce the risk of large losses.

4. Statement and Proof of the Law of Large Numbers

4.1 Weak Law of Large Numbers

Let X₁, X₂, …, X_n be a sequence of iid random variables with expected value μ and variance σ². The weak law of large numbers states that the sample mean $\bar{X_n} = \frac{1}{n}\sum_{i=1}^{n} X_i$ converges in probability to the expected value μ as the sample size n approaches infinity, i.e.,

$\displaystyle \lim_{n \to \infty} P(|\bar{X_n} - \mu| \ge \epsilon) = 0, \quad \forall \epsilon > 0$

Intuitively, the larger the sample size, the closer the sample mean is to the true mean. The proof of the weak law of large numbers is based on Chebyshev’s inequality and the central limit theorem.

4.2 Proof of the Weak Law of Large Numbers

Suppose we have a sequence of independent and identically distributed (i.i.d.) random variables X1, X2, …, Xn with a finite expected value μ and finite variance σ². Let S_n = X₁ + X₂ + … + X_n be the sum of the first n terms in the sequence, and let μ_n = E[S_n]/n be the sample mean of the first n terms.

The weak law of large numbers states that for any ε > 0, the probability that the absolute difference between the sample mean and the true mean exceeds ε goes to zero as n goes to infinity. In other words,

$\displaystyle \lim_{n \to \infty} P(|\mu_{n} - \mu| > \epsilon) = 0, \quad \forall \epsilon > 0$

To prove this, we can use Chebyshev’s inequality, which states that for any random variable Y with finite mean μ_Y and finite variance σ², and any k > 0, the probability of observing a deviation from the mean of the order of k x σ is:

$\displaystyle P(|Y - \mu_Y| \ge k \times \sigma ) \le \dfrac{1}{k^2}$

Using this inequality, we can bound the probability of the event {|μ_n – μ| > ε} as follows:

P(|μ_n – μ| > ε) = P(|S_n/n – μ| > ε) = P(|S_n – nμ| > εn) ≤ Var(S_n)/ε²n² (by Chebyshev’s inequality) = σ²/nε².

Taking the limit as n goes to infinity, we have

$\displaystyle \lim_{n \to \infty} P(|\mu_n - \mu| \ge \epsilon) \le \dfrac{\sigma^2}{n\epsilon^2} \to 0$

Therefore, we have shown that the probability that the sample mean deviates from the true mean by more than ε goes to zero as n goes to infinity, which proves the weak law of large numbers.

Note that this proof assumes that the random variables are i.i.d. with finite variance, a common assumption in probability theory. If the random variables have different distributions or properties, the proof may need to be adapted accordingly.

4.3 The Central Limit Theorem

The law of large numbers describes the value of the sample mean as the sample size approaches infinity, but it does not say much about the sample mean’s convergence rate or distribution. To understand these properties, we turn to the Central Limit Theorem.

The central limit theorem is one of the most important theorems in statistics and probability theory. It states that the sum or average of many independent and identically distributed random variables will approach a normal distribution N(0, 1), regardless of the underlying distribution of the individual random variables.

In other words, as the sample size increases, the distribution of the sample mean tends to approach a normal distribution with a mean equal to the true mean and variance equal to the true variance divided by the sample size.

To illustrate the central limit theorem, consider the following examples:

Rolling dice: Suppose we repeatedly roll a fair six-sided die and sum the numbers obtained on each roll. As the number of rolls increases, the sum will approach a normal distribution with a mean equal to the expected sum of a single roll (3.5) multiplied by the number of rolls and variance equal to the expected variance of a single roll (35/12) multiplied by the number of rolls.
Measuring heights: Suppose we measure the heights of many people and compute the average height. As the sample size increases, the sample mean will approach a normal distribution with a mean equal to the true mean height of the population and a variance equal to the true variance of the population divided by the sample size.

4.2 Strong Law of Large Numbers

Let X₁, X₂, …, X_n be a sequence of iid random variables with expected value $\mu$ and variance $\sigma^2$ . The strong law of large numbers states that the sample mean $\bar{X}_n = \frac{1}{n}\sum_{i=1}^{n} X_i$ converges almost surely to the expected value $\mu$ as the sample size n approaches infinity, i.e.,

$P(\displaystyle \lim_{n \to\infty} \bar{X_n} = \mu) = 1$

Intuitively, this means that with probability one, the sample mean approaches the true mean as the sample size increases. The proof of the strong law of large numbers is more involved and usually requires advanced probability theory techniques, such as martingales.

5. Interpreting the Law of Large Numbers

5.1 Frequency Interpretation

The law of large numbers can be interpreted as saying that the relative frequency of an event approaches its probability as the number of trials increases.

In other words, if you perform an experiment many times and count the number of times a certain outcome occurs, the proportion of occurrences should approach the theoretical probability of that outcome.

5.2 Convergence Interpretation

Another way of interpreting the law of large numbers can be by saying that the sample mean converges to the population mean as the sample size increases.

In other words, if you take a random sample from a population and calculate the mean of the sample, that means should approach the population’s true mean as the sample size gets larger.

5.3 Information Interpretation

The information-based interpretation of the law of large numbers is as follows: the more information we have about a random variable, the more accurately we can predict its behaviour. As we collect more data about a random variable, we can make more accurate predictions about its future behaviour.

5.4 Empirical Interpretation

Finally, the law of large numbers can mean that empirical averages become more accurate as the sample size increases. In other words, as we collect more data and calculate more averages, the accuracy of those averages improves and becomes more reliable.

5.5 Which Interpretation Is Correct?

These interpretations are not mutually exclusive and may be more relevant in different contexts. However, they all reflect the basic idea that as the number of observations increases, the random fluctuations become less important, and the true underlying pattern emerges.

6. Applications of the Law of Large Numbers

The law of large numbers has many practical applications in various fields, including:

6.1 Gambling

In gambling, the law of large numbers can be used to predict the long-term expected value of a game. For example, if a gambler plays a fair game of roulette with a single-number bet, the expected value of each bet is negative since the odds of winning are 1/38 and the payout is only 35 to 1. However, if the gambler makes many bets, the law of large numbers implies that the average payout will approach the expected value, which is negative in this case. The gambler is expected to lose money in the long run.

6.2 Finance

In finance, the law of large numbers is used to model the behaviour of financial markets and estimate the risk of investments. For example, the law of large numbers implies that the average return of a large portfolio of stocks or bonds tends to converge to the expected return as the number of assets increases. This allows investors to diversify their portfolios and reduce the risk of large losses.

6.3 Statistics

In statistics, the law of large numbers is used to estimate the parameters of a population based on a sample. For example, suppose we want to estimate the mean and variance of the height of all people in a city. In that case, we can take a sample of individuals and use the sample mean and sample variance as estimates of the population parameters. The law of large numbers ensures that as the sample size increases, the sample mean and variance become more accurate estimates of the population parameters.

7. Limitations of the Law of Large Numbers

While the law of large numbers is a powerful tool in probability theory, several limitations and assumptions must be considered. Some of the limitations include the following:

7.1 Dependent variables

The law of large numbers assumes that the random variables are independent and identically distributed. If the variables are dependent or non-identically distributed, the law of large numbers may not hold.

7.2 Finite variance

The law of large numbers requires that the random variables have a finite variance. The law of large numbers may not hold if the variance is infinite or undefined.

7.3 Convergence rate

The law of large numbers only guarantees the convergence of the sample mean to the expected value in the limit as the sample size approaches infinity. The convergence rate may be slow or erratic, making it difficult to estimate the expected value accurately for small sample sizes.

8. Conclusion

The law of large numbers is a fundamental concept in probability theory with numerous applications in finance, statistics, and engineering. The law states that the average of many independent and identically distributed random variables tends to converge to a fixed value as the sample size increases.

The law of large numbers has been extensively studied and proved in various forms, including the weak and strong laws of large numbers. However, the law has several limitations and assumptions that must be considered in practical applications. By understanding the law of large numbers, we can better estimate probabilities, model financial markets, and make informed decisions based on statistical data.