Statistics - Descriptive Statistics, Distributions, and Inference

Table of Contents

1. Descriptive Statistics
2. Probability Distributions
3. Sampling Distributions
4. Estimation
5. Hypothesis Testing
6. Regression
7. Analysis of Variance
8. Nonparametric Methods

---

Descriptive Statistics

Measures of Center

Mean (average): $\bar{x} = \frac{1}{n}\sum_{i=1}^n x_i$

Median: Middle value (50th percentile) If $n$ odd: middle value If $n$ even: average of two middle values

Mode: Most frequent value

Trimmed mean: Mean with outliers removed

Measures of Spread

Variance: $s^2 = \frac{1}{n-1}\sum_{i=1}^n (x_i - \bar{x})^2$

Standard deviation: $s = \sqrt{s^2}$

Range: $\max - \min$

Interquartile Range (IQR): $Q_3 - Q_1$ ($Q_1 = 25th percentile, Q_3 = 75th percentile$)

Outlier rule: Values outside $[Q_1 - 1.5 \cdot \text{IQR}, Q_3 + 1.5 \cdot \text{IQR}]$

Box Plot

• Whiskers extend to min/max within 1.5IQR
• Box from $Q_1$ to $Q_3$
• Median line inside box

Standardized Score (z-score)

$z = \frac{x - \bar{x}}{s}$

Chebyshev’s Inequality: At least $1 - 1/k^2$ of data within $k$ standard deviations of mean.

Empirical Rule (Normal): 68-95-99.7 within 1, 2, 3 standard deviations.

---

Probability Distributions

Discrete Distributions

Uniform: $P(X = k) = 1/n$ for $k = 1, \dots, n$

Bernoulli: $P(X = 1) = p, P(X = 0) = 1-p$ $E[X] = p, \text{Var}(X) = p(1-p)$

Binomial: $X \sim \text{Bin}(n, p)$ $P(X = k) = \binom{n}{k}p^k(1-p)^{n-k}$ $E[X] = np, \text{Var}(X) = np(1-p)$

Geometric: Number of trials until first success $P(X = k) = (1-p)^{k-1}p$ $E[X] = 1/p, \text{Var}(X) = (1-p)/p^2$

Negative Binomial: Number of trials until $r$ successes $P(X = k) = \binom{k-1}{r-1}p^r(1-p)^{k-r}$

Hypergeometric: Sampling without replacement $N$ total objects, $M$ successes, $n$ draws: $P(X = k) = \frac{\binom{M}{k}\binom{N-M}{n-k}}{\binom{N}{n}}$

Poisson: $\lambda$ parameter $P(X = k) = \frac{\lambda^k e^{-\lambda}}{k!}$ $E[X] = \text{Var}(X) = \lambda$

Continuous Distributions

Uniform: $f(x) = 1/(b-a)$ for $a \leq x \leq b$ $E[X] = (a+b)/2, \text{Var}(X) = (b-a)^2/12$

Exponential: $f(x) = \lambda e^{-\lambda x}$ for $x \geq 0$ $E[X] = 1/\lambda, \text{Var}(X) = 1/\lambda^2$ Memoryless property: $P(X > s+t | X > s) = P(X > t)$

Normal: $X \sim N(\mu, \sigma^2)$ $f(x) = \frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{(x-\mu)^2}{2\sigma^2}}$ $E[X] = \mu, \text{Var}(X) = \sigma^2$ Standard normal: $Z \sim N(0, 1)$

Standardizing: $Z = (X - \mu)/\sigma$

Gamma: $f(x) = \frac{\lambda^\alpha}{\Gamma(\alpha)}x^{\alpha-1}e^{-\lambda x}$ $E[X] = \alpha/\lambda, \text{Var}(X) = \alpha/\lambda^2$

Chi-square ((\chi^2_\nu)): Gamma with $\alpha = \nu/2, \lambda = 1/2$ Degrees of freedom parameter $\nu$

t-distribution: Heavy-tailed, parameter $\nu$ (degrees of freedom) Approaches normal as $\nu \to \infty$

F-distribution: $F_{\nu_1,\nu_2}$ (two degrees of freedom)

Central Limit Theorem

If $X_1, \dots, X_n$ i.i.d. with $E[X_i] = \mu, \text{Var}(X_i) = \sigma^2$:

$\frac{\bar{X} - \mu}{\sigma/\sqrt{n}} \to N(0,1)$

In distribution as $n \to \infty$

Law of Large Numbers

Sample mean converges to population mean

$\bar{X}_n = \frac{1}{n}\sum_{i=1}^n X_i \to \mu$

Almost surely (strong law).

---

Sampling Distributions

Sample Mean

If $X_1, \dots, X_n \sim N(\mu, \sigma^2)$ i.i.d.:

$\bar{X} \sim N\left(\mu, \frac{\sigma^2}{n}\right)$

Standardized:

$\frac{\bar{X} - \mu}{\sigma/\sqrt{n}} \sim N(0,1)$

Sample Variance

$S^2 = \frac{1}{n-1}\sum_{i=1}^n (X_i - \bar{X})^2$

Chi-square Distribution

If $X_1, \dots, X_n \sim N(\mu, \sigma^2)$ i.i.d.:

$\frac{(n-1)S^2}{\sigma^2} \sim \chi^2_{n-1}$

t-distribution

$T = \frac{\bar{X} - \mu}{S/\sqrt{n}} \sim t_{n-1}$

Comparison: Student’s t versus normal (use t when $\sigma$ unknown).

Two Samples

Difference of means:

$\bar{X}_1 - \bar{X}_2 \sim N\left(\mu_1 - \mu_2, \frac{\sigma_1^2}{n_1} + \frac{\sigma_2^2}{n_2}\right)$

Pooled variance: If $\sigma_1 = \sigma_2$:

$t = \frac{\bar{X}_1 - \bar{X}_2}{s_p\sqrt{1/n_1 + 1/n_2}} \sim t_{n_1+n_2-2}$

where $s_p^2 = \frac{(n_1-1)s_1^2 + (n_2-1)s_2^2}{n_1+n_2-2}$

---

Estimation

Point Estimation

Estimator: Function of sample (random variable)

Estimate: Realization for specific sample

Unbiased: $E[\hat{\theta}] = \theta$

Consistent: $\hat{\theta}_n \to \theta$ (in probability)

Efficient: Minimum variance among unbiased estimators

Maximum Likelihood Estimation

Likelihood:

$L(\theta) = \prod_{i=1}^n f(x_i; \theta)$

MLE: $\hat{\theta}$ maximizes $L(\theta)$ or equivalently $\log L(\theta)$

Invariance: If $\hat{\theta}$ is MLE for $\theta$, then $g(\hat{\theta})$ is MLE for $g(\theta)$

Method of Moments

Moment estimator: Equate sample moments to population moments

$E[X^k] = \sum x_i^k/n$

Confidence Intervals

Interpretation: C% of intervals contain true parameter

Normal, known variance: $\left[\bar{x} - z_{\alpha/2}\frac{\sigma}{\sqrt{n}}, \bar{x} + z_{\alpha/2}\frac{\sigma}{\sqrt{n}}\right]$

Normal, unknown variance: $\left[\bar{x} - t_{n-1,\alpha/2}\frac{s}{\sqrt{n}}, \bar{x} + t_{n-1,\alpha/2}\frac{s}{\sqrt{n}}\right]$

Proportion: $\left[\hat{p} - z_{\alpha/2}\sqrt{\frac{\hat{p}(1-\hat{p})}{n}}, \hat{p} + z_{\alpha/2}\sqrt{\frac{\hat{p}(1-\hat{p})}{n}}\right]$

(For large $n$)

Sample size needed: $n = \left(\frac{z_{\alpha/2}\sigma}{E}\right)^2$

where $E$ is desired margin of error.

Bootstrapping

Resampling method: Sample with replacement from data

Use empirical distribution as approximate population distribution.

---

Hypothesis Testing

Hypotheses

$H_0$: Null hypothesis (status quo) $H_1$: Alternative hypothesis

Type I error: Reject $H_0$ when $H_0$ true (significance level $\alpha$)

Type II error: Fail to reject $H_0$ when $H_1$ true Power: $1 - P(\text{Type II error}) = P(\text{reject } H_0 | H_1 \text{ true})$

Test Statistic

One-sample z-test (normal, $\sigma$ known): $z = \frac{\bar{x} - \mu_0}{\sigma/\sqrt{n}}$

One-sample t-test (normal, $\sigma$ unknown): $t = \frac{\bar{x} - \mu_0}{s/\sqrt{n}} \sim t_{n-1}$

Two-sample t-test: $t = \frac{\bar{x}_1 - \bar{x}_2}{s_p\sqrt{1/n_1 + 1/n_2}} \sim t_{n_1+n_2-2}$

p-value

Probability of observing statistic at least as extreme assuming $H_0$ true

Decision: Reject $H_0$ if $p < \alpha$

Interpretation: Smaller p-value is stronger evidence against $H_0$

Rejection Regions

Two-tailed: Reject if |test statistic| > critical value

One-tailed: Reject if test statistic > critical value (upper) or < -critical value (lower)

t-tests

One-sample, two-sided: $H_0: \mu = \mu_0$ vs $H_1: \mu \neq \mu_0$

Reject if $|t| > t_{n-1,\alpha/2}$

One-sample, upper: $H_0: \mu \leq \mu_0$ vs $H_1: \mu > \mu_0$

Reject if $t > t_{n-1,\alpha}$

Tests for Proportions

One-sample: $z = \frac{\hat{p} - p_0}{\sqrt{p_0(1-p_0)/n}} \sim N(0,1)$

Two-sample: $z = \frac{\hat{p}_1 - \hat{p}_2}{\sqrt{\hat{p}(1-\hat{p})(1/n_1 + 1/n_2)}}$

where $\hat{p} = (x_1 + x_2)/(n_1 + n_2)$

Chi-square Tests

Goodness of fit:

$χ^2 = \sum\frac{(O_i - E_i)^2}{E_i} \sim χ^2_{k-1-m}$

where $O_i$ observed, $E_i$ expected, $m$ parameters estimated.

Test of independence (contingency table):

$χ^2 = \sum\sum\frac{(O_{ij} - E_{ij})^2}{E_{ij}} \sim χ^2_{(r-1)(c-1)}$

where $E_{ij} = (\text{row } i \text{ total})(\text{column } j \text{ total})/n$.

---

Regression

Simple Linear Regression

Model: $y = \beta_0 + \beta_1x + \epsilon, \epsilon \sim N(0, \sigma^2)$

Least squares estimates:

$b_1 = \frac{\sum(x_i - \bar{x})(y_i - \bar{y})}{\sum(x_i - \bar{x})^2} = \frac{S_{xy}}{S_{xx}}$

$b_0 = \bar{y} - b_1\bar{x}$

Inference on Slope

t-test: $H_0: \beta_1 = 0$ vs $H_1: \beta_1 \neq 0$

$t = \frac{b_1}{s/\sqrt{S_{xx}}} \sim t_{n-2}$

where $s = \sqrt{\frac{SSE}{n-2}}$ (residual standard error)

Confidence interval:

$b_1 \pm t_{n-2,\alpha/2}\frac{s}{\sqrt{S_{xx}}}$

Confidence and Prediction Intervals

For mean: $\mu_{y|x_0}$

$\hat{y}_0 \pm t_{n-2,\alpha/2}s\sqrt{\frac{1}{n} + \frac{(x_0-\bar{x})^2}{S_{xx}}}$

For new observation: $y_0$

$\hat{y}_0 \pm t_{n-2,\alpha/2}s\sqrt{1 + \frac{1}{n} + \frac{(x_0-\bar{x})^2}{S_{xx}}}$

Multiple Regression

y = $\beta_0 + \beta_1x_1 + \dots + \beta_k x_k + \epsilon$

Matrix form: $y = X\beta + \epsilon$

Least squares:

$\hat{\beta} = (X^T X)^{-1} X^T y$

Inference: F-test for model, t-tests for individual coefficients

R²: Proportion of variance explained $R^2 = SSR/SST = 1 - SSE/SST$

---

Analysis of Variance

One-Way ANOVA

Model: $y_{ij} = \mu_i + \epsilon_{ij}, \epsilon_{ij} \sim N(0, \sigma^2)$

Hypothesis: $H_0: \mu_1 = \mu_2 = \dots = \mu_k$

F-test:

$F = \frac{MSTr}{MSE} \sim F_{k-1, n-k}$

where:

• $MSTr = SSTr/(k-1)$
• $MSE = SSE/(n-k)$

Test statistic:

$F = \frac{\sum n_i(\bar{y}_i - \bar{y})^2/(k-1)}{\sum\sum(y_{ij} - \bar{y}_i)^2/(n-k)}$

ANOVA Table:

Source	df	SS	MS	F
Treatments	k-1	SSTr	MSTr	MSTr/MSE
Error	n-k	SSE	MSE
Total	n-1	SST

Two-Way ANOVA

Model: $y_{ijk} = \mu + \alpha_i + \beta_j + (\alpha\beta)_{ij} + \epsilon_{ijk}$

Tests for main effects and interaction.

---

Nonparametric Methods

Sign Test

For median $\tilde{\mu}_0$:

Test $H_0: \tilde{\mu} = \tilde{\mu}_0$ using signs of $(x_i - \tilde{\mu}_0)$

Wilcoxon Signed-Rank Test

For paired differences (nonparametric alternative to paired t-test)

Use ranks of |differences|, account for signs.

Wilcoxon Rank-Sum Test

For two independent samples (nonparametric alternative to two-sample t-test)

Mann-Whitney: Sum ranks from one sample

Kruskal-Wallis Test

Multi-sample nonparametric test

Uses ranks, alternative to one-way ANOVA

$H = \frac{12}{n(n+1)}\sum\frac{R_i^2}{n_i} - 3(n+1) \sim χ^2_{k-1}$

---

Supplementary Statistics & Probability Reference (from mathematics_GPT)

Unique Example Summaries

1. Chebyshev’s Inequality: If a data set has a mean of 100 and a standard deviation of 20, at least 75% of the data falls within 60 to 140.

2. Empirical Rule (Normal): For a normal distribution, 68% of the data falls within one standard deviation of the mean, 95% within two, and 99.7% within three.

3. Law of Large Numbers: If you flip a fair coin repeatedly, the proportion of heads will converge to 0.5 as the number of flips increases.

4. Central Limit Theorem: If you repeatedly roll a fair six-sided die, the average of the results will be approximately normally distributed, regardless of the number of rolls.

5. Chi-square Distribution: If you have a large number of independent, identically distributed random variables, their sum will be approximately chi-square distributed.

6. t-distribution: If you take a sample from a normal distribution and calculate the sample mean and sample standard deviation, the ratio of the sample mean to the sample standard deviation divided by the square root of the sample size will be approximately t-distributed.

Mnemonic Tables

1. Probability Distributions:

   • Uniform: All outcomes equally likely.
   • Bernoulli: Binary outcome (success/failure).
   • Binomial: Multiple independent Bernoulli trials.
   • Geometric: Number of trials until first success.
   • Negative Binomial: Number of trials until r successes.
   • Hypergeometric: Sampling without replacement.
   • Poisson: Counts of rare events.

2. Continuous Distributions:

   • Uniform: Flat line.
   • Exponential: Decaying curve.
   • Normal: Bell curve.
   • Gamma: Right-skewed.
   • Chi-square: Right-skewed.
   • t-distribution: Heavy-tailed.
   • F-distribution: Fatter than normal.

3. Hypothesis Testing:

   • One-sample z-test: Normal, σ known.
   • One-sample t-test: Normal, σ unknown.
   • Two-sample t-test: Two samples, pooled variance.
   • Chi-square tests: Goodness of fit, independence.

4. Confidence Intervals:

   • Normal, known variance: z-score.
   • Normal, unknown variance: t-distribution.
   • Proportion: Normal approximation.

5. Regression:

   • Simple linear: Least squares, t-test.
   • Multiple: Matrix form, F-test.

Practical Rules

1. Sampling:

   • For large populations, sampling with replacement is often sufficient.
   • For small populations, sampling without replacement is necessary.

2. Hypothesis Testing:

   • If the sample size is large (n > 30), z-tests can be used.
   • If the sample size is small (n < 30), t-tests are preferred.
   • If σ is unknown, use t-tests.

3. Confidence Intervals:

   • For large n, z-intervals are accurate.
   • For small n, t-intervals are more robust.
   • For proportions, use normal approximation for large n.

4. Regression:

   • R² measures the proportion of variance explained.
   • A high R² does not imply causality.
   • Always check residuals for normality and independence.

---

Next: Probability Theory

---

Last updated: Comprehensive statistics reference covering descriptive, inferential, and regression methods.