Asymptotic Theory

Asymptotic Theory

Rigorous framework for statistical inference and efficiency in modern methodology

Use this skill when working on: asymptotic properties of estimators, influence functions, semiparametric efficiency, double robustness, variance estimation, confidence intervals, hypothesis testing, M-estimation, or deriving limiting distributions.

Efficiency Bounds

Semiparametric Efficiency Theory

Cramér-Rao Lower Bound: For any unbiased estimator, $$\text{Var}(\hat{\theta}) \geq \frac{1}{nI(\theta)}$$

where $I(\theta)$ is the Fisher information.

Semiparametric Efficiency Bound: The variance of the efficient influence function: $$V_{eff} = E[\phi^*(\theta_0)^2]$$

where $\phi^*$ is the efficient influence function (EIF).

Influence Function Notation: $IF(O; \theta, P)$ represents the influence of observation $O$ on parameter $\theta$ under distribution $P$: $$IF(O; \theta, P) = \lim_{\epsilon \to 0} \frac{T((1-\epsilon)P + \epsilon \delta_O) - T(P)}{\epsilon}$$

Semiparametric Variance: For RAL estimators, $$\sqrt{n}(\hat{\theta} - \theta_0) \xrightarrow{d} N(0, E[IF(O)^2])$$

Estimating Equations: M-estimators solve $\sum_{i=1}^n \psi(O_i; \theta) = 0$, with asymptotic variance: $$V = \left(\frac{\partial}{\partial \theta} E[\psi(O; \theta)]\right)^{-1} E[\psi(O; \theta)\psi(O; \theta)^T] \left(\frac{\partial}{\partial \theta} E[\psi(O; \theta)]\right)^{-T}$$

Efficiency for Mediation Estimands

# Compute semiparametric efficiency bound
compute_efficiency_bound <- function(data, estimand = "ATE") {
  n <- nrow(data)

  if (estimand == "ATE") {
    # Estimate nuisance functions
    ps_model <- glm(A ~ X, data = data, family = binomial)
    pi_hat <- predict(ps_model, type = "response")

    mu1_model <- lm(Y ~ X, data = subset(data, A == 1))
    mu0_model <- lm(Y ~ X, data = subset(data, A == 0))

    mu1_hat <- predict(mu1_model, newdata = data)
    mu0_hat <- predict(mu0_model, newdata = data)

    # Efficient influence function
    psi_hat <- mean(mu1_hat - mu0_hat)
    phi <- with(data, {
      A/pi_hat * (Y - mu1_hat) -
      (1-A)/(1-pi_hat) * (Y - mu0_hat) +
      mu1_hat - mu0_hat - psi_hat
    })

    # Efficiency bound = variance of EIF
    list(
      efficiency_bound = var(phi),
      standard_error = sqrt(var(phi) / n),
      eif_values = phi
    )
  }
}

Empirical Process Theory

Key Concepts

Empirical Process: $\mathbb{G}_n(f) = \sqrt{n}(\mathbb{P}n - P)f = \frac{1}{\sqrt{n}}\sum{i=1}^n (f(O_i) - Pf)$

Uniform Convergence: For function class $\mathcal{F}$, $$\sup_{f \in \mathcal{F}} |\mathbb{G}n(f)| \xrightarrow{d} \sup{f \in \mathcal{F}} |\mathbb{G}(f)|$$

where $\mathbb{G}$ is a Gaussian process.

Complexity Measures

# Estimate Rademacher complexity via Monte Carlo
estimate_rademacher <- function(f_class, data, n_reps = 1000) {
  n <- nrow(data)

  sup_values <- replicate(n_reps, {
    # Random Rademacher variables
    epsilon <- sample(c(-1, 1), n, replace = TRUE)

    # Compute supremum over function class
    sup_f <- max(sapply(f_class, function(f) {
      abs(mean(epsilon * f(data)))
    }))

    sup_f
  })

  mean(sup_values)
}

Donsker Classes

Definition and Importance

A function class $\mathcal{F}$ is Donsker if $\mathbb{G}_n \rightsquigarrow \mathbb{G}$ in $\ell^\infty(\mathcal{F})$, where $\mathbb{G}$ is a tight Gaussian process.

Key Donsker Classes

Donsker Theorem Applications

For M-estimation: If $\psi(O, \theta)$ belongs to a Donsker class, then $$\sqrt{n}(\hat{\theta} - \theta_0) \xrightarrow{d} N(0, V)$$

where $V = (\partial_\theta E[\psi])^{-1} \text{Var}(\psi) (\partial_\theta E[\psi])^{-T}$

# Verify Donsker conditions for empirical process
check_donsker_conditions <- function(psi_class, data) {
  # Estimate bracketing entropy integral
  epsilon_grid <- seq(0.01, 1, by = 0.01)
  bracket_numbers <- sapply(epsilon_grid, function(eps) {
    # Estimate N_[](eps, F, L_2)
    estimate_bracketing_number(psi_class, data, eps)
  })

  # Donsker if integral converges
  entropy_integral <- integrate(
    function(eps) sqrt(log(approxfun(epsilon_grid, bracket_numbers)(eps))),
    lower = 0, upper = 1
  )

  list(
    is_donsker = entropy_integral$value < Inf,
    entropy_integral = entropy_integral$value,
    bracket_numbers = data.frame(epsilon = epsilon_grid, N = bracket_numbers)
  )
}

Core Concepts

Why Asymptotics?

1. Exact distributions often unavailable for complex estimators
2. Large-sample approximations provide tractable inference
3. Efficiency theory guides optimal estimator construction
4. Robustness properties clarified through asymptotic analysis

Fundamental Sequence

Estimator θ̂ₙ → Consistency → Asymptotic Normality → Efficiency → Inference
                    ↓              ↓                     ↓            ↓
               θ̂ₙ →ᵖ θ₀    √n(θ̂ₙ-θ₀) →ᵈ N(0,V)    V = V_eff    CIs, tests

Modes of Convergence

Convergence in Probability ($\xrightarrow{p}$)

$X_n \xrightarrow{p} X$ if $\forall \epsilon > 0$: $P(|X_n - X| > \epsilon) \to 0$

Consistency: $\hat{\theta}_n \xrightarrow{p} \theta_0$

Convergence in Distribution ($\xrightarrow{d}$)

$X_n \xrightarrow{d} X$ if $F_{X_n}(x) \to F_X(x)$ at all continuity points

Asymptotic normality: $\sqrt{n}(\hat{\theta}_n - \theta_0) \xrightarrow{d} N(0, V)$

Almost Sure Convergence ($\xrightarrow{a.s.}$)

$X_n \xrightarrow{a.s.} X$ if $P(\lim_{n\to\infty} X_n = X) = 1$

Relationship: $\xrightarrow{a.s.} \Rightarrow \xrightarrow{p} \Rightarrow \xrightarrow{d}$

Stochastic Order Notation

Key Theorems

Laws of Large Numbers

Weak LLN: If $X_1, \ldots, X_n$ iid with $E|X| < \infty$: $$\bar{X}_n \xrightarrow{p} E[X]$$

Strong LLN: If $X_1, \ldots, X_n$ iid with $E|X| < \infty$: $$\bar{X}_n \xrightarrow{a.s.} E[X]$$

Uniform LLN: For $\sup_{\theta \in \Theta}$ convergence, need additional conditions (compactness, envelope).

Central Limit Theorem

Classical CLT: If $X_1, \ldots, X_n$ iid with $E[X] = \mu$, $Var(X) = \sigma^2 < \infty$: $$\sqrt{n}(\bar{X}_n - \mu) \xrightarrow{d} N(0, \sigma^2)$$

Lindeberg-Feller CLT: For triangular arrays with: $$\sum_{i=1}^n E[X_{ni}^2 \mathbf{1}(|X_{ni}| > \epsilon)] \to 0 \quad \forall \epsilon > 0$$

Multivariate CLT: $$\sqrt{n}(\bar{X}_n - \mu) \xrightarrow{d} N(0, \Sigma)$$

Slutsky's Theorem

If $X_n \xrightarrow{d} X$ and $Y_n \xrightarrow{p} c$ (constant):

• $X_n + Y_n \xrightarrow{d} X + c$
• $X_n Y_n \xrightarrow{d} cX$
• $X_n/Y_n \xrightarrow{d} X/c$ (if $c \neq 0$)

Continuous Mapping Theorem

If $X_n \xrightarrow{d} X$ and $g$ continuous: $$g(X_n) \xrightarrow{d} g(X)$$

Delta Method

If $\sqrt{n}(\hat{\theta}_n - \theta_0) \xrightarrow{d} N(0, V)$ and $g$ differentiable at $\theta_0$: $$\sqrt{n}(g(\hat{\theta}_n) - g(\theta_0)) \xrightarrow{d} N(0, g'(\theta_0)^\top V g'(\theta_0))$$

Multivariate: Replace $g'(\theta_0)$ with Jacobian matrix.

M-Estimation Theory

Setup

Estimator $\hat{\theta}_n$ solves: $$\hat{\theta}n = \arg\max{\theta \in \Theta} M_n(\theta)$$

where $M_n(\theta) = n^{-1} \sum_{i=1}^n m(O_i; \theta)$

Consistency Conditions

1. Uniform convergence: $\sup_\theta |M_n(\theta) - M(\theta)| \xrightarrow{p} 0$
2. Identification: $M(\theta)$ uniquely maximized at $\theta_0$
3. Compactness: $\Theta$ compact (or identification at distance from boundary)

Result: $\hat{\theta}_n \xrightarrow{p} \theta_0$

Asymptotic Normality Conditions

1. $\theta_0$ interior point of $\Theta$
2. $M(\theta)$ twice differentiable at $\theta_0$
3. $\ddot{M}(\theta_0)$ non-singular
4. $\sqrt{n} \dot{M}_n(\theta_0) \xrightarrow{d} N(0, V)$

Result: $$\sqrt{n}(\hat{\theta}_n - \theta_0) \xrightarrow{d} N(0, [-\ddot{M}(\theta_0)]^{-1} V [-\ddot{M}(\theta_0)]^{-1})$$

Standard Errors

Sandwich estimator: $$\hat{V} = \hat{A}^{-1} \hat{B} \hat{A}^{-1}$$

where:

• $\hat{A} = -n^{-1} \sum_i \ddot{m}(O_i; \hat{\theta}_n)$ (Hessian)
• $\hat{B} = n^{-1} \sum_i \dot{m}(O_i; \hat{\theta}_n) \dot{m}(O_i; \hat{\theta}_n)^\top$ (outer product)

Influence Functions

Definition

The influence function of a functional $T(P)$ at distribution $P$ is: $$\phi(o) = \lim_{\epsilon \to 0} \frac{T((1-\epsilon)P + \epsilon \delta_o) - T(P)}{\epsilon}$$

where $\delta_o$ is point mass at $o$.

Properties

1. Mean zero: $E_P[\phi(O)] = 0$
2. Variance = asymptotic variance: If $\sqrt{n}(\hat{T}_n - T) \xrightarrow{d} N(0, V)$, then $V = E[\phi(O)^2]$
3. Linearization: $\sqrt{n}(\hat{T}_n - T) = \sqrt{n} \mathbb{P}_n[\phi] + o_p(1)$

Examples

Deriving Influence Functions

Method 1: Gateaux derivative (definition)

Method 2: Estimating equation approach If $\hat{\theta}$ solves $\mathbb{P}n[\psi(O; \theta)] = 0$, then: $$\phi(O) = -E[\partial\theta \psi]^{-1} \psi(O; \theta_0)$$

Method 3: Functional delta method For $\psi = g(T_1, T_2, \ldots)$: $$\phi_\psi = \sum_j \frac{\partial g}{\partial T_j} \phi_{T_j}$$

Semiparametric Efficiency

Semiparametric Models

Model $\mathcal{P}$ contains distributions satisfying: $$\theta = \Psi(P), \quad P \in \mathcal{P}$$

The "nuisance" is infinite-dimensional (e.g., unknown baseline distribution).

Tangent Space

Parametric submodels: One-dimensional smooth paths ${P_t : t \in \mathbb{R}}$ through $P_0$.

Score: $S = \partial_t \log p_t \big|_{t=0}$

Tangent space $\mathcal{T}$: Closed linear span of all such scores.

Efficiency Bound

The efficient influence function (EIF) is the projection of any influence function onto the tangent space.

Semiparametric efficiency bound: $$V_{eff} = E[\phi_{eff}(O)^2]$$

No regular estimator can have asymptotic variance smaller than $V_{eff}$.

Achieving Efficiency

An estimator is semiparametrically efficient if its influence function equals the EIF: $$\phi_{\hat{\theta}} = \phi_{eff}$$

Strategies:

1. Solve efficient score equation
2. Targeted learning (TMLE)
3. One-step estimator with EIF-based correction

Double Robustness

Concept

An estimator is doubly robust if it is consistent when either:

• Outcome model correctly specified, OR
• Treatment model (propensity score) correctly specified

AIPW Estimator

For ATE $\psi = E[Y(1) - Y(0)]$:

$$\hat{\psi}_{DR} = \mathbb{P}_n\left[\frac{A(Y - \hat{\mu}_1(X))}{\hat{\pi}(X)} + \hat{\mu}_1(X)\right] - \mathbb{P}_n\left[\frac{(1-A)(Y - \hat{\mu}_0(X))}{1-\hat{\pi}(X)} + \hat{\mu}_0(X)\right]$$

where:

• $\hat{\mu}_a(X) = \hat{E}[Y|A=a,X]$ (outcome model)
• $\hat{\pi}(X) = \hat{P}(A=1|X)$ (propensity score)

Why It Works

Bias decomposition: $$\hat{\psi}_{DR} - \psi = \text{(outcome error)} \times \text{(propensity error)} + o_p(n^{-1/2})$$

If either error is zero, bias is zero.

Efficiency Under Double Robustness

When both models correct:

• Achieves semiparametric efficiency bound
• Asymptotic variance = $E[\phi_{eff}^2]$

When one model wrong:

• Still consistent
• But less efficient than when both correct

Variance Estimation

Analytic (Sandwich)

$$\hat{V} = \frac{1}{n} \sum_{i=1}^n \hat{\phi}(O_i)^2$$

where $\hat{\phi}$ is estimated influence function.

Bootstrap

Nonparametric bootstrap:

1. Resample $n$ observations with replacement
2. Compute $\hat{\theta}^*_b$ for $b = 1, \ldots, B$
3. $\hat{V} = \text{Var}(\hat{\theta}^_1, \ldots, \hat{\theta}^_B)$

Bootstrap validity: Requires $\sqrt{n}$-consistent, regular estimators.

Influence Function-Based Bootstrap

More stable than full recomputation: $$\hat{\theta}^b = \hat{\theta} + n^{-1} \sum{i=1}^n (W_i^ - 1) \hat{\phi}(O_i)$$

where $W_i^*$ are bootstrap weights.

Inference

Confidence Intervals

Wald interval: $$\hat{\theta} \pm z_{1-\alpha/2} \cdot \hat{SE}$$

Percentile bootstrap: $$[\hat{\theta}^_{(\alpha/2)}, \hat{\theta}^_{(1-\alpha/2)}]$$

BCa bootstrap (bias-corrected accelerated): Corrects for bias and skewness.

Hypothesis Testing

Wald test: $W = (\hat{\theta} - \theta_0)^2 / \hat{V} \sim \chi^2_1$

Score test: Based on score at null.

Likelihood ratio test: $2(\ell(\hat{\theta}) - \ell(\theta_0)) \sim \chi^2_k$

Product of Coefficients (Mediation)

Setup

Mediation effect = $\alpha \beta$ (or $\alpha_1 \beta_1 \gamma_2$ for sequential)

Distribution of Products

Not normal: Product of normals is NOT normal.

Exact distribution: Complex (involves Bessel functions for two normals).

Approximations:

1. Sobel test: Normal approximation via delta method
2. PRODCLIN: Distribution of product method (RMediation)
3. Monte Carlo: Simulate from joint distribution

Delta Method Variance

For $\psi = \alpha\beta$: $$Var(\hat{\alpha}\hat{\beta}) \approx \beta^2 Var(\hat{\alpha}) + \alpha^2 Var(\hat{\beta}) + Var(\hat{\alpha})Var(\hat{\beta})$$

The last term often omitted (Sobel) but matters when effects are small.

Product of Three

For sequential mediation $\psi = \alpha_1 \beta_1 \gamma_2$:

• Distribution more complex
• Monte Carlo or specialized methods needed
• Your "product of three" manuscript addresses this

Regularity Conditions Checklist

For Consistency

• Parameter space compact (or bounded away from boundary)
• Objective function continuous in $\theta$
• Uniform convergence of criterion
• Unique maximizer at $\theta_0$

For Asymptotic Normality

• $\theta_0$ interior point
• Twice differentiable criterion
• Non-singular Hessian
• CLT applies to score
• Lindeberg/Lyapunov conditions if non-iid

For Efficiency

• Model correctly specified
• Nuisance parameters consistently estimated
• Sufficient smoothness for influence function calculation
• Rate conditions on nuisance estimation (for doubly robust)

Common Pitfalls

1. Ignoring Estimation of Nuisance Parameters

Wrong: Treat $\hat{\eta}$ as known when computing variance. Right: Account for $\hat{\eta}$ uncertainty or use cross-fitting.

2. Slow Nuisance Estimation

For doubly robust estimators, need: $$|\hat{\mu} - \mu_0| \cdot |\hat{\pi} - \pi_0| = o_p(n^{-1/2})$$

If both converge at $n^{-1/4}$, product is $n^{-1/2}$.

3. Bootstrap Failure

Bootstrap can fail for:

• Non-differentiable functionals
• Super-efficient estimators
• Boundary parameters

4. Underestimating Variance

Sandwich estimator assumes correct influence function. Model misspecification → wrong variance.

Template: Asymptotic Result

\begin{theorem}[Asymptotic Distribution]
Under Assumptions \ref{A1}--\ref{An}:
\begin{enumerate}
\item (Consistency) $\hat{\theta}_n \xrightarrow{p} \theta_0$
\item (Asymptotic normality) $\sqrt{n}(\hat{\theta}_n - \theta_0) \xrightarrow{d} N(0, V)$
\item (Variance) $V = E[\phi(O)^2]$ where $\phi$ is the influence function
\item (Variance estimation) $\hat{V} \xrightarrow{p} V$
\end{enumerate}
\end{theorem}

\begin{proof}
\textbf{Step 1 (Consistency):}
[Apply M-estimation or direct argument]

\textbf{Step 2 (Expansion):}
Taylor expand around $\theta_0$:
\[
0 = \mathbb{P}_n[\psi(O; \hat{\theta})] = \mathbb{P}_n[\psi(O; \theta_0)]
    + \mathbb{P}_n[\dot{\psi}(\tilde{\theta})](\hat{\theta} - \theta_0)
\]

\textbf{Step 3 (Rearrangement):}
\[
\sqrt{n}(\hat{\theta} - \theta_0) = -[\mathbb{P}_n[\dot{\psi}]]^{-1} \sqrt{n}\mathbb{P}_n[\psi(O; \theta_0)]
\]

\textbf{Step 4 (CLT):}
$\sqrt{n}\mathbb{P}_n[\psi(O; \theta_0)] \xrightarrow{d} N(0, E[\psi\psi^\top])$ by CLT.

\textbf{Step 5 (Slutsky):}
$\mathbb{P}_n[\dot{\psi}] \xrightarrow{p} E[\dot{\psi}]$ by WLLN. Apply Slutsky.

\textbf{Step 6 (Identify $V$):}
$V = E[\dot{\psi}]^{-1} E[\psi\psi^\top] E[\dot{\psi}]^{-\top}$.
\end{proof}

Integration with Other Skills

This skill works with:

• proof-architect - For structuring asymptotic proofs
• identification-theory - Identification precedes estimation/inference
• simulation-architect - Validate asymptotic approximations
• methods-paper-writer - Present results in manuscripts

Key References

• Bickel

• Newey

• Robins

• van der Vaart, A.W. (1998). Asymptotic Statistics

• Tsiatis, A.A. (2006). Semiparametric Theory and Missing Data

• Kennedy, E.H. (2016). Semiparametric Theory and Empirical Processes

• van der Laan, M.J. & Rose, S. (2011). Targeted Learning

Version: 1.0 Created: 2025-12-08 Domain: Asymptotic Statistics, Semiparametric Inference