Probability Distributions

Uniform Distribution

The uniform distribution on the interval $[a,b]$ has density $$f(x)=\{\begin{array}{ll}\frac{1}{b-a},& \text{if}x\in [a,b]\\ 0& \text{otherwise}\end{array}.$$

Normal Distribution

A random vector $X$ of dimension $n$ is said to have a Normal distribution with mean $\mu$ and covariance matrix $\Sigma$ if $\Sigma$ is positive definite and its density function is given by $$f(x)=\frac{1}{(2\pi {)}^{n/2}{\left|\Sigma \right|}^{1/2}}{e}^{-\frac{1}{2}(x-\mu {)}^{\top }{\Sigma }^{-1}(x-\mu )}.$$ In this case, we write $$X\sim N(\mu ,\Sigma ).$$

Regression Property

Suppose that $X$ is a random vector with a multivariate Normal distribution and partition the vector as $X=(X_1,X_2)$ so that $$label\mathrm{RegressionDistribution}\left(\begin{array}{c}{X}_1\\ X_2\end{array}\right)\sim N[\left(\begin{array}{c}{\mu }_1\\ {\mu }_2\end{array}\right),\left(\begin{array}{cc}{\Sigma }_{11}& {\Sigma }_{12}\\ {\Sigma }_{21}& {\Sigma }_{22}\end{array}\right)].$$ The conditional distribution of $X_1$ given $X_2=x_2$ is $$label\mathrm{RegressionConditionalDistribution}{X}_1|X_2=x_2\sim N({\mu }_1+{\Sigma }_{12}{\Sigma }_{22}^{-1}(x_2-{\mu }_2),{\Sigma }_{11}-{\Sigma }_{12}{\Sigma }_{22}^{-1}{\Sigma }_{21}).$$

Conversely, whenever (eq:RegressionConditionalDistribution) holds and when the marginal distribution of $X_2$ satisfies $X_2\sim N({\mu }_2,{\Sigma }_{22})$, then the joint distribution of $(X_1,X_2)$ is given by (eq:RegressionDistribution).

Bivariate Normal Distribution

Suppose the random variables $X_1,X_2$ are jointly distributed according to the bivariate normal distribution. This distribution can be characterized by five parameters:

• ${\mu }_1$: the mean of $X_1$,
• ${\mu }_2$: the mean of $X_2$,
• ${\sigma }_1$: the standard deviation of $X_1$,
• ${\sigma }_2$: the standard deviation of $X_2$,
• $\rho$: the correlation of $X_1$ and $X_2$.

Their joint density is

$$f(x_1,x_2)=\frac{1}{2\pi {\sigma }_1{\sigma }_2\sqrt{1-{\rho }^2}}\exp \{-\frac{1}{2(1-{\rho }^2)}[{\left(\frac{x_1-{\mu }_1}{{\sigma }_1}\right)}^2-2\rho \frac{x_1-{\mu }_1}{{\sigma }_1}\frac{x_2-{\mu }_2}{{\sigma }_2}+{\left(\frac{x_2-{\mu }_2}{{\sigma }_2}\right)}^2]\}.$$

Sampling From the Bivariate Normal Distribution

The following algorithm can be used to sample from the bivariate normal distribution:

1. Let $z_1$ and $z_2$ be independent draws from the standard normal distribution $N(0,1)$.

2. Then, $x_1$ and $x_2$ calculated as follows will have a joint bivariate normal distribution with parameters $({\mu }_1,{\mu }_2,{\sigma }_1,{\sigma }_2,\rho )$:

$$\begin{array}{rl}{x}_1& ={\mu }_1+{\sigma }_1{z}_1\\ x_2& ={\mu }_2+{\sigma }_2[z_1\rho +z_2\sqrt{1-{\rho }^2}]\\ \end{array}$$

Truncated Normal Distribution

Let $X$ be normally distributed with mean $\mu$ and variance ${\sigma }^2$ and consider the conditional distribution of $X$ in the interval $[a,b]\subset ℝ$. The distribution of $X$ conditional on $X\in [a,b]$ is the truncated normal distribution. The conditional density is $$f(x|x\in [a,b])=\frac{\frac{1}{\sigma }\varphi \left(\frac{x-\mu }{\sigma }\right)}{\Phi \left(\frac{b-\mu }{\sigma }\right)-\Phi \left(\frac{a-\mu }{\sigma }\right)}$$ for $a\le x\le b$, where $\varphi$ and $\Phi$ denote respectively the standard normal density and CDF.

Censored Regression

In econometrics, this distribution is used in the censored regression model (also commonly called the Tobit model). In this model, we observe an outcome $y_i$ and covariates $x_i$ according to the model where ${\epsilon }_i\sim N(0,{\sigma }^2)$. $y_i^{*}$ is a latent variable which is only observed when it is above a certain threshold (normalized to zero here).

The expected value of $y_i$ and the likelihood function can be derived using the properties of the truncated normal distribution.

Derivation of the Mean

$$\begin{array}{rl}E[X|X\in [a,b]]& =\mu +{\int }_a^b(x-\mu )\frac{\frac{1}{\sigma }\varphi \left(\frac{x-\mu }{\sigma }\right)}{\Phi \left(\frac{b-\mu }{\sigma }\right)-\Phi \left(\frac{a-\mu }{\sigma }\right)}\mathrm{dx}\\ & =\mu +\frac{1}{\Phi \left(\frac{b-\mu }{\sigma }\right)-\Phi \left(\frac{a-\mu }{\sigma }\right)}{\int }_a^b\frac{x-\mu }{\sigma }\frac{1}{\sqrt{2\pi }}{e}^{-\frac{1}{2}{\left(\frac{x-\mu }{\sigma }\right)}^2}\mathrm{dx}\\ & =\mu +\frac{1}{\Phi \left(\frac{b-\mu }{\sigma }\right)-\Phi \left(\frac{a-\mu }{\sigma }\right)}{\int }_a^b\sigma \frac{\partial }{\partial x}[-\frac{1}{\sqrt{2\pi }}{e}^{-\frac{1}{2}{\left(\frac{x-\mu }{\sigma }\right)}^2}]\mathrm{dx}\\ & =\mu +\frac{1}{\Phi \left(\frac{b-\mu }{\sigma }\right)-\Phi \left(\frac{a-\mu }{\sigma }\right)}\sigma {[-\frac{1}{\sqrt{2\pi }}{e}^{-\frac{1}{2}{\left(\frac{x-\mu }{\sigma }\right)}^2}]}_a^b\\ & =\mu -\sigma \frac{\varphi \left(\frac{b-\mu }{\sigma }\right)-\varphi \left(\frac{a-\mu }{\sigma }\right)}{\Phi \left(\frac{b-\mu }{\sigma }\right)-\Phi \left(\frac{a-\mu }{\sigma }\right)}.\end{array}$$

Special Cases

• Left censoring of standard normal: $[a,b]=[a,\mathrm{\infty }]$, $\mu =0$, $\sigma =1$.

$$E[X|X>a]=\frac{\varphi (a)}{1-\Phi (a)}.$$ This expression is called the inverse Mills ratio and is denoted $$\lambda (a)\equiv \frac{\varphi (a)}{1-\Phi (a)}.$$ It is the hazard function of the Normal distribution.

Exponential Distribution

The probability density function (pdf) of an exponential distribution with rate parameter $\lambda$ is

$$f(x;\lambda )=\{\begin{array}{ll}\lambda e^{-\lambda x},& x\ge 0,\\ 0,& x<0.\end{array}$$

The cumulative distribution function (cdf) is

$$F(x;\lambda )=\{\begin{array}{ll}1-e^{-\lambda x},& x\ge 0,\\ 0,& x<0.\end{array}$$

The distribution has support $[0,\mathrm{\infty })$. If a random variable $X$ has this distribution, we write $X\sim Expo(\lambda ).$

Alternative parameterization

Sometimes the exponential distribution is parameterized by $\beta =\frac{1}{\lambda }$ where $\beta$ is the mean of the $Expo(\lambda )$ distribution. Thus, to avoid ambiguity it is important to specify whether the parameter denotes the rate or mean.

Weibull Distribution

The Weibull distribution (Weibull, 1951) has cdf $$F(x)=1-e^{-\lambda x^{\alpha }}$$ and pdf $$f(x)=\{\begin{array}{ll}\lambda x^{\alpha -1}{e}^{-\lambda x^{\alpha }},& x\ge 0,\\ 0,& x<0.\\ \end{array}$$ Here, $\lambda >0$ is a scale parameter and $\alpha >0$ is a shape parameter.

Special Cases

For certain values of $\alpha$, the Weibull distribution reduces to other common distributions: The Weibull distribution with $\alpha =1$ reduces to the exponential distribution with rate parameter $\lambda$.

• When $\alpha =1$, it is equivalent to the Exponential distribution with rate parameter $\lambda$.
• When $\alpha =2$, it is equivalent to the Rayleigh distribution with variance ${\lambda }^2/2$.
• When $\alpha =3.4$, it is approximately the univariate normal distribution.
• As $\alpha \to \mathrm{\infty }$, it converges to the Dirac delta function.

References

• Cameron, A. C. and P. K. Trivedi (2005). Microeconometrics: Methods and Applications. New York: Cambridge University Press.

• Weibull, W. (1951). A statistical distribution function of wide applicability. J. Appl. Mech.-Trans. ASME 18, 293–297.