## Linear Mixed-Effects Models

Linear mixed-effects models are extensions of linear regression models for data that are collected and summarized in groups. These models describe the relationship between a response variable and independent variables, with coefficients that can vary with respect to one or more grouping variables. A mixed-effects model consists of two parts, fixed effects and random effects. Fixed-effects terms are usually the conventional linear regression part, and the random effects are associated with individual experimental units drawn at random from a population. The random effects have prior distributions whereas fixed effects do not. Mixed-effects models can represent the covariance structure related to the grouping of data by associating the common random effects to observations that have the same level of a grouping variable. The standard form of a linear mixed-effects model is

$$y=\underset{fixed}{\underbrace{X\beta}}+\underset{random}{\underbrace{Zb}}+\underset{error}{\underbrace{\epsilon}},$$

where

*y*is the*n*-by-1 response vector, and*n*is the number of observations.*X*is an*n*-by-*p*fixed-effects design matrix.*β*is a*p*-by-1 fixed-effects vector.*Z*is an*n*-by-*q*random-effects design matrix.*b*is a*q*-by-1 random-effects vector.*ε*is the*n*-by-1 observation error vector.

The assumptions for the linear mixed-effects model are:

Random-effects vector,

*b*, and the error vector,*ε*, have the following prior distributions:$$\begin{array}{l}b~N\left(0,{\sigma}^{2}D\left(\theta \right)\right),\\ \epsilon ~N\left(0,\sigma {}^{2}I\right),\end{array}$$

where

*D*is a symmetric and positive semidefinite matrix, parameterized by a variance component vector*θ*,*I*is an*n*-by-*n*identity matrix, and*σ*^{2}is the error variance.Random-effects vector,

*b*, and the error vector,*ε*, are independent from each other.

Mixed-effects models are also called *multilevel models* or *hierarchical models* depending on the context. Mixed-effects models is a more general term than the latter two. Mixed-effects models might include factors that are not necessarily multilevel or hierarchical, for example crossed factors. That is why mixed-effects is the terminology preferred here. Sometimes mixed-effects models are expressed as multilevel regression models (first level and grouping level models) that are fit simultaneously. For example, a varying or random intercept model, with one continuous predictor variable *x* and one grouping variable with *M* levels, can be expressed as

$$\begin{array}{l}{y}_{im}={\beta}_{0m}+{\beta}_{1}{x}_{im}+{\epsilon}_{im},\text{\hspace{1em}}i=1,2,\mathrm{..},n,\text{\hspace{1em}}m=1,2,\mathrm{...},M,\text{\hspace{1em}}{\epsilon}_{im}~N\left(0,{\sigma}^{2}\right),\\ {\beta}_{0m}={\beta}_{00}+{b}_{0m},\text{\hspace{1em}}{b}_{0m}~N\left(0,{\sigma}_{0}^{2}\right),\end{array}$$

where *y*_{im} corresponds to data for observation *i* and group *m*, *n* is the total number of observations, and b_{0m} and ε_{im} are independent of each other. After substituting the group-level parameters in the first-level model, the model for the response vector becomes

$${y}_{im}=\underset{fixed\text{\hspace{0.17em}}effects}{\underbrace{{\beta}_{00}+{\beta}_{1}{x}_{im}}}+\underset{random\text{\hspace{0.17em}}effects}{\underbrace{{b}_{0m}}}+{\epsilon}_{im}.$$

A random intercept and slope model with one continuous predictor variable *x*, where both the intercept and slope vary independently by a grouping variable with *M* levels is

$$\begin{array}{l}{y}_{im}={\beta}_{0m}+{\beta}_{1m}{x}_{im}+{\epsilon}_{im},\text{\hspace{1em}}i=1,2,\mathrm{...},n,\text{\hspace{1em}}m=1,2,\mathrm{...},M,\text{\hspace{1em}}{\epsilon}_{im}~N\left(0,{\sigma}^{2}\right),\\ {\beta}_{0m}={\beta}_{00}+{b}_{0m},\text{\hspace{1em}}{b}_{0m}~N\left(0,{\sigma}_{0}^{2}\right),\text{\hspace{1em}}\\ {\beta}_{1m}={\beta}_{10}+{b}_{1m},\text{\hspace{1em}}{b}_{1m}~N\left(0,{\sigma}_{1}^{2}\right),\end{array}$$

or

$${b}_{m}=\left(\begin{array}{l}{b}_{0m}\\ {b}_{1m}\end{array}\right)~N\left(0,\left(\begin{array}{cc}{\sigma}_{0}^{2}& 0\\ 0& {\sigma}_{1}^{2}\end{array}\right)\right).$$

You might also have correlated random effects. In general, for a model with a random intercept and slope, the distribution of the random effects is

$${b}_{m}=\left(\begin{array}{l}{b}_{0m}\\ {b}_{1m}\end{array}\right)~N\left(0,\sigma {}^{2}D\left(\theta \right)\right),$$

where *D* is a 2-by-2 symmetric and positive semidefinite matrix, parameterized by a variance component vector *θ*.

After substituting the group-level parameters in the first-level model, the model for the response vector is

$${y}_{im}=\underset{fixed\text{\hspace{0.17em}}effects}{\underbrace{{\beta}_{00}+{\beta}_{10}{x}_{im}}}+\underset{random\text{\hspace{0.17em}}effects}{\underbrace{{b}_{0m}+{b}_{1m}{x}_{im}}}+{\epsilon}_{im},\text{\hspace{1em}}i=1,2,\mathrm{...},n,\text{\hspace{1em}}m=1,2,\mathrm{...},M.$$

If you express the group-level variable, *x*_{im}, in the random-effects term by *z*_{im}, this model is

$${y}_{im}=\underset{fixed\text{\hspace{0.17em}}effects}{\underbrace{{\beta}_{00}+{\beta}_{10}{x}_{im}}}+\underset{random\text{\hspace{0.17em}}effects}{\underbrace{{b}_{0m}+{b}_{1m}{z}_{im}}}+{\epsilon}_{im},\text{\hspace{1em}}i=1,2,\mathrm{...},n,\text{\hspace{1em}}m=1,2,\mathrm{...},M.$$

In this case, the same terms appear in both the fixed-effects design matrix and random-effects design matrix. Each *z _{im}* and

*x*correspond to the level

_{im}*m*of the grouping variable.

It is also possible to explain more of the group-level variations by adding more group-level predictor variables. A random-intercept and random-slope model with one continuous predictor variable *x*, where both the intercept and slope vary independently by a grouping variable with *M* levels, and one group-level predictor variable *v*_{m} is

$$\begin{array}{l}{y}_{im}={\beta}_{0im}+{\beta}_{1im}{x}_{im}+{\epsilon}_{im},\text{\hspace{1em}}i=1,2,\mathrm{...},n,\text{\hspace{1em}}m=1,2,\mathrm{...},M,\text{\hspace{1em}}{\epsilon}_{im}~N\left(0,{\sigma}^{2}\right),\\ {\beta}_{0im}={\beta}_{00}+{\beta}_{01}{v}_{im}+{b}_{0m},\text{\hspace{1em}}{b}_{0m}~N\left(0,{\sigma}_{0}^{2}\right),\text{\hspace{1em}}\\ {\beta}_{1im}={\beta}_{10}+{\beta}_{11}{v}_{im}+{b}_{1m},\text{\hspace{1em}}{b}_{1m}~N\left(0,{\sigma}_{1}^{2}\right).\end{array}$$

This model results in main effects of the group-level predictor and an interaction term between the first-level and group-level predictor variables in the model for the response variable as

$$\begin{array}{l}{y}_{im}={\beta}_{00}+{\beta}_{01}{v}_{im}+{b}_{0m}+\left({\beta}_{10}+{\beta}_{11}{v}_{im}+{b}_{1m}\right){x}_{im}+{\epsilon}_{im},\text{\hspace{1em}}i=1,2,\mathrm{...},n,\text{\hspace{1em}}m=1,2,\mathrm{...},M,\\ \text{\hspace{1em}}\text{\hspace{1em}}=\underset{fixed\text{\hspace{0.17em}}effects}{\underbrace{{\beta}_{00}+{\beta}_{10}{x}_{im}+{\beta}_{01}{v}_{im}+{\beta}_{11}{v}_{im}{x}_{im}}}+\underset{random\text{\hspace{0.17em}}effects}{\underbrace{{b}_{0m}+{b}_{1m}{x}_{im}}}+{\epsilon}_{im}.\end{array}$$

The term *β*_{11}*v*_{m}*x*_{im} is often called a cross-level interaction in many textbooks on multilevel models. The model for the response variable *y* can be expressed as

$$\begin{array}{l}{y}_{im}=\left[\begin{array}{cccc}1& {x}_{1}{}_{im}& {v}_{im}& {v}_{im}{x}_{1im}\end{array}\right]\left[\begin{array}{c}{\beta}_{00}\\ {\beta}_{10}\\ {\beta}_{01}\\ {\beta}_{11}\end{array}\right]+\left[\begin{array}{cc}1& {x}_{1im}\end{array}\right]\left[\begin{array}{c}{b}_{0m}\\ {b}_{1m}\end{array}\right]+{\epsilon}_{im},\text{\hspace{1em}}i=1,2,\mathrm{...},n,\text{\hspace{1em}}m=1,2,\mathrm{...},M,\\ \text{\hspace{1em}}\text{\hspace{1em}}\end{array}$$

which corresponds to the standard form given earlier,

$$\text{\hspace{1em}}y=X\beta +Zb+\epsilon .$$

In general, if there are *R* grouping variables, and *m*(*r*,*i*) shows the level of grouping variable *r*, for observation *i*, then the model for the response variable for observation *i* is

$${y}_{i}={x}_{i}^{T}\beta +{\displaystyle \sum _{r=1}^{R}{z}_{ir}{b}_{m(r,i)}^{(r)}}+{\epsilon}_{i},\text{\hspace{1em}}i=1,2,\mathrm{...},n,$$

where *β* is a *p*-by-1 fixed-effects vector, *b*^{(r)}_{m(r,i)} is a *q*(*r*)-by-1 random-effects vector for the* r*th grouping variable and level *m*(*r*,*i*), and *ε*_{i} is a 1-by-1 error term for observation *i*.

## References

[1] Pinherio, J. C., and D. M. Bates. *Mixed-Effects Models in S and S-PLUS*. Statistics and Computing Series, Springer, 2004.

[2] Hariharan, S. and J. H. Rogers. “Estimation Procedures for Hierarchical Linear Models.”
*Multilevel Modeling of Educational Data* (A. A. Connell and D. B. McCoach, eds.). Charlotte, NC: Information Age Publishing, Inc., 2008.

[3] Hox, J. *Multilevel Analysis, Techniques and Applications*. Lawrence Erlbaum Associates, Inc., 2002

[4] Snidjers, T. and R. Bosker. *Multilevel Analysis*. Thousand Oaks, CA: Sage Publications, 1999.

[5] Gelman, A. and J. Hill. *Data Analysis Using Regression and Multilevel/Hierarchical Models*. New York, NY: Cambridge University Press, 2007.

## See Also

`LinearMixedModel`

| `fitlme`

| `fitlmematrix`