In the multiple linear regression model we learned that:
What we will learn in this chapter is how to test if a subset of X_1, X_2, \dots, X_k are jointly useful. This test is called a Partial F-test.
In terms of definitions, we call the complete model the model with k independent variables: \mathbb{E}\left[Y_i|x_{i1},\dots,x_{ik}\right]= \beta_0+\beta_1 x_{i1} +\beta_2 x_{i2}+\dots +\beta_g x_{ig} + \beta_{g+1}x_{i,g+1} + \dots +\beta_k x_{ik} The reduced model is the model with g<k independent variables: \mathbb{E}\left[Y_i|x_{i1},\dots,x_{ig}\right]=\beta_0+\beta_1 x_{i1} +\beta_2 x_{i2}+\dots +\beta_g x_{ig}
What the partial F-test does is test if the k-g variables X_{g+1}, X_{g+2}, , X_{k} are jointly useful in the model.
The null and alternative hypotheses are: \begin{split} H_0 &: \beta_{g+1}=\beta_{g+2}=\dots=\beta_k=0\\ H_1 &: \text{ at least one of } \beta_{j}\neq 0 \text{ for } j=g+1,\dots,k \end{split}
This is quite general notation so to help fix ideas suppose the complete model has 5 variables and the reduced model has 3 variables. The complete model is then: \mathbb{E}\left[Y_i|x_{i1},\dots,x_{ik}\right]= \beta_0+\beta_1 x_{i1} +\beta_2 x_{i2}+\beta_3 x_{i3}+\beta_4 x_{i4}+\beta_5 x_{i5} The reduced model is then: \mathbb{E}\left[Y_i|x_{i1},\dots,x_{ig}\right]=\beta_0+\beta_1 x_{i1} +\beta_2 x_{i2}+\beta_3 x_{i3}
The null and alternative hypotheses of the partial F-test are then: \begin{split} H_0 &: \beta_{4}=\beta_{5}=0\\ H_1 &: \text{ at least one of } \beta_j\neq 0 \text{ for } j=4,5 \end{split}
We first discuss the intuition for the partial F-test test statistic.
The total variation in the y-data is measured by the total sum of squared SST=\sum_{i=1}^n (y_i - \bar{y})^2. This is like the sum of squared errors without any model: where we just use the mean \bar{y} to predict y_i. With the reduced model, the sum of squared errors is SSE_r and with the complete model the sum of squared errors is SSE_c.
By adding more variables to our model, we always reduce the sum of squared errors, so it holds that SSE_r\geq SSE_c always. If the complete model reduces the sum of squared errors “a lot” (i.e. SSE_r - SSE_c is large), then the new k-g variables are useful additions to the model. If the complete model’s sum of square errors is “very similar” to the reduced model (i.e. SSE_r - SSE_c is small), then the new k-g variables are not very useful additions to the model.
The partial F test statistic captures the size of this difference. Under H_0: F=\frac{\left(SSE_r-SSE_c\right)/\left( k-g \right)}{SSE_c/\left(n-k-1 \right)} \sim F_{k-g,n-k-1} That is, this test statistic follows an F distribution with k-g numerator and n-k-1 denominator degrees of freedom.
To show how to carry out the rest of the test we will use an example. Because a partial F-test only makes sense in models with at least 3 variables, we will return to the clothing expenditure model we saw in Chapter 15.
Why do we need at least 3 variables? With only 2 variables, we can use a regular t-test to test the usefulness of one variable, and we can use a regular F-test to test the usefulness of both variables. To have a subset with at least 2 variables we need a complete model with at least 3 variables.
Using the clothing expenditure model, we ask the following question: Does a household’s clothing expenditure depend on the household composition (number of people and number of children) after we control for household income (using \alpha=0.05)?
The question is asking if the number of children and the household size are useful additions to the model.
The complete model is: \mathbb{E}\left[ Y_i|x_{i1},x_{i2},x_{i3}\right]=\beta_0 + \beta_1 x_{i1} + \beta_2 x_{i2} + \beta_3 x_{i3} where x_{i1} is household income, x_{i2} is the number of children and x_{i3} is the household size.
The reduced model does not include the household composition variables: \mathbb{E}\left[ Y_i|x_{i1}\right]=\beta_0 + \beta_1 x_{i1}
The null & alternative hypotheses are:
We form the test statistic: Under H_0:
F=\frac{\left(SSE_r-SSE_c\right)/\left( k-g \right)}{SSE_c/\left(n-k-1 \right)}\sim F_{k-g,n-k-1}
We now need to calculate the realized value of the test statistic in our sample. To do this we follow a very similar approach to our we calculate the SSE, SSR and SST. We estimate both the reduced model and the complete model and call then m1 and m2, respectively, and then use the anova() function.
df <- read.csv("clothing-exp.csv")
m1 <- lm(clothing_exp ~ hh_inc, data = df)
m2 <- lm(clothing_exp ~ hh_inc + num_kids + hh_size, data = df)
anova(m1, m2)Analysis of Variance Table
Model 1: clothing_exp ~ hh_inc
Model 2: clothing_exp ~ hh_inc + num_kids + hh_size
Res.Df RSS Df Sum of Sq F Pr(>F)
1 298 3.3809
2 296 3.1442 2 0.23671 11.142 0.00002161 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1The SSE_r is 3.3809 under RSS for model 1. The SSE_c is 3.1442 under RSS for model 2. Notice that the SSE_c is smaller than SSE_r. This will always be the case because additional variables will always reduce the sum of squared errors in the model. What the partial F-test is testing is whether this reduction is relatively large. The 0.23671 under Sum of Sq is the difference SSE_r-SSE_c in the numerator of the partial F-test statistic formula.
The resulting partial F-test test statistic is under F and is 11.142. The associated p-value is next to it and equals 0.00002161. The table also shows the numerator and denominator degrees of freedom for the partial F-test: The 2 under Df is k-g and the 296 under Res.Df for model 2 is n-k-1. Putting everything together from the formula \frac{\left(SSE_r-SSE_c\right)/\left( k-g \right)}{SSE_c/\left(n-k-1 \right)} we can confirm that these give the same F as in the 5th column:
(0.23671 / 2) / (3.1442 / 296)[1] 11.14213If we want to extract the value of the test statistic we can do:
anova(m1, m2)$F[2][1] 11.14205If we are following the critical value approach we compare this test statistic to the critical value. Similar to the regular F test we get the critical value using the qf() function but instead of k-g numerator degrees of freedom and n-k-1 denominator degrees of freedom:
qf(0.95, 2, 296)[1] 3.026257The test statistic is greater than the critical value so we reject the null hypothesis. We also get the same conclusion looking at the critical value, which is less than 0.05. Thus the household composition variables are useful additions to the model after controlling for household income.
More generally, if we are carrying out a partial F test where Model 1 is our reduced model and Model 2 is our complete model, the contents of the anova() output is as follows:
\;Res.Df |
RSS |
\;Df |
\;\;\;Sum of Sq |
F |
Pr(>F) |
|
|---|---|---|---|---|---|---|
| 1 | n-g-1 | SSE_r | ||||
| 2 | n-k-1 | SSE_c | k-g | SSE_r-SSE_c | value of test statistic | p-value |
The partial F-test is actually a generalization of the regular F test we learned about before. If the reduced model is simply: \mathbb{E}\left[Y_i\right]=\beta_0 then the partial F-test turns into a regular F-test. This is the case with g=0.
To see this, consider the test statistic F=\frac{\left(SSE_r-SSE_c\right)/\left( k-g \right)}{SSE_c/\left(n-k-1 \right)}\sim F_{k-g,n-k-1} With a reduced model of just a constant, the SSE_r is the same as the SST. This is exactly how we learned how to obtain the SSE, SSR and SST in Chapter 17. We had to estimate a reduced model with just a constant. With g=0 and calling SSE_c simply SSE, the test statistic becomes: F=\frac{\left(SST-SSE\right)/k}{SSE/\left(n-k-1 \right)}\sim F_{k,n-k-1} Finally, using the identity SST=SSE+SSR we can write: F=\frac{SSR/k}{SSE/\left(n-k-1 \right)}\sim F_{k,n-k-1} The is exactly the same as the standard F-test!
qf(1-alpha, k-g, n-k-1).1-pf(f, k-g, n-k-1).