1.1 Principle of parsimony (Occam’s razor)

• when two models explain the data equally well, the simpler model is preferred
• the simpler model is the one with fewer parameters

2.1 AIC corrected for small sample sizes

\[ AICc = AIC + \frac{2 \cdot k \cdot (k + 1)}{n - k - 1} \] with \(n\) being the sample size

• use it when \(\frac{n}{k} < 40\)
• for larger sample sizes the \(AICc\) converges to the \(AIC\)

2.2 How to interpret the AIC(c)

• rank the models by their AIC(c) values, start with the lowest
• compare the the relative differences of the information criteria of the models
• models with \(\Delta \leq 2\) are equally good

3.1 F-Test

4.1 Ridge regression

• the RSS is penalized by the sum of the squared coefficients

\[ \begin{align} RSS_{ridge} &= \sum_{i=1}^{n} (y_i - \hat{y}_i)^2 + \lambda \sum_{j=1}^{p} \beta_j^2 \\ \hat{\beta}_{ridge} &= \underset{\beta}{\operatorname{argmin}} \left\{ RSS_{ridge} \right\} \end{align} \]

4.2 Lasso regression

• the RSS is penalized by the sum of the absolute coefficients

\[ \begin{align} RSS_{lasso} &= \sum_{i=1}^{n} (y_i - \hat{y}_i)^2 + \lambda \sum_{j=1}^{p} |\beta_j| \\ \hat{\beta}_{lasso} &= \underset{\beta}{\operatorname{argmin}} \left\{ RSS_{lasso} \right\} \end{align} \]

4.3 Comparison of ridge and lasso regression

• ridge regression shrinks the coefficients towards zero
• lasso regression shrinks the coefficients towards zero and sets some coefficients to zero

s <- 2
df <- tibble(
  x = seq(-10, 10, length.out = 400),
  Laplace = (dexp(x, 1/s) / 2)[c(400:201, 201:400)],
  Normal = dnorm(x, mean = 0, sd = s)) %>% 
  pivot_longer(cols = c(Laplace, Normal), 
               names_to = "Distribution", values_to = "y")

ggplot(df, aes(x, y, color = Distribution)) +
  geom_line(linewidth = 1)  

4.4 Comparison of ridge and lasso regression

df <- tibble(
  x1 = seq(2, 10, length.out = 50),
  x2 = rnorm(50, mean = 0, sd = 1),
  y = rnorm(length(x1), mean = 2 * x2))
  
fit_lasso <- cv.glmnet(y ~ x1 + x2, data = df, alpha = 1)  
fit_ridge <- cv.glmnet(y ~ x1 + x2, data = df, alpha = 0)
fit_lm <- lm(y ~ x1 + x2, data = df)

df %>%
  mutate(
    Lasso = predict(fit_lasso, newdata = df, s = "lambda.1se")[, 1],
    Ridge = predict(fit_ridge, newdata = df, s = "lambda.1se")[, 1],
    LM = predict(fit_lm, newdata = df)) %>%
  select(-y) %>% 
  pivot_longer(c(Lasso, Ridge, LM), names_to = "method") %>%
  mutate(method = factor(method, levels = c("LM", "Ridge", "Lasso"))) -> df_pred

ggplot(df, aes(x2, y)) +
  geom_point(size = 5, alpha = 0.7) +
  geom_line(aes(x2, value, color = method), data = df_pred, 
            linewidth = 1)

tibble(
  LM = coef(fit_lm)[-1],
  Ridge = coef(fit_ridge, s = "lambda.1se")[-1],
  Lasso = coef(fit_lasso, s = "lambda.1se")[-1])

# A tibble: 2 × 3
       LM   Ridge Lasso
    <dbl>   <dbl> <dbl>
1 -0.0308 -0.0238  0   
2  1.95    1.42    1.42

4.5 Elastic net

• the RSS is penalized by a combination of the sum of the squared coefficients and the sum of the absolute coefficients

\[ \begin{align} RSS_{elastic} &= \sum_{i=1}^{n} (y_i - \hat{y}_i)^2 + \lambda \left( \alpha \sum_{j=1}^{p} |\beta_j| + (1 - \alpha) \sum_{j=1}^{p} \beta_j^2 \right) \\ \hat{\beta}_{elastic} &= \underset{\beta}{\operatorname{argmin}} \left\{ RSS_{elastic} \right\} \end{align} \]

• if \(\alpha = 0\) the elastic net is equivalent to ridge regression
• if \(\alpha = 1\) the elastic net is equivalent to lasso regression

4.6 How to choose \(\lambda\) and \(\alpha\)

5.1 Should I split the data into training and test data sets?

• if you have enough data and want to make predictions: yes!
• if you have little data or you want to do exploration or inference: no.