05-regression.Rmd

(ref:moderndivepart) Statistical Modeling with `moderndive`

```{r regression-conditional-text, echo=FALSE, results="asis", purl=FALSE}
if (is_latex_output()) {
  cat("# (PART) (ref:moderndivepart) {-}")
} else {
  cat("# (PART) Statistical Modeling with moderndive {-} ")
}
```

# Simple Linear Regression {#regression}

```{r regression-setup, include=FALSE, purl=FALSE}
# Used to define Learning Check numbers:
chap <- 5
lc <- 0

# Set R code chunk defaults:
opts_chunk$set(
  echo = TRUE,
  eval = TRUE,
  warning = FALSE,
  message = TRUE,
  tidy = FALSE,
  purl = TRUE,
  out.width = "\\textwidth",
#  fig.height = 4,
  fig.align = "center"
)

# Set output digit precision
options(scipen = 99, digits = 3)

# In kable printing replace all NA's with blanks
options(knitr.kable.NA = "")

# Set random number generator see value for replicable pseudo-randomness.
set.seed(76)
```

We have introduced data visualization in Chapter \@ref(viz), data wrangling in Chapter \@ref(wrangling), and data importing and "tidy" data in Chapter \@ref(tidy). In this chapter, we work with **regression**, a method that helps us study the relationship between an *outcome variable* or *response* and one or more *explanatory variables* or *regressors*.
The method starts by proposing a *statistical model*. Data is then collected and used to estimate the coefficients or parameters for the model, and these results are typically used for two purposes: 

1. For **explanation** when we want to describe how changes in one or more of the regressors are associated with changes in the response, quantify those changes, establish which of the regressors truly have an association with the response, or determine whether the model used to describe the relationship between the response and the explanatory variables seems appropriate.
1. For **prediction** when we want to determine, based on the observed values of the regressors, what will the value of the response be? We are not concerned about how all the regressors relate and interact with one another or with the response, we simply want as good predictions as possible.

As an illustration, assume that we want to study the relationship between blood pressure and potential risk factors such as daily salt intake, age, and physical activity levels. 
The response is blood pressure, and the regressors are the risk factors. 
If we use linear regression for explanation, we may want to determine whether reducing daily salt intake has a real effect on lowering blood pressure, or by how much blood pressure decreases if an individual reduces their salt intake by half. 
This information may help target individuals of a specific age group with advice on dietary changes to manage blood pressure. 
On the other hand, if we use linear regression for prediction, we would like to determine, as accurately as possible, the blood pressure of a given individual based on the data collected about their salt intake, age, and physical activity levels. In this chapter, we will use linear regression for explanation. 

The most basic and commonly-used type of regression is *linear regression*\index{regression!linear}. 
Linear regression involves a *numerical* response and one or more regressors that can be *numerical* or *categorical*. It is called linear regression because the **statistical model** that describes the relationship between the expected response and the regressors is assumed to be linear. In particular, when the model has a single regressor, the linear regression is the equation of a line.
Linear regression is the foundation for almost any other type of regression or related method.

In Chapter \@ref(regression), we introduce linear regression with only one regressor. In Section \@ref(model1), the explanatory variable is numerical. This scenario is known as *simple linear regression*. In Section \@ref(model2), the explanatory variable is categorical.

In Chapter \@ref(multiple-regression) on multiple regression, we extend these ideas and work with models with two explanatory variables.  In Section \@ref(model4), we work with two numerical explanatory variables. In Section \@ref(model3), we work with one numerical and one categorical explanatory variable and study the model with and without interactions.

In Chapter \@ref(inference-for-regression) on inference for regression, we revisit the regression models and analyze the results using *statistical inference*, a method discussed in Chapters \@ref(sampling), \@ref(confidence-intervals), and \@ref(hypothesis-testing) on sampling, bootstrapping and confidence intervals, and hypothesis testing and $p$-values, respectively. The focus there is also be on using linear regression for prediction instead of explanation.

We begin with regression with a single explanatory variable\index{regression!basic}. We also introduce the *correlation coefficient*, discuss "correlation versus causation," and determine whether the model *fits* the data observed.



### Needed packages {-#reg-packages}

We now load all the packages needed for this chapter (this assumes you've already installed them). In this chapter, we introduce some new packages:

1. The `tidyverse` "umbrella" [@R-tidyverse] package. Recall from our discussion in Section \@ref(tidyverse-package) that loading the `tidyverse` package by running `library(tidyverse)` loads the following commonly used data science packages all at once:
    + `ggplot2` for data visualization
    + `dplyr` for data wrangling
    + `tidyr` for converting data to "tidy" format
    + `readr` for importing spreadsheet data into R
    + As well as the more advanced `purrr`, `tibble`, `stringr`, and `forcats` packages
2. The `moderndive` \index{R packages!moderndive} package of datasets and functions for tidyverse-friendly introductory linear regression as well as a data frame summary function.

If needed, read Section \@ref(packages) for information on how to install and load R packages.

```{r regression-load-packages, message=FALSE}
library(tidyverse)
library(moderndive)
```


```{r regression-load-internal, message=FALSE, echo=FALSE, purl=FALSE}
# Packages needed internally, but not in text.
library(mvtnorm)
library(broom)
library(kableExtra)
library(patchwork)
```





## One numerical explanatory variable {#model1}

Before we introduce the model needed for simple linear regression, we present an example.
Why do some countries exhibit high fertility rates while others have significantly lower ones? Are there correlations between fertility rates and life expectancy across different continents and nations? Could underlying socioeconomic factors be influencing these trends? 

These are all questions that are of interest to demographers and policy makers, as understanding fertility rates is important for planning and development. By analyzing the dataset of UN member states, which includes variables such as country codes (ISO), fertility rates, and life expectancy for 2022, researchers can uncover patterns and make predictions about fertility rates based on life expectancy.

```{r regression-dynamic-text, echo=FALSE, purl=FALSE}
# This code is used for dynamic non-static in-line text output purposes
n_countries <- un_member_states_2024 |> nrow()
```

In this section, we aim to explain differences in fertility rates as a function of one numerical variable: life expectancy. Could it be that countries with higher life expectancy also have lower fertility rates? Could it be instead that countries with higher life expectancy tend to have higher fertility rates? Or could it be that there is no relationship between life expectancy and fertility rates? We answer these questions by modeling the relationship between fertility rates and life expectancy using *simple linear regression* \index{regression!simple linear} where we have:

1. A numerical outcome variable $y$ (the country's fertility rate) and
2. A single numerical explanatory variable $x$ (the country's life expectancy).


### Exploratory data analysis {#model1EDA}

The data on the `r n_countries` current UN member states (as of 2024) can be found in the `un_member_states_2024` data frame included in the `moderndive` package. However, to keep things simple we include only those rows that don't have missing data with `na.omit()` and `select()` only the subset of the variables we'll consider in this chapter, and save this data in a new data frame called `UN_data_ch5`:

```{r regression-create-UN_data_ch5}
UN_data_ch5 <- un_member_states_2024 |>
  select(iso, 
         life_exp = life_expectancy_2022, 
         fert_rate = fertility_rate_2022, 
         obes_rate = obesity_rate_2016)|>
  na.omit()
```

```{r regression-create-n_demo_ch5, include=FALSE}
n_demo_ch5 <- nrow(UN_data_ch5)
```

A crucial step before doing any kind of analysis or modeling is performing an *exploratory data analysis*, \index{data analysis!exploratory} or EDA for short. EDA gives you a sense of the distributions of the individual variables in your data, whether any potential relationships exist between variables, whether there are outliers and/or missing values, and (most importantly) how to build your model. Here are three common steps in an EDA:

1. Most crucially, looking at the raw data values.
1. Computing summary statistics, such as means, medians, and maximums.
1. Creating data visualizations.

We perform the first common step in an exploratory data analysis: looking at the raw data values. Because this step seems so trivial, unfortunately many data analysts ignore it. However, getting an early sense of what your raw data looks like can often prevent many larger issues down the road. 

You can do this by using RStudio's spreadsheet viewer or by using the `glimpse()` function as introduced in Subsection \@ref(exploredataframes) on exploring data frames:

```{r regression-glimpse-UN_data_ch5}
glimpse(UN_data_ch5)
```

Observe that ``Rows: `r n_demo_ch5` `` indicates that there are `r n_demo_ch5` rows/observations in `UN_data_ch5` after filtering out the missing values, where each row corresponds to one observed country/member state. It is important to note that the *observational unit* \index{observational unit} is an individual country. Recall from Subsection \@ref(exploredataframes) that the observational unit is the "type of thing" that is being measured by our variables.

A full description of all the variables included in `un_member_states_2024` can be found by reading the associated help file (run [`?un_member_states_2024`](https://moderndive.github.io/moderndive/reference/un_member_states_2024.html) in the console). Let's describe only the `r UN_data_ch5 |> ncol()` variables we selected in `UN_data_ch5`:

1. `iso`: An identification variable used to distinguish between the `r n_demo_ch5` countries in the filtered dataset.
2. `fert_rate`: A numerical variable representing the country's fertility rate in 2022 corresponding to the expected number of children born per woman in child-bearing years. This is the outcome variable $y$ of interest.
3. `life_exp`: A numerical variable representing the country's average life expectancy in 2022 in years. This is the primary explanatory variable $x$ of interest.
4. `obes_rate`: A numerical variable representing the country's obesity rate in 2016. This will be another explanatory variable $x$ that we use in the *Learning check* at the end of this subsection.

```{r regression-create-sample_size, echo=FALSE}
sample_size <- 5
```


An alternative way to look at the raw data values is by choosing a random sample of the rows in `UN_data_ch5` by piping it into the `slice_sample()` \index{R packages!dplyr!slice\_sample()} function from the `dplyr` package. Here we set the `n` argument to be ``r sample_size``, indicating that we want a random sample of `r sample_size` rows. We display the results in Table \@ref(tab:random-countries). Note that due to the random nature of the sampling, you will likely end up with a different subset of `r sample_size` rows.

```{r regression-sample-rows, eval=FALSE}
UN_data_ch5 |>
  slice_sample(n = 5)
```

```{r random-countries, echo=FALSE, purl=FALSE}
UN_data_ch5 |> 
  slice_sample(n = sample_size) |> 
  kbl(
    digits = 3,
    caption = paste("A random sample of", sample_size, 
                    "out of the 193 total countries (181 without missing data)"),
    booktabs = TRUE,
    linesep = ""
  ) |>
  kable_styling(
    font_size = ifelse(is_latex_output(), 10, 16),
    latex_options = c("HOLD_position")
  )
```

We have looked at the raw values in our `UN_data_ch5` data frame and got a preliminary sense of the data. We can now compute summary statistics. We start by computing the mean and median of our numerical outcome variable `fert_rate` and our numerical explanatory variable `life_exp`. We do this by using the `summarize()` function from `dplyr` along with the `mean()` and `median()` summary functions we saw in Section \@ref(summarize).

```{r regression-compute-mean, eval=FALSE}
UN_data_ch5 |>
  summarize(mean_life_exp = mean(life_exp), 
            mean_fert_rate = mean(fert_rate),
            median_life_exp = median(life_exp), 
            median_fert_rate = median(fert_rate))
```

```{r regression-compute-mean-sized, echo=FALSE}
UN_data_ch5 |>
  summarize(mean_life_exp = mean(life_exp), 
            mean_fert_rate = mean(fert_rate),
            median_life_exp = median(life_exp), 
            median_fert_rate = median(fert_rate)) |> 
  kbl() |>
  kable_styling(
    font_size = ifelse(is_latex_output(), 10, 16),
    latex_options = c("HOLD_position")
  )
```

However, what if we want other summary statistics as well, such as the standard deviation (a measure of spread), the minimum and maximum values, and various percentiles? 

Typing all these summary statistic functions in `summarize()` would be long and tedious. Instead, we use the convenient [`tidy_summary()`](https://moderndive.github.io/moderndive/reference/tidy_summary.html) function in the `moderndive` \index{R packages!moderndive!tidy\_summary()} package. This function takes in a data frame, summarizes it, and returns commonly used summary statistics in tidy format. We take our `UN_data_ch5` data frame, `select()` only the outcome and explanatory variables `fert_rate` and `life_exp`, and pipe them into the `tidy_summary` function:

```{r regression-select-vars, eval=FALSE}
UN_data_ch5 |> 
  select(fert_rate, life_exp) |> 
  tidy_summary()
```

```{r regression-select-vars-sized, echo=FALSE}
UN_data_ch5 |> 
  select(fert_rate, life_exp) |> 
  tidy_summary() |> 
  kbl() |>
  kable_styling(
    font_size = ifelse(is_latex_output(), 10, 16),
    latex_options = c("HOLD_position")
  )
```

We can also do this more directly by providing which `columns` we'd like a summary of inside the `tidy_summary()` function:

```{r regression-alt, eval=FALSE}
UN_data_ch5 |> 
  tidy_summary(columns = c(fert_rate, life_exp))
```

```{r regression-conditional, echo=FALSE}
UN_data_ch5 |> 
  tidy_summary(columns = c(fert_rate, life_exp)) |> 
  kbl() |>
  kable_styling(
    font_size = ifelse(is_latex_output(), 10, 16),
    latex_options = c("HOLD_position")
  )
```


Both return the same results for the numerical variables `fert_rate` and `life_exp`:

- `column`: the name of the column being summarized
- `n`: the number of non-missing values
- `group`: `NA` (missing) for numerical columns, but will break down a categorical variable into its levels
- `type`: which type of column is it (`numeric`, `character`, `factor`, or `logical`)
- `min`: the *minimum* value
- `Q1`: the 1st quartile: the value at which 25% of observations are smaller than it (the *25th percentile*)
- `mean`: the average value for measuring central tendency
- `median`: the 2nd quartile: the value at which 50% of observations are smaller than it (the *50th percentile*)
- `Q3`: the 3rd quartile: the value at which 75% of observations are smaller than it (the *75th percentile*)
- `max`: the *maximum* value
- `sd`: the standard deviation value for measuring spread


```{r regression-create-summary_df, echo=FALSE}
summary_df <- UN_data_ch5 |> 
  select(fert_rate, life_exp) |> 
  tidy_summary() |> 
  filter(type == "numeric")
fert_summary_df <- summary_df  |> 
  filter(column == "fert_rate")
life_summary_df <- summary_df  |>
  filter(column == "life_exp")
```

Looking at this output, we can see how the values of both variables distribute. For example, the median fertility rate was `r fert_summary_df$median[1]`, whereas the median life expectancy was `r life_summary_df$median[1]` years. The middle 50% of fertility rates was between `r fert_summary_df$Q1[[1]]` and `r fert_summary_df$Q3[[1]]` (the first and third quartiles), and the middle 50% of life expectancies was from `r life_summary_df$Q1[[1]]` to `r life_summary_df$Q3[[1]]`.

The `tidy_summary()` function only returns what are known as *univariate* \index{univariate} summary statistics: functions that take a single variable and return some numerical summary of that variable. However, there also exist *bivariate* \index{bivariate} summary statistics: functions that take in two variables and return some summary of those two variables. 

In particular, when the two variables are numerical, we can compute the \index{correlation (coefficient)} *correlation coefficient*. Generally speaking, *coefficients* are quantitative expressions of a specific phenomenon. A *correlation coefficient* measures the *strength of the linear relationship between two numerical variables*. Its value goes from -1 and 1 where:

* -1 indicates a perfect *negative relationship*: 
  As one variable increases, the value of the other variable tends to go down, following a straight line.
* 0 indicates no relationship: 
  The values of both variables go up/down independently of each other.
* +1 indicates a perfect *positive relationship*: 
  As the value of one variable goes up, the value of the other variable tends to go up as well in a linear fashion.

Figure \@ref(fig:correlation1) gives examples of nine different correlation coefficient values for hypothetical numerical variables $x$ and $y$. 

```{r correlation1, echo=FALSE, fig.cap="Nine different correlation coefficients.", fig.height=ifelse(knitr::is_latex_output(), 3, 4), purl=FALSE}
correlation <- c(-0.9999, -0.9, -0.75, -0.3, 0, 0.3, 0.75, 0.9, 0.9999)
n_sim <- 100
values <- NULL
for (i in seq_along(correlation)) {
  rho <- correlation[i]
  sigma <- matrix(c(5, rho * sqrt(50), rho * sqrt(50), 10), 2, 2)
  sim <- rmvnorm(
    n = n_sim,
    mean = c(20, 40),
    sigma = sigma
  ) |>
    as.data.frame() |>
    as_tibble() |>
    mutate(correlation = round(rho, 2))
  
  values <- bind_rows(values, sim)
}

corr_plot <- ggplot(data = values, mapping = aes(V1, V2)) +
  geom_point() +
  facet_wrap(~correlation, ncol = 3) +
  labs(x = "x", y = "y") +
  theme(
    axis.text.x = element_blank(),
    axis.text.y = element_blank(),
    axis.ticks = element_blank()
  )

if (is_latex_output()) {
  corr_plot +
    theme(
      strip.text = element_text(colour = "black"),
      strip.background = element_rect(fill = "grey93")
    )
} else {
  corr_plot
}
```

For example, observe in the top right plot that for a correlation coefficient of -0.75 there is a negative linear relationship between $x$ and $y$, but it is not as strong as the negative linear relationship between $x$ and $y$ when the correlation coefficient is -0.9 or -1.

The correlation coefficient can be computed using the [`get_correlation()`](https://moderndive.github.io/moderndive/reference/get_correlation.html) \index{R packages!moderndive!get\_correlation()} function in the `moderndive` package. In this case, the inputs to the function are the two numerical variables for which we want to calculate the correlation coefficient. 

We put the name of the outcome variable on the left-hand side of the `~` "tilde" sign, while putting the name of the explanatory variable on the right-hand side. This is known as R's \index{R!formula notation} *formula notation*. We will use this same "formula" syntax with regression later in this chapter.

```{r regression-demo-code}
UN_data_ch5 |> 
  get_correlation(formula = fert_rate ~ life_exp)
```

An alternative way to compute correlation is to use the `cor()` summary function within a `summarize()`:

```{r regression-summarize, eval=FALSE}
UN_data_ch5 |> 
  summarize(correlation = cor(fert_rate, life_exp))
```

```{r regression-summarize-v2, echo=FALSE, purl=FALSE}
cor_ch5 <- UN_data_ch5 |>
  summarize(correlation = cor(fert_rate, life_exp,
                              use = "complete.obs")) |>
  round(3) |>
  pull()
```

In our case, the correlation coefficient of `r cor_ch5` indicates that the relationship between fertility rate and life expectancy is "moderately negative." There is a certain amount of subjectivity in interpreting correlation coefficients, especially those that are not close to the extreme values of -1, 0, and 1. To develop your intuition about correlation coefficients, play the "Guess the Correlation" 1980's style video game mentioned in Subsection \@ref(additional-resources-basic-regression).

We now perform the last step in EDA: creating data visualizations. Since both the `fert_rate` and `life_exp` variables are numerical, a scatterplot is an appropriate graph to visualize this data. We do this using `geom_point()` and display the result in Figure \@ref(fig:numxplot1). Furthermore, we set the `alpha` value to `0.1` to check for any overplotting.

```{r numxplot1, fig.cap="Scatterplot of relationship of life expectancy and fertility rate.", fig.height=ifelse(knitr::is_latex_output(), 4.5, 5)}
ggplot(UN_data_ch5, 
       aes(x = life_exp, y = fert_rate)) +
  geom_point(alpha = 0.1) +
  labs(x = "Life Expectancy", y = "Fertility Rate")
```

We do not see much for overplotting due to little to no overlap in the points. Most life expectancy entries appear to fall between 70 and 80 years, while most fertility rate entries fall between 1.5 and 3.5 births. Furthermore, while opinions may vary, it is our opinion that the relationship between fertility rate and life expectancy is "moderately negative." This is consistent with our earlier computed correlation coefficient of `r cor_ch5`.

We build on the scatterplot in Figure \@ref(fig:numxplot1) by adding a "best-fitting" line: of all possible lines we can draw on this scatterplot, it is the line that "best" fits through the cloud of points. We do this by adding a new `geom_smooth(method = "lm", se = FALSE)` layer to the `ggplot()` code that created the scatterplot in Figure \@ref(fig:numxplot1). The `method = "lm"` argument sets the line to be a "**l**inear **m**odel." The `se = FALSE` \index{R packages!ggplot2!geom\_smooth()} argument suppresses _standard error_ uncertainty bars. (We'll define the concept of _standard error_ later in Subsection \@ref(sampling-variation).)

```{r numxplot3, fig.cap="Scatterplot of life expectancy and fertility rate with regression line.", message=FALSE, fig.height=ifelse(knitr::is_latex_output(), 4, 5)}
ggplot(UN_data_ch5, aes(x = life_exp, y = fert_rate)) +
  geom_point(alpha = 0.1) +
  labs(x = "Life Expectancy", 
    y = "Fertility Rate",
    title = "Relationship of life expectancy and fertility rate") +
  geom_smooth(method = "lm", se = FALSE)
```

The line in the resulting Figure \@ref(fig:numxplot3) is called a "regression line." The regression line \index{regression!line} is a visual summary of the relationship between two numerical variables, in our case the outcome variable `fert_rate` and the explanatory variable `life_exp`. The negative slope of the blue line is consistent with our earlier observed correlation coefficient of `r cor_ch5` suggesting that there is a negative relationship between these two variables: as a country's population has higher life expectancy it tends to have a lower fertility rate. We'll see later, however, that while the correlation coefficient and the slope of a regression line always have the same sign (positive or negative), they typically do not have the same value.

Furthermore, a regression line is "best-fitting" in that it minimizes some mathematical criteria. We present these mathematical criteria in Subsection \@ref(leastsquares), but we suggest you read this subsection only after first reading the rest of this section on regression with one numerical explanatory variable.

```{block, type="learncheck", purl=FALSE}
\vspace{-0.15in}
**_Learning check_**
\vspace{-0.1in}
```

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** Conduct a new exploratory data analysis with the same outcome variable $y$ being `fert_rate` but with `obes_rate` as the new explanatory variable $x$. Remember, this involves three things:

(a) Looking at the raw data values.
(a) Computing summary statistics.
(a) Creating data visualizations.

What can you say about the relationship between obesity rate and fertility rate based on this exploration?

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`**
What is the main purpose of performing an exploratory data analysis (EDA) before fitting a regression model?

- A. To predict future values. 
- B. To understand the relationship between variables and detect potential issues. 
- C. To create more variables. 
- D. To generate random samples. 

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** Which of the following is correct about the correlation coefficient?

- A. It ranges from -2 to 2.
- B. It only measures the strength of non-linear relationships.
- C. It ranges from -1 to 1 and measures the strength of linear relationships.
- D. It is always zero.

```{block, type="learncheck", purl=FALSE}
\vspace{-0.25in}
\vspace{-0.25in}
```


### Simple linear regression {#model1table}

You may recall from secondary/high school algebra that the equation of a line is $y = a + b\cdot x$. (Note that the $\cdot$ symbol is equivalent to the $\times$ "multiply by" mathematical symbol. We'll use the $\cdot$ symbol in the rest of this book as it is more succinct.) It is defined by two coefficients $a$ and $b$. The intercept coefficient $a$ is the value of $y$ when $x = 0$. The slope coefficient $b$ for $x$ is the increase in $y$ for every increase of one in $x$. This is also called the "rise over run."

However, when defining a regression line like the one in Figure \@ref(fig:numxplot3), we use slightly different notation: the equation of the regression line is $\widehat{y} = b_0 + b_1 \cdot x$ \index{regression!equation of a line}. The intercept coefficient is $b_0$, so $b_0$ is the value of $\widehat{y}$ when $x = 0$. The slope coefficient for $x$ is $b_1$, i.e., the increase in $\widehat{y}$ for every increase of one in $x$. Why do we put a "hat" on top of the $y$? It's a form of notation commonly used in regression to indicate that we have a \index{regression!fitted value} "fitted value," or the value of $y$ on the regression line for a given $x$ value as discussed further in Subsection \@ref(model1points).

We know that the regression line in Figure \@ref(fig:numxplot3) has a negative slope $b_1$ corresponding to our explanatory $x$ variable `life_exp`. Why? Because as countries tend to have higher `life_exp` values, they tend to have lower `fert_rate` values. However, what is the numerical value of the slope $b_1$? What about the intercept $b_0$?  We do not compute these two values by hand, but rather we use a computer!

We can obtain the values of the intercept $b_0$ and the slope for `life_exp` $b_1$ by outputting the *linear regression coefficients*. This is done in two steps:

1. We first "fit" the linear regression model using the `lm()` function and save it in `demographics_model`.
1. We get the regression coefficients by applying `coef()`  to `demographics_model`.

```{r regression-lm-fertility, eval=FALSE}
# Fit regression model:
demographics_model <- lm(fert_rate ~ life_exp, data = UN_data_ch5)
# Get regression coefficients
coef(demographics_model)
```
```{r regression-lm-fertility-alt, echo=FALSE, purl=FALSE}
demographics_model <- lm(fert_rate ~ life_exp, 
                         data = UN_data_ch5)
demo_line <- demographics_model |>
  get_regression_table() |>
  pull(estimate)
```

We first focus on interpreting the regression coefficients, and later revisit the code that produced it. The coefficients are the intercept $b_0 = `r demo_line[1]`$ and the slope $b_1 = `r demo_line[2]`$ for `life_exp`. Thus the equation of the regression line in Figure \@ref(fig:numxplot3) follows:

$$
\begin{aligned}
\widehat{y} &= b_0 + b_1 \cdot x\\
\widehat{\text{fertility}\_\text{rate}} &= b_0 + b_{\text{life}\_\text{expectancy}} \cdot \text{life}\_\text{expectancy}\\
&= `r demo_line[1] |> round(3)` + (`r demo_line[2]`) \cdot \text{life}\_\text{expectancy}
\end{aligned}
$$

The intercept $b_0$ = `r demo_line[1]` is the average fertility rate $\widehat{y}$ = $\widehat{\text{fertility}\_\text{rate}}$ for those countries that had a `life_exp` of 0. Or in graphical terms, where the line intersects the $y$ axis for $x$ = 0. Note, however, that while the \index{regression!equation of a line!intercept} intercept of the regression line has a mathematical interpretation, it has no *practical* interpretation here, since observing a `life_exp` of 0 is impossible. Furthermore, looking at the scatterplot with the regression line in Figure \@ref(fig:numxplot3), no countries had a life expectancy anywhere near 0.

Of greater interest is the \index{regression!equation of a line!slope} slope $b_{\text{life}\_\text{expectancy}}$ for `life_exp` of `r demo_line[2]`. This summarizes the relationship between the fertility rate and life expectancy variables. Note that the sign is negative, suggesting a negative relationship between these two variables. This means countries with higher life expectancies tend to have lower fertility rates. Recall that the correlation coefficient is `r cor_ch5`. They both have the same negative sign, but have a different value. Recall also that the correlation's interpretation is the "strength of linear association." The \index{regression!interpretation of the slope} slope's interpretation is a little different:

> For every increase of 1 unit in `life_exp`, there is an *associated* decrease of, *on average*, `r abs(demo_line[2])` units of `fert_rate`.

We only state that there is an *associated* increase and not necessarily a *causal* increase. Perhaps it may not be that higher life expectancies directly cause lower fertility rates. Instead, wealthier countries could tend to have stronger educational backgrounds, improved health, a higher standard of living, and have lower fertility rates, while at the same time these wealthy countries also tend to have higher life expectancies. Just because two variables are strongly associated, it does not necessarily mean that one causes the other. This is summed up in the often-quoted phrase, "correlation is not necessarily causation."  We discuss this idea further in Subsection \@ref(correlation-is-not-causation).  

Furthermore, we say that this associated decrease is *on average* `r abs(demo_line[2])` units of `fert_rate`, because you might have two countries whose `life_exp` values differ by 1 unit, but their difference in fertility rates may not be exactly $`r demo_line[2]`$. What the slope of $`r demo_line[2]`$ is saying is that across all possible countries, the *average* difference in fertility rate between two countries whose life expectancies differ by one is $`r demo_line[2]`$.

Now that we have learned how to compute the equation for the regression line in Figure \@ref(fig:numxplot3) using the model coefficient values and how to interpret the resulting intercept and slope, we revisit the code that generated these coefficients:

```{r regression-lm-fertility-alt2, eval=FALSE}
# Fit regression model:
demographics_model <- lm(fert_rate ~ life_exp, data = UN_data_ch5)
# Get regression coefficients:
coef(demographics_model)
```

First, we "fit" the linear regression model to the `data` using the `lm()` \index{lm()} function and save this as `demographics_model`. When we say "fit," we mean "find the best fitting line to this data." `lm()` stands for "linear model" and is used as `lm(y ~ x, data = df_name)` where:

* `y` is the outcome variable, followed by a tilde `~`. In our case, `y` is set to `fert_rate`.
* `x` is the explanatory variable. In our case, `x` is set to `life_exp`.
* The combination of `y ~ x` is called a *model formula*. (Note the order of `y` and `x`.) In our case, the model formula is `fert_rate ~ life_exp`. We saw such model formulas earlier with the [`get_correlation()`](https://moderndive.github.io/moderndive/reference/get_correlation.html) function in Subsection \@ref(model1EDA).
* `df_name` is the name of the data frame that contains the variables `y` and `x`. In our case, `data` is the `UN_data_ch5` data frame.

Second, we take the saved model in `demographics_model` and apply the `coef()` function to it to obtain the regression coefficients. This gives us the components of the regression equation line: the intercept $b_0$ and the slope $b_1$. 

```{block, type="learncheck", purl=FALSE}
\vspace{-0.15in}
**_Learning check_**
\vspace{-0.1in}
```

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** Fit a simple linear regression using `lm(fert_rate ~ obes_rate, data = UN_data_ch5)` where `obes_rate` is the new explanatory variable $x$. Learn about the "best-fitting" line from the regression coefficients by applying the `coef()` function. How do the regression results match up with your earlier exploratory data analysis?

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** What does the intercept term $b_0$ represent in simple linear regression? 

- A. The change in the outcome for a one-unit change in the explanatory variable.
- B. The predicted value of the outcome when the explanatory variable is zero.
- C. The standard error of the regression.
- D. The correlation between the outcome and explanatory variables.

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** What best describes the "slope" of a simple linear regression line?  

- A. The increase in the explanatory variable for a one-unit increase in the outcome.
- B. The average of the explanatory variable.
- C. The change in the outcome for a one-unit increase in the explanatory variable.
- D. The minimum value of the outcome variable.

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** What does a negative slope in a simple linear regression indicate?

- A. The outcome variable decreases as the explanatory variable increases.
- B. The explanatory variable remains constant as the outcome variable increases.
- C. The correlation coefficient is zero.
- D. The outcome variable increases as the explanatory variable increases.

```{block, type="learncheck", purl=FALSE}
\vspace{-0.25in}
\vspace{-0.25in}
```


### Observed/fitted values and residuals {#model1points}

We just saw how to get the value of the intercept and the slope of a regression line from the output of the `coef()` function. Now instead say we want information on individual observations. For example, we focus on the 21st of the `r n_demo_ch5` countries in the `UN_data_ch5` data frame in Table \@ref(tab:country-21). This corresponds to the UN member state of Bosnia and Herzegovina (BIH).

```{r country-21, echo=FALSE, purl=FALSE}
index <- which(UN_data_ch5$iso == "BIH")
target_point <- demographics_model |>
  get_regression_points() |>
  slice(index)
x <- target_point$life_exp
y <- target_point$fert_rate
y_hat <- target_point$fert_rate_hat
resid <- target_point$residual
UN_data_ch5 |>
  slice(index) |>
  kbl(
#    digits = 4,
    caption = "Data for the 21st country out of 193",
    booktabs = TRUE,
    linesep = ""
  ) |>
  kable_styling(
    font_size = ifelse(is_latex_output(), 10, 16),
    latex_options = c("HOLD_position")
  )
```

What is the value $\widehat{y}$ on the regression line corresponding to this country's `life_exp` value of `r x`? In Figure \@ref(fig:numxplot4) we mark three values corresponding to these results for Bosnia and Herzegovina and give their statistical names:

* Circle: The *observed value* $y$ = `r y` is this country's actual fertility rate.
* Square: The *fitted value* $\widehat{y}$ is the value on the regression line for $x =  \texttt{life\_exp} = `r x`$, computed with the intercept and slope in the previous regression table: 

$$\widehat{y} = b_0 + b_1 \cdot x = `r demo_line[1]` + (`r demo_line[2]`) \cdot `r x` = `r y_hat`$$
* Arrow: The length of this arrow is the *residual* \index{regression!residual} and is computed by subtracting the fitted value $\widehat{y}$ from the observed value $y$. The residual can be thought of as a model's error or "lack of fit" for a particular observation.  In the case of this country, it is $y - \widehat{y} = `r y` - `r y_hat` = `r resid`$.

```{r numxplot4, echo=FALSE, fig.cap="Example of observed value, fitted value, and residual.", fig.height=ifelse(knitr::is_latex_output(), 3, 4), message=FALSE, purl=FALSE}
best_fit_plot <- ggplot(UN_data_ch5, aes(x = life_exp, y = fert_rate)) +
  geom_point(color = "grey") +
  labs(
    x = "Life Expectancy", y = "Fertility Rate",
    title = "Relationship of Fertility Rate and Life Expectancy"
  ) +
  geom_smooth(method = "lm", se = FALSE) +
  annotate("point", x = x, y = y_hat, col = "red", shape = 15, size = 4) +
  annotate("segment",
           x = x, xend = x, y = y, yend = y_hat, color = "blue",
           arrow = arrow(type = "closed", length = unit(0.04, "npc"))
  ) +
  annotate("point", x = x, y = y, col = "red", size = 4)
best_fit_plot
```

Now say we want to compute both the fitted value $\widehat{y} = b_0 + b_1 \cdot x$ and the residual $y - \widehat{y}$ for *all* `r n_demo_ch5` UN member states with complete data as of 2024. Recall that each country corresponds to one of the `r n_demo_ch5` rows in the `UN_data_ch5` data frame and also one of the `r n_demo_ch5` points in the regression plot in Figure \@ref(fig:numxplot4).

We could repeat the previous calculations we performed by hand `r n_demo_ch5` times, but that would be tedious and time consuming. Instead, we use a computer with the `get_regression_points()` function. We apply the `get_regression_points()` function to `demographics_model`, which is where we saved our `lm()` model in the previous section. In Table \@ref(tab:regression-points-1) we present the results of only the 21st through 24th courses for brevity.

```{r regression-reg-points, eval=FALSE}
regression_points <- get_regression_points(demographics_model)
regression_points
```

```{r regression-points-alt, echo=FALSE, purl=FALSE}
set.seed(76)
regression_points <- get_regression_points(demographics_model)
regression_points |>
  slice(c(index, index + 1, index + 2, index + 3)) |>
  kable(
    digits = 3,
    caption = "Regression points (for only the 21st through 24th countries)",
    booktabs = TRUE,
    linesep = ""
  )

# This code is used for dynamic non-static in-line text output purposes
n_regression_points <- regression_points |> nrow()
fert_rate_24 <- regression_points$fert_rate[24]
life_exp_24 <- regression_points$life_exp[24] |> format(round(2), nsmall = 3)
fert_rate_hat_24 <- regression_points$fert_rate_hat[24] |> round(3) |> format(nsmall = 3)
residual_24 <- regression_points$residual[24] |> round(3) |> format(nsmall = 3)
```

This function is an example of what is known in computer programming as a *wrapper function*. \index{functions!wrapper} It takes other pre-existing functions and "wraps" them into a single function that hides its inner workings. This concept is illustrated in Figure \@ref(fig:moderndive-figure-wrapper).

```{r moderndive-figure-wrapper, echo=FALSE, fig.cap="The concept of a wrapper function.", out.width="50%", purl=FALSE}
include_graphics("images/shutterstock/wrapper_function.png")
```

So all you need to worry about is what the inputs look like and what the outputs look like; you leave all the other details "under the hood of the car." In our regression modeling example, the `get_regression_points()` function takes a saved `lm()` linear regression model as input and returns a data frame of the regression predictions as output. If you are interested in learning more about the `get_regression_points()` function's inner workings, check out Subsection \@ref(underthehood).

We inspect the individual columns and match them with the elements of Figure \@ref(fig:numxplot4):

* The `fert_rate` column represents the observed outcome variable $y$. This is the y-position of the `r n_regression_points` black points.
* The `life_exp` column represents the values of the explanatory variable $x$. This is the x-position of the `r n_regression_points` black points.
* The `fert_rate_hat` column represents the fitted values $\widehat{y}$. This is the corresponding value on the regression line for the `r n_regression_points` $x$ values.
* The `residual` column represents the residuals $y - \widehat{y}$. This is the `r n_regression_points` vertical distances between the `r n_regression_points` black points and the regression line.

Just as we did for the 21st country in the `UN_data_ch5` dataset (in the first row of the table), we repeat the calculations for the 24th country (in the fourth row of Table \@ref(tab:regression-points-1)). This corresponds to the country of Brunei (BRN):

* `fert_rate` $= `r fert_rate_24`$ is the observed `fert_rate` $y$ for this country.
* `life_exp` $= `r life_exp_24`$ is the value of the explanatory variable `life_exp` $x$ for Brunei.
* `fert_rate_hat` $= `r fert_rate_hat_24` = `r demo_line[1]` + (`r demo_line[2]`) \cdot `r life_exp_24`$ is the fitted value $\widehat{y}$ on the regression line for this country.
* `residual` $= `r residual_24` = `r fert_rate_24` - `r fert_rate_hat_24`$ is the value of the residual for this country. In other words, the model's fitted value was off by `r residual_24` fertility rate units for Brunei.

If you like, you can skip ahead to Subsection \@ref(leastsquares) to learn about the processes behind what makes "best-fitting" regression lines. As a primer, a "best-fitting" line refers to the line that minimizes the *sum of squared residuals* out of all possible lines we can draw through the points. In Section \@ref(model2), we'll discuss another common scenario of having a categorical explanatory variable and a numerical outcome variable.

```{block, type="learncheck", purl=FALSE}
\vspace{-0.15in}
**_Learning check_**
\vspace{-0.1in}
```

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** What is a "wrapper function" in the context of statistical modeling in R?  

- A. A function that directly fits a regression model without using any other functions.
- B. A function that combines other functions to simplify complex operations and provide a user-friendly interface.
- C. A function that removes missing values from a dataset before analysis.
- D. A function that only handles categorical data in regression models.

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** Generate a data frame of the residuals of the *Learning check* model where you used `obes_rate` as the explanatory $x$ variable.

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** Which of the following statements is true about the regression line in a simple linear regression model?

- A. The regression line represents the average of the outcome variable.
- B. The regression line minimizes the sum of squared differences between the observed and predicted values.
- C. The regression line always has a slope of zero.
- D. The regression line is only useful when there is no correlation between variables.

```{block, type="learncheck", purl=FALSE}
\vspace{-0.25in}
\vspace{-0.25in}
```

## One categorical explanatory variable {#model2}

It is an unfortunate truth that life expectancy is not the same across all countries in the world. International development agencies are interested in studying these differences in life expectancy in the hopes of identifying where governments should allocate resources to address this problem. In this section, we explore differences in life expectancy in two ways:

1. Differences between continents: Are there significant differences in average life expectancy between the six populated continents of the world: Africa, North America, South America, Asia, Europe, and Oceania?
1. Differences within continents: How does life expectancy vary within the world's five continents? For example, is the spread of life expectancy among the countries of Africa larger than the spread of life expectancy among the countries of Asia?

To answer such questions, we use an updated version of the `gapminder` data frame we visualized in Figure \@ref(fig:gapminder) in Subsection \@ref(gapminder) on the grammar of graphics. This updated `un_member_states_2024` data we saw earlier in this chapter. It is included in the `moderndive` \index{R packages!moderndive} package and has international development statistics such as life expectancy, GDP per capita, and population for `r n_countries` countries for years near 2024. We use this data for basic regression again, but now using an explanatory variable $x$ that is categorical, as opposed to the numerical explanatory variable model we used in the previous Section \@ref(model1):

1. A numerical outcome variable $y$ (a country's life expectancy) and
1. A single categorical explanatory variable $x$ (the continent that the country is a part of).

When the explanatory variable $x$ is categorical, the concept of a "best-fitting" regression line is a little different than the one we saw previously in Section \@ref(model1) where the explanatory variable $x$ was numerical. We study these differences shortly in Subsection \@ref(model2table), but first we conduct an exploratory data analysis.


### Exploratory data analysis {#model2EDA}

The data on the `r n_countries` countries can be found in the `un_member_states_2024` data frame included in the `moderndive` package. However, to keep things simple, we `select()` only the subset of the variables we'll consider in this chapter and focus only on rows where we have no missing values with `na.omit()`. We'll save this data in a new data frame called `gapminder2022`:

```{r regression-create-gapminder2022, message=FALSE}
gapminder2022 <- un_member_states_2024 |>
  select(country, life_exp = life_expectancy_2022, continent, gdp_per_capita) |> 
  na.omit()
```

```{r regression-create-life_exp_worldwide, echo=FALSE, purl=FALSE}
# Hidden: internally compute mean and median life expectancy
life_exp_worldwide <- gapminder2022 |>
  summarize(median = median(life_exp, na.rm = TRUE), 
            mean = mean(life_exp, na.rm = TRUE))
mean_africa <- gapminder2022 |>
  filter(continent == "Africa") |>
  summarize(mean_africa = mean(life_exp)) |>
  pull(mean_africa)

# This code is used for dynamic non-static in-line text output purposes
gapminder2022_rows <- gapminder2022 |> nrow()
```

We perform the first common step in an exploratory data analysis: looking at the raw data values. You can do this by using RStudio's spreadsheet viewer or by using the `glimpse()` command as introduced in Subsection \@ref(exploredataframes) on exploring data frames:

```{r regression-glimpse-gapminder2022}
glimpse(gapminder2022)
```

Observe that ``Rows: `r gapminder2022_rows` `` indicates that there are `r gapminder2022_rows` rows/observations in `gapminder2022`, where each row corresponds to one country. In other words, the *observational unit* is an individual country. Furthermore, observe that the variable `continent` is of type `<fct>`, which stands for _factor_, which is R's way of encoding categorical variables.

A full description of all the variables included in `un_member_states_2024` can be found by reading the associated help file (run [`?un_member_states_2024`](https://moderndive.github.io/moderndive/reference/un_member_states_2024.html) in the console). However, we fully describe only the `r ncol(gapminder2022)` variables we selected in `gapminder2022`:

1. `country`: An identification variable of type character/text used to distinguish the `r gapminder2022_rows` countries in the dataset.
1. `life_exp`: A numerical variable of that country's life expectancy at birth. This is the outcome variable $y$ of interest.
1. `continent`: A categorical variable with five levels. Here "levels" correspond to the possible categories: Africa, Asia, Americas, Europe, and Oceania. This is the explanatory variable $x$ of interest.
1. `gdp_per_capita`: A numerical variable of that country's GDP per capita in US inflation-adjusted dollars that we'll use as another outcome variable $y$ in the *Learning check* at the end of this subsection.

We next look at a random sample of three out of the `r gapminder2022_rows` countries in Table \@ref(tab:model2-data-preview). 

```{r regression-alt2, eval=FALSE}
gapminder2022 |> sample_n(size = 3)
```
```{r model2-data-preview, echo=FALSE, purl=FALSE}
gapminder2022 |>
  sample_n(3) |>
  kbl(
    digits = 3,
    caption = "Random sample of 3 out of 188 countries",
    booktabs = TRUE,
    linesep = ""
  ) |>
  kable_styling(
    font_size = ifelse(is_latex_output(), 10, 16),
    latex_options = c("HOLD_position")
  )
```

Random sampling will likely produce a different subset of 3 rows for you than what's shown. Now that we have looked at the raw values in our `gapminder2022` data frame and got a sense of the data, we compute summary statistics. We again apply `tidy_summary()` from the `moderndive` package. Recall that this function takes in a data frame, summarizes it, and returns commonly used summary statistics. We take our `gapminder2022` data frame, `select()` only the outcome and explanatory variables `life_exp` and `continent`, and pipe them into `tidy_summary()` in Table \@ref(tab:lifeexp-cont):

```{r regression-select-vars-alt, eval=FALSE}
gapminder2022 |> select(life_exp, continent) |> tidy_summary()
```

```{r lifeexp-cont, echo=FALSE}
gapminder2022 |>
  select(life_exp, continent) |>
  tidy_summary() |> 
  kbl(
    caption = "Summary of life expectancy and continent variables",
    booktabs = TRUE,
    linesep = ""
  ) |>
  kable_styling(
    font_size = ifelse(is_latex_output(), 9, 16),
    latex_options = c("HOLD_position")
  )
```

```{r regression-alt2-dup1, include=FALSE}
# For discussion in bullet points below
gapminder2022 |> count(continent)
```

The `tidy_summary()` output now reports summaries for categorical variables and for the numerical variables we reviewed before. Let's focus just on discussing the results for the categorical `factor` variable `continent`:

- `n`: The number of non-missing entries for each group
- `group`: Breaks down a categorical variable into its unique levels. For this variable, it is corresponding to Africa, Asia, North and South America, Europe, and Oceania. 
- `type`: The data type of the variable. Here, it is a `factor`.
- `min` to `sd`: These are missing since calculating the five-number summary, the mean, and standard deviation for categorical variables doesn't make sense.

```{r regression-assign-median_life_exp, echo=FALSE, purl=FALSE}
# This code is used for dynamic non-static in-line text output purposes
median_life_exp <- life_summary_df$median |> round(2)
mean_life_exp <- life_summary_df$mean |> round(2)
```

Turning our attention to the summary statistics of the numerical variable `life_exp`, we observe that the global median life expectancy in 2022 was `r median_life_exp`. Thus, half of the world's countries (`r floor(n_countries/2)` countries) had a life expectancy of less than `r median_life_exp`. The mean life expectancy of `r mean_life_exp` is lower, however. Why is the mean life expectancy lower than the median?

We can answer this question by performing the last of the three common steps in an exploratory data analysis: creating data visualizations. We visualize the distribution of our outcome variable $y$ = `life_exp` in Figure \@ref(fig:lifeexp2022hist).

```{r lifeexp2022hist, echo=TRUE, fig.cap="Histogram of life expectancy in 2022.", fig.height=ifelse(knitr::is_latex_output(), 3, 4)}
ggplot(gapminder2022, aes(x = life_exp)) +
  geom_histogram(binwidth = 5, color = "white") +
  labs(x = "Life expectancy", 
       y = "Number of countries",
       title = "Histogram of distribution of worldwide life expectancies")
```

We see that this data is *left-skewed*, also known as *negatively* \index{skew} skewed: there are a few countries with low life expectancy that are bringing down the mean life expectancy. However, the median is less sensitive to the effects of such outliers; hence, the median is greater than the mean in this case.

Remember, however, that we want to compare life expectancies both between continents and within continents. In other words, our visualizations need to incorporate some notion of the variable `continent`. We can do this easily with a faceted histogram. Recall from Section \@ref(facets) that facets allow us to split a visualization by the different values of another variable. We display the resulting visualization in Figure \@ref(fig:catxplot0b) by adding a \index{R packages!ggplot2!facet\_wrap()} `facet_wrap(~ continent, nrow = 2)` layer.

```{r regression-facet-hist-lifeexp, eval=FALSE} 
ggplot(gapminder2022, aes(x = life_exp)) +
  geom_histogram(binwidth = 5, color = "white") +
  labs(x = "Life expectancy", 
       y = "Number of countries",
       title = "Histogram of distribution of worldwide life expectancies") +
  facet_wrap(~ continent, nrow = 2)
```

```{r catxplot0b, echo=FALSE, fig.cap="Life expectancy in 2022 by continent (faceted).", fig.height=ifelse(knitr::is_latex_output(), 4, 4), purl=FALSE}
faceted_life_exp <- ggplot(gapminder2022, aes(x = life_exp)) +
  geom_histogram(binwidth = 5, color = "white") +
  labs(
    x = "Life expectancy", y = "Number of countries",
    title = "Histogram of distribution of worldwide life expectancies"
  ) +
  facet_wrap(~continent, nrow = 2)

# Make the text black and reduce darkness of the grey in the facet labels
if (is_latex_output()) {
  faceted_life_exp +
    theme(
      strip.text = element_text(colour = "black"),
      strip.background = element_rect(fill = "grey93")
    )
} else {
  faceted_life_exp
}
```

Observe that unfortunately the distribution of African life expectancies is much lower than the other continents. In Europe, life expectancies tend to be higher and furthermore do not vary as much. On the other hand, both Asia and Africa have the most variation in life expectancies.

Recall that an alternative method to visualize the distribution of a numerical variable split by a categorical variable is by using a side-by-side boxplot. We map the categorical variable `continent` to the $x$-axis and the different life expectancies within each continent on the $y$-axis in Figure \@ref(fig:catxplot1).

```{r catxplot1, fig.cap="Life expectancy in 2022 by continent (boxplot).", fig.height=ifelse(knitr::is_latex_output(), 2.5, 4)}
ggplot(gapminder2022, aes(x = continent, y = life_exp)) +
  geom_boxplot() +
  labs(x = "Continent", y = "Life expectancy",
       title = "Life expectancy by continent")
```

Some people prefer comparing the distributions of a numerical variable between different levels of a categorical variable using a boxplot instead of a faceted histogram. This is because we can make quick comparisons between the categorical variable's levels with imaginary horizontal lines. For example, observe in Figure \@ref(fig:catxplot1) that we can quickly convince ourselves that Europe has the highest median life expectancies by drawing an imaginary horizontal line near $y = 81$. Furthermore, as we observed in the faceted histogram in Figure \@ref(fig:catxplot0b), Africa and Asia have the largest variation in life expectancy as evidenced by their large interquartile ranges (the size of the boxes).

It’s important to remember, however, that the solid lines in the middle of the boxes correspond to the medians (the middle value) rather than the mean (the average). So, for example, if you look at Asia, the solid line denotes the median life expectancy of around 75 years. This tells us that half of all countries in Asia have a life expectancy below 75 years, whereas half have a life expectancy above 75 years. We compute the median and mean life expectancy for each continent with a little more data wrangling and display the results in Table \@ref(tab:catxplot0).

```{r regression-grouped-summary, eval=TRUE, results='hide'}
life_exp_by_continent <- gapminder2022 |>
  group_by(continent) |>
  summarize(median = median(life_exp), mean = mean(life_exp))
life_exp_by_continent
```
```{r catxplot0, echo=FALSE, purl=FALSE}
life_exp_by_continent |>
  kbl(
    digits = 3,
    caption = "Life expectancy by continent",
    booktabs = TRUE,
    linesep = ""
  ) |>
  kable_styling(
    font_size = ifelse(is_latex_output(), 10, 16),
    latex_options = c("HOLD_position")
  )
```

Observe the order of the second column `median` life expectancy: Africa is lowest, Europe the highest, and the others have similar medians between Africa and Europe. This ordering corresponds to the ordering of the solid black lines inside the boxes in our side-by-side boxplot in Figure \@ref(fig:catxplot1). 

```{r regression-create-life_exp_model, echo=FALSE, purl=FALSE}
# This code is used for dynamic non-static in-line text output purposes
# Coding model earlier to calculate the intercepts etc below
life_exp_model <- lm(life_exp ~ continent, data = gapminder2022)

# Africa / Intercept
intercept <- get_regression_table(life_exp_model) |>
  filter(term == "intercept") |>
  pull(estimate) |>
  round(2)

mean_africa <- intercept

# Americas
offset_north_america <- get_regression_table(life_exp_model) |>
  filter(term == "continent: North America") |>
  pull(estimate) |>
  round(2)
mean_north_america <- intercept + offset_north_america

offset_south_america <- get_regression_table(life_exp_model) |>
  filter(term == "continent: South America") |>
  pull(estimate) |>
  round(2)
mean_south_america <- intercept + offset_south_america

# Asia
offset_asia <- get_regression_table(life_exp_model) |>
  filter(term == "continent: Asia") |>
  pull(estimate) |>
  round(2)
mean_asia <- intercept + offset_asia

# Europe
offset_europe <- get_regression_table(life_exp_model) |>
  filter(term == "continent: Europe") |>
  pull(estimate) |>
  round(2)
mean_europe <- intercept + offset_europe

# Oceania
offset_oceania <- get_regression_table(life_exp_model) |>
  filter(term == "continent: Oceania") |>
  pull(estimate) |>
  round(2)
mean_oceania <- intercept + offset_oceania
```

We now turn our attention to the values in the third column `mean`. Using Africa's mean life expectancy of `r intercept` as a *baseline for comparison*, we start making comparisons to the mean life expectancies of the other four continents and put these values in Table \@ref(tab:continent-mean-life-expectancies), which we'll revisit later on in this section.

1. For Asia, it is `r mean_asia` - `r intercept` = `r offset_asia` years higher.
1. For Europe, it is `r mean_europe` - `r intercept` = `r offset_europe` years higher.
1. For North America, it is `r mean_north_america` - `r intercept` = `r offset_north_america` years higher.
1. For Oceania, it is `r mean_oceania` - `r intercept` = `r offset_oceania` years higher.
1. For South America, it is `r mean_south_america` - `r intercept` = `r offset_south_america` years higher.

```{r continent-mean-life-expectancies, echo=FALSE, purl=FALSE}
gapminder2022 |>
  group_by(continent) |>
  summarize(mean = mean(life_exp, na.rm = TRUE)) |>
  mutate(`Difference versus Africa` = mean - mean_africa) |>
  kbl(
    digits = 2,
    caption = "Mean life expectancy by continent and relative differences from mean for Africa",
    booktabs = TRUE,
    linesep = ""
  ) |>
  kable_styling(
    font_size = ifelse(is_latex_output(), 10, 16),
    latex_options = c("HOLD_position")
  )
```

```{block, type="learncheck", purl=FALSE}
\vspace{-0.15in}
**_Learning check_**
\vspace{-0.1in}
```

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** Conduct a new exploratory data analysis with the same explanatory variable $x$ being `continent` but with `gdp_per_capita` as the new outcome variable $y$. What can you say about the differences in GDP per capita between continents based on this exploration?

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** When using a categorical explanatory variable in regression, what does the baseline group represent?  

  - A. The group with the highest mean
  - B. The group chosen for comparison with all other groups
  - C. The group with the most data points
  - D. The group with the lowest standard deviation

```{block, type="learncheck", purl=FALSE}
\vspace{-0.25in}
\vspace{-0.25in}
```


### Linear regression {#model2table}

In Subsection \@ref(model1table) we introduced simple linear regression, which involves modeling the relationship between a numerical outcome variable $y$ and a numerical explanatory variable $x$. In our life expectancy example, we now instead have a categorical explanatory variable `continent`. Our model will not yield a "best-fitting" regression line like in Figure \@ref(fig:numxplot3), but rather *offsets* relative to a baseline for comparison.\index{offset}

As we did in Subsection \@ref(model1table) when studying the relationship between fertility rates and life expectancy, we output the regression coefficients for this model. Recall that this is done in two steps:

1. We "fit" the linear regression model using `lm(y ~ x, data)` and save it in `life_exp_model`.
1. We get the regression coefficients by applying the `coef()` function to `life_exp_model`.

```{r regression-fit-lm}
life_exp_model <- lm(life_exp ~ continent, data = gapminder2022)
coef(life_exp_model)
```

We once again focus on the values in these coefficient values. Why are there now 6 entries? We break them down one by one:

1. `intercept` corresponds to the mean life expectancy of countries in Africa of `r intercept` years.

1. `continentAsia` corresponds to countries in Asia and the value +`r offset_asia` is the same difference in mean life expectancy relative to Africa we displayed in Table \@ref(tab:continent-mean-life-expectancies). In other words, the mean life expectancy of countries in Asia is $`r intercept` + `r offset_asia` = `r mean_asia`$.
1. `continentEurope` corresponds to countries in Europe and the value +`r offset_europe` is the same difference in mean life expectancy relative to Africa we displayed in Table \@ref(tab:continent-mean-life-expectancies). In other words, the mean life expectancy of countries in Europe is $`r intercept` + `r offset_europe` = `r mean_europe`$.
1. `continentNorth America` corresponds to countries in North America and the value +`r offset_north_america` is the same difference in mean life expectancy relative to Africa we displayed in Table \@ref(tab:continent-mean-life-expectancies). In other words, the mean life expectancy of countries in North America is $`r intercept` + `r offset_north_america` = `r mean_north_america`$.
1. `continentOceania` corresponds to countries in Oceania and the value +`r offset_oceania` is the same difference in mean life expectancy relative to Africa we displayed in Table \@ref(tab:continent-mean-life-expectancies). In other words, the mean life expectancy of countries in Oceania is $`r intercept` + `r offset_oceania` = `r mean_oceania`$.
1. `continentSouth America` corresponds to countries in South America and the value +`r offset_south_america` is the same difference in mean life expectancy relative to Africa we displayed in Table \@ref(tab:continent-mean-life-expectancies). In other words, the mean life expectancy of countries in South America is $`r intercept` + `r offset_south_america` = `r mean_south_america`$.

To summarize, the 6 values for the regression coefficients correspond to the "baseline for comparison" continent Africa (the intercept) as well as five "offsets" from this baseline for the remaining 5 continents: Asia, Europe, North America, Oceania, and South America.

You might be asking at this point why was Africa chosen as the "baseline for comparison" group. This is the case for no other reason than it comes first alphabetically of the six continents; by default R arranges factors/categorical variables in alphanumeric order. You can change this baseline group to be another continent if you manipulate the variable `continent`'s factor "levels" using the `forcats` package. See [Chapter 15](https://r4ds.had.co.nz/factors.html) of *R for Data Science* [@rds2016] for examples. 

We now write the equation for our fitted values $\widehat{y} = \widehat{\text{life exp}}$.

$$
\begin{aligned}
\widehat{y} = \widehat{\text{life exp}} &= b_0 + b_{\text{Asia}}\cdot\mathbb{1}_{\text{Asia}}(x) + b_{\text{Europe}}\cdot\mathbb{1}_{\text{Europe}}(x) \\
& \qquad + b_{\text{North America}}\cdot\mathbb{1}_{\text{North America}}(x) + b_{\text{Oceania}}\cdot\mathbb{1}_{\text{Oceania }}(x) \\
& \qquad + b_{\text{South America}}\cdot\mathbb{1}_{\text{South America}}(x)\\
&= `r intercept` + `r offset_asia`\cdot\mathbb{1}_{\text{Asia}}(x) + `r offset_europe`\cdot\mathbb{1}_{\text{Euro}}(x)  \\
& \qquad + `r offset_north_america`\cdot\mathbb{1}_{\text{North America}}(x) + `r offset_oceania`\cdot\mathbb{1}_{\text{Oceania}}(x) \\
& \qquad + `r offset_south_america`\cdot\mathbb{1}_{\text{South America}}(x)
\end{aligned}
$$

Whoa! That looks daunting! Don't fret, however, as once you understand what all the elements mean, things simplify greatly. First, $\mathbb{1}_{A}(x)$ is what's known in mathematics as an "indicator function." It returns only one of two possible values, 0 and 1, where

$$
\mathbb{1}_{A}(x) = \left\{
\begin{array}{ll}
1 & \text{if } x \text{ is in } A \\
0 & \text{if } \text{otherwise} \end{array}
\right.
$$

In a statistical modeling context, this is also known as a *dummy variable*. \index{dummy variable} In our case, we consider the first such indicator variable $\mathbb{1}_{\text{Amer}}(x)$. This indicator function returns 1 if a country is in the Asia, 0 otherwise:

$$
\mathbb{1}_{\text{Amer}}(x) = \left\{
\begin{array}{ll}
1 & \text{if } \text{country } x \text{ is in Asia} \\
0 & \text{otherwise}\end{array}
\right.
$$

Second, $b_0$ corresponds to the intercept as before; in this case, it is the mean life expectancy of all countries in Africa. Third, the $b_{\text{Asia}}$, $b_{\text{Europe}}$, $b_{\text{North America}}$, $b_{\text{Oceania}}$, and $b_{\text{South America}}$ represent the 5 "offsets relative to the baseline for comparison" in the regression coefficients.

We put this all together and compute the fitted value $\widehat{y} = \widehat{\text{life exp}}$ for a country in Africa. Since the country is in Africa, all five indicator functions $\mathbb{1}_{\text{Asia}}(x)$, $\mathbb{1}_{\text{Europe}}(x)$, $\mathbb{1}_{\text{North America}}(x)$, $\mathbb{1}_{\text{Oceania}}(x)$, and $\mathbb{1}_{\text{South America}}(x)$ will equal 0, and thus:

$$
\begin{aligned}
\widehat{\text{life exp}} &= b_0 + b_{\text{Asia}}\cdot\mathbb{1}_{\text{Asia}}(x)
+ b_{\text{Europe}}\cdot\mathbb{1}_{\text{Europe}}(x) \\
& \qquad  + b_{\text{North America}}\cdot\mathbb{1}_{\text{North America}}(x) +  b_{\text{Oceania}}\cdot\mathbb{1}_{\text{Oceania}}(x) \\
& \qquad + b_{\text{South America}}\cdot\mathbb{1}_{\text{South America}}(x) \\
&= `r intercept` + `r offset_asia`\cdot\mathbb{1}_{\text{Asia}}(x)
+ `r offset_europe`\cdot\mathbb{1}_{\text{Europe}}(x)\\
& \qquad + `r offset_north_america`\cdot\mathbb{1}_{\text{North America}}(x) + `r offset_oceania`\cdot\mathbb{1}_{\text{Oceania}}(x) \\
& \qquad + `r offset_south_america`\cdot\mathbb{1}_{\text{South America}}(x)\\
&= `r intercept` + `r offset_asia`\cdot 0 + `r offset_europe`\cdot 0 + `r offset_north_america`\cdot 0 + `r offset_oceania`\cdot 0 + `r offset_south_america`\cdot 0\\
&= `r intercept`
\end{aligned}
$$

In other words, all that is left is the intercept $b_0$, corresponding to the average life expectancy of African countries of `r intercept` years. Next, say we are considering a country in Asia. In this case, only the indicator function $\mathbb{1}_{\text{Asia}}(x)$ for Asia will equal 1, while all the others will equal 0, and thus:

$$
\begin{aligned}
\widehat{\text{life exp}} &= b_0 + b_{\text{Asia}}\cdot\mathbb{1}_{\text{Asia}}(x)
+ b_{\text{Europe}}\cdot\mathbb{1}_{\text{Europe}}(x)\\
& \qquad + b_{\text{North America}}\cdot\mathbb{1}_{\text{North America}}(x) +  b_{\text{Oceania}}\cdot\mathbb{1}_{\text{Oceania}}(x) \\
& \qquad + b_{\text{South America}}\cdot\mathbb{1}_{\text{South America}}(x) \\
&= `r intercept` + `r offset_asia`\cdot\mathbb{1}_{\text{Asia}}(x) 
+ `r offset_europe`\cdot\mathbb{1}_{\text{Europe}}(x) \\
& \qquad + `r offset_north_america`\cdot\mathbb{1}_{\text{North America}}(x) + `r offset_oceania`\cdot\mathbb{1}_{\text{Oceania}}(x) \\
& \qquad + `r offset_south_america`\cdot\mathbb{1}_{\text{South America}}(x)\\
&= `r intercept` + `r offset_asia`\cdot 1 + `r offset_europe`\cdot 0 + `r offset_north_america`\cdot 0 + `r offset_oceania`\cdot 0 + `r offset_south_america`\cdot 0\\
&= `r intercept` + `r offset_asia` \\
& = `r mean_asia`
\end{aligned}
$$

which is the mean life expectancy for countries in Asia of `r mean_asia` years in Table \@ref(tab:continent-mean-life-expectancies). Note the "offset from the baseline for comparison" is +`r offset_asia` years. 

We do one more. Say we are considering a country in South America. In this case, only the indicator function $\mathbb{1}_{\text{South America}}(x)$ for South America will equal 1, while all the others will equal 0, and thus:

$$
\begin{aligned}
\widehat{\text{life exp}} &= b_0 + b_{\text{Asia}}\cdot\mathbb{1}_{\text{Asia}}(x)
+ b_{\text{Europe}}\cdot\mathbb{1}_{\text{Europe}}(x) \\
& \qquad + b_{\text{North America}}\cdot\mathbb{1}_{\text{North America}}(x) +  b_{\text{Oceania}}\cdot\mathbb{1}_{\text{Oceania}}(x) \\
& \qquad + b_{\text{South America}}\cdot\mathbb{1}_{\text{South America}}(x) \\
&= `r intercept` + `r offset_asia`\cdot\mathbb{1}_{\text{Asia}}(x)
+ `r offset_europe`\cdot\mathbb{1}_{\text{Europe}}(x)\\
& \qquad + `r offset_north_america`\cdot\mathbb{1}_{\text{North America}}(x) + `r offset_oceania`\cdot\mathbb{1}_{\text{Oceania}}(x) + `r offset_south_america`\cdot\mathbb{1}_{\text{South America}}(x)\\
&= `r intercept` + `r offset_asia`\cdot 0 + `r offset_europe`\cdot 0 + `r offset_north_america`\cdot 0 + `r offset_oceania`\cdot 0 + `r offset_south_america`\cdot 1\\
&= `r intercept` + `r offset_south_america` \\
& = `r mean_south_america`
\end{aligned}
$$

which is the mean life expectancy for South American countries of `r mean_south_america` years in Table \@ref(tab:continent-mean-life-expectancies). The "offset from the baseline for comparison" here is +`r offset_asia` years.

We generalize this idea a bit. If we fit a linear regression model using a categorical explanatory variable $x$ that has $k$ possible categories, the regression table will return an intercept and $k - 1$ "offsets." In our case, since there are $k = 6$ continents, the regression model returns an intercept corresponding to the baseline for comparison group of Africa and $k - 1 = 5$ offsets corresponding to Asia, Europe, North America, Oceania, and South America.

Understanding a regression table output when you are using a categorical explanatory variable is a topic those new to regression often struggle with. The only real remedy for these struggles is practice, practice, practice. However, once you equip yourselves with an understanding of how to create regression models using categorical explanatory variables, you'll be able to incorporate many new variables into your models, given the large amount of the world's data that is categorical.


```{block, type="learncheck", purl=FALSE}
\vspace{-0.15in}
**_Learning check_**
\vspace{-0.1in}
```

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** Fit a linear regression using `lm(gdp_per_capita ~ continent, data = gapminder2022)` where `gdp_per_capita` is the new outcome variable. Get information about the "best-fitting" line from the regression coefficients. How do the regression results match up with the results from your previous exploratory data analysis?

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** How many "offsets" or differences from the baseline will a regression model output for a categorical variable with 4 levels?  

- A. 1
- B. 2
- C. 3
- D. 4

```{block, type="learncheck", purl=FALSE}
\vspace{-0.25in}
\vspace{-0.25in}
```


### Observed/fitted values and residuals {#model2points}

Recall in Subsection \@ref(model1points), we defined the following three concepts:

1. Observed values $y$, or the observed value of the outcome variable \index{regression!observed values}
1. Fitted values $\widehat{y}$, or the value on the regression line for a given $x$ value
1. Residuals $y - \widehat{y}$, or the error between the observed value and the fitted value

We obtained these values and other values using the `get_regression_points()` function from the `moderndive` package. This time, however, we add an argument setting `ID = "country"`, which uses the variable `country` in `gapminder2022` as an *identification variable* in the output. This will help contextualize our analysis by matching values to countries.

```{r regression-reg-points-alt, eval=FALSE}
get_regression_points(life_exp_model, ID = "country")
```
```{r model2-residuals, echo=FALSE, purl=FALSE}
regression_points <- get_regression_points(life_exp_model, ID = "country")
regression_points |>
  filter(country %in% c("Afghanistan", "Albania", "Algeria", "Angola", "Argentina", "Barbados")) |>
  kbl(
    digits = 3,
    caption = "Regression points (Sample of 6 out of 142 countries)",
    booktabs = TRUE,
    linesep = ""
  ) |>
  kable_styling(
    font_size = ifelse(is_latex_output(), 10, 16),
    latex_options = c("HOLD_position")
  )
```

```{r regression-create-afghanistan_life_e, echo=FALSE, purl=FALSE}
# This code is for inline dynamic coding
afghanistan_life_exp <- regression_points |>
  filter(country == "Afghanistan") |>
  pull(life_exp) |>
  round(1)
afghanistan_offset <- (afghanistan_life_exp - mean_asia) |> round(1)
```

Observe in Table \@ref(tab:model2-residuals) that `life_exp_hat` contains the fitted values $\widehat{y}$ = $\widehat{\text{life exp}}$. If you look closely, there are only 5 possible values for `life_exp_hat`. These correspond to the five mean life expectancies for the 5 continents that we displayed in Table \@ref(tab:continent-mean-life-expectancies) and computed using the regression coefficient values.

The `residual` column is simply $y - \widehat{y}$ = `life_exp - life_exp_hat`. These values can be interpreted as the deviation of a country's life expectancy from its continent's average life expectancy. For example, observe the first row of Table \@ref(tab:model2-residuals) corresponding to Afghanistan. The residual of $y - \widehat{y} = `r afghanistan_life_exp` - `r mean_asia` = `r afghanistan_offset`$ refers to Afghanistan's life expectancy being `r -1 * afghanistan_offset` years lower than the mean life expectancy of all Asian countries. This is partly explained by the years of war that country has suffered.

```{block, type="learncheck", purl=FALSE}
\vspace{-0.15in}
**_Learning check_**
\vspace{-0.1in}
```

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** Which interpretation is correct for a positive coefficient in a regression model with a categorical explanatory variable? 

  - A. It indicates the baseline group.
  - B. It represents the mean value of the baseline group.
  - C. The corresponding group has a higher response mean than the baseline's.
  - D. The corresponding group has a lower response mean than the baseline's.

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** Which of the following statements about residuals in regression is true?  
    
  - A. Residuals are the differences between the fitted and observed response values.
  - B. Residuals are always positive.
  - C. Residuals are not important for model evaluation.
  - D. Residuals are the predicted values in the model.

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** Using either the sorting functionality of RStudio's spreadsheet viewer or using the data wrangling tools you learned in Chapter \@ref(wrangling), identify the five countries with the five smallest (most negative) residuals? What do these negative residuals say about their life expectancy relative to their continents' life expectancy?

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** Repeat this process, but identify the five countries with the five largest (most positive) residuals. What do these positive residuals say about their life expectancy relative to their continents' life expectancy?

```{block, type="learncheck", purl=FALSE}
\vspace{-0.25in}
\vspace{-0.25in}
```





## Related topics {#reg-related-topics}

### Correlation is not necessarily causation {#correlation-is-not-causation}

Throughout this chapter we have been cautious when interpreting regression slope coefficients. We always discussed the "associated" effect of an explanatory variable $x$ on an outcome variable $y$. For example, our statement from Subsection \@ref(model1table) that "for every increase of 1 unit in `life_exp`, there is an *associated* decrease of on average `r abs(demo_line[2])` units of `fert_rate`." We include the term "associated" to be extra careful not to suggest we are making a *causal* statement. So while `life_exp` is negatively correlated with `fert_rate`, we can't necessarily make any statements about life expectancy's direct causal effect on fertility rates without more information. 

Here is another example depicted in Figure \@ref(fig:moderndive-figure-causal-graph-2): a not-so-great medical doctor goes through medical records and finds that patients who slept with their shoes on tended to wake up more with headaches. So this doctor declares, "Sleeping with shoes on causes headaches!"

```{r moderndive-figure-causal-graph-2, echo=FALSE, fig.cap="Does sleeping with shoes on cause headaches?", out.width="60%", purl=FALSE}
include_graphics("images/shutterstock/shoes_headache.png")
```

However, there is a good chance that if someone is sleeping with their shoes on, it is potentially because they are intoxicated from alcohol. Furthermore, higher levels of drinking leads to more hangovers, and hence more headaches. The amount of alcohol consumption here is what's known as a *confounding/lurking* variable\index{confounding variable}. It "lurks" behind the scenes, confounding the causal relationship (if any) of "sleeping with shoes on" with "waking up with a headache." We can summarize this in Figure \@ref(fig:moderndive-figure-causal-graph) with a *causal graph* where Y is a *response* variable and X is a *treatment* variable whose causal effect we are interested in. \index{variables!treatment}

```{r moderndive-figure-causal-graph, echo=FALSE, out.width="30%", fig.cap="Causal graph.", purl=FALSE}
include_graphics("images/flowcharts/flowchart.009-cropped.png")
```

To study the relationship between Y and X, we could use a regression model where the outcome variable is set to Y and the explanatory variable is set to be X, as you've been doing throughout this chapter. However, Figure \@ref(fig:moderndive-figure-causal-graph) also includes a third variable with arrows pointing at both X and Y:

* Z is a *confounding* variable \index{variables!confounding} that affects both X and Y, thereby "confounding" their relationship. Here the confounding variable is alcohol.

Alcohol will cause people to be both more likely to sleep with their shoes on as well as be more likely to wake up with a headache. Thus any regression model of the relationship between X and Y should also use Z as an explanatory variable. In other words, our doctor needs to take into account who had been drinking the night before. In the next chapter, we'll start covering multiple regression models that allow us to incorporate more than one variable in our regression models.

Establishing causation is a tricky problem and frequently takes either carefully designed experiments or methods to control for the effects of confounding variables. Both these approaches attempt, as best they can, either to take all possible confounding variables into account or negate their impact. This allows researchers to focus only on the relationship of interest: the relationship between the outcome variable Y and the treatment variable X.

As you read news stories, be careful not to fall into the trap of thinking that correlation necessarily implies causation.  Check out the [Spurious Correlations](http://www.tylervigen.com/spurious-correlations) website for some comical examples of variables that are correlated, but are not causally related.

Coming back to our UN member states data, a confounding variable could be the level of economic development of a country. This could affect both life expectancy and fertility rates. A proxy for looking at the level of economic development could be the Human Development Index (HDI). It measures a country's average achievements in three basic aspects of human development: health (life expectancy), education (mean and expected years of schooling), and standard of living (gross national income per capita). This is stored in the `hdi_2022` column of `un_member_states_2024`. We explore its relationship with life expectancy and fertility rates:

```{r regression-scatter-lifeexp, fig.height=1.5}
ggplot(data = un_member_states_2024, 
       aes(x = hdi_2022, y = life_expectancy_2022)) +
  geom_point() +
  labs(x = "Human Development Index (HDI)", y = "Life Expectancy")
```

```{r regression-scatter, fig.height=1.5}
ggplot(data = un_member_states_2024, 
       aes(x = hdi_2022, y = fertility_rate_2022)) +
  geom_point() +
  labs(x = "Human Development Index (HDI)", y = "Fertility Rate")
```

```{r regression-alt2-dup2}
un_member_states_2024 |> 
  get_correlation(life_expectancy_2022 ~ hdi_2022, na.rm = TRUE)
```

```{r regression-alt2-dup3}
un_member_states_2024 |> 
  get_correlation(fertility_rate_2022 ~ hdi_2022, na.rm = TRUE)
```

Looking at both of the scatterplots above, we see a strong positive linear relationship between the Human Development Index (HDI) and life expectancy, as well as a strong negative correlation between the Human Development Index (HDI) and fertility rates. The correlation coefficients between the Human Development Index (HDI) and life expectancy, as well as the Human Development Index (HDI) and fertility rates, are also given. These findings as well as some additional understanding of socioeconomic factors suggest that the Human Development Index (HDI) is a confounding variable in the relationship between life expectancy and fertility rates.     

### Best-fitting line {#leastsquares}

Regression lines are also known as "best-fitting" lines. But what do we mean by "best"? We unpack the criteria that is used in regression to determine "best." Recall Figure \@ref(fig:numxplot4), where for Bosnia and Herzegovina we marked the *observed value* $y$ with a circle, the *fitted value* $\widehat{y}$ with a square, and the *residual* $y - \widehat{y}$ with an arrow. We re-display Figure \@ref(fig:numxplot4) in the top-left plot of Figure \@ref(fig:best-fitting-line) in addition to three more arbitrarily chosen countries:

```{r best-fitting-line, fig.height=ifelse(knitr::is_latex_output(), 5.4, 7), echo=FALSE, fig.cap="Example of observed value, fitted value, and residual.", purl=FALSE, message=FALSE}
# First residual
demographics_model <- lm(fert_rate ~ life_exp, data = UN_data_ch5)
index <- with(UN_data_ch5, which(iso == "BIH"))
target_point_bosnia <- get_regression_points(demographics_model) |>
  slice(index)
x_bosnia <- target_point_bosnia$life_exp
y_bosnia <- target_point_bosnia$fert_rate
y_bosnia_hat <- target_point_bosnia$fert_rate_hat
resid_bosnia <- target_point_bosnia$residual

best_fit_plot <- ggplot(UN_data_ch5, aes(x = life_exp, y = fert_rate)) +
  geom_point(size = 0.8, color = "grey") +
  labs(x = "Life Expectancy", y = "Fertility Rate") +
  geom_smooth(method = "lm", se = FALSE) +
  annotate("point", x = x_bosnia, y = y_bosnia, col = "red", size = 2) +
  annotate("point", x = x_bosnia, y = y_bosnia_hat, col = "red", shape = 15, size = 2) +
  annotate("segment",
           x = x_bosnia, xend = x_bosnia, y = y_bosnia, yend = y_bosnia_hat, color = "blue",
           arrow = arrow(type = "closed", length = unit(0.02, "npc"))
  ) 
p1 <- best_fit_plot + labs(title = "Bosnia and Herzegovina's residual") +
  theme(plot.title = element_text(size=10))
#ggplotly(p1)

# Second residual
index <- with(UN_data_ch5, which(iso == "TCD"))
target_point <- get_regression_points(demographics_model) |>
  slice(index)
x_chad <- target_point$life_exp
y_chad <- target_point$fert_rate
y_chad_hat <- target_point$fert_rate_hat
resid_chad <- target_point$residual

best_fit_plot <- best_fit_plot +
  annotate("point", x = x_chad, y = y_chad, col = "red", size = 2) +
  annotate("point", x = x_chad, y = y_chad_hat, col = "red", shape = 15, size = 2) +
  annotate("segment",
           x = x_chad, xend = x_chad, y = y_chad, yend = y_chad_hat, color = "blue",
           arrow = arrow(type = "closed", length = unit(0.02, "npc"))
  )
p2 <- best_fit_plot + labs(title = "Chad's residual added") +
  theme(plot.title = element_text(size=10))

# Third residual
index <- with(UN_data_ch5, which(iso == "IND"))
target_point <- get_regression_points(demographics_model) |>
  slice(index)
x_india <- target_point$life_exp
y_india <- target_point$fert_rate
y_india_hat <- target_point$fert_rate_hat
resid_india <- target_point$residual

best_fit_plot <- best_fit_plot +
  annotate("point", x = x_india, y = y_india, col = "red", size = 2) +
  annotate("point", x = x_india, y = y_india_hat, col = "red", shape = 15, size = 2) +
  annotate("segment",
           x = x_india, xend = x_india, y = y_india, yend = y_india_hat,
           color = "blue",
           arrow = arrow(type = "closed", length = unit(0.02, "npc"))
  )
p3 <- best_fit_plot + labs(title = "India's residual added") +
  theme(plot.title = element_text(size=10))

index <- with(UN_data_ch5, which(iso == "SLB"))
demographics_model <- lm(fert_rate ~ life_exp, data = UN_data_ch5)
target_point <- get_regression_points(demographics_model) |>
  slice(index)
x_sol <- target_point$life_exp
y_sol <- target_point$fert_rate
y_sol_hat <- target_point$fert_rate_hat
resid_sol <- target_point$residual

best_fit_plot <- best_fit_plot +
  annotate("point", x = x_sol, y = y_sol, col = "red", size = 2) +
  annotate("point", x = x_sol, y = y_sol_hat, col = "red", shape = 15, size = 2) +
  annotate("segment",
           x = x_sol, xend = x_sol, y = y_sol, yend = y_sol_hat, color = "blue",
           arrow = arrow(type = "closed", length = unit(0.02, "npc"))
  )
p4 <- best_fit_plot + labs(title = "Solomon Islands' residual added") +
  theme(plot.title = element_text(size=10))

p1 + p2 + p3 + p4 + plot_layout(nrow = 2)
```

```{r regression-create-four_countries, include=FALSE}
four_countries <- c("BIH", "TCD", "IND", "SLB")
country_lookup_table <- UN_data_ch5 |>
  filter(iso %in% four_countries) |>
  select(iso, life_exp, fert_rate)
bosnia <- country_lookup_table |> 
  filter(iso == "BIH") |> 
  mutate(residual = resid_bosnia,
         fert_rate_hat = y_bosnia_hat)
chad <- country_lookup_table |>
  filter(iso == "TCD") |> 
  mutate(residual = resid_chad,
         fert_rate_hat = y_chad_hat)
india <- country_lookup_table |>
  filter(iso == "IND") |> 
  mutate(residual = resid_india,
         fert_rate_hat = y_india_hat)
solomon <- country_lookup_table |>
  filter(iso == "SLB") |> 
  mutate(residual = resid_sol,
         fert_rate_hat = y_sol_hat)
```

The four plots refer to:

1. The country of Bosnia and Herzegovina had a life expectancy $x = `r bosnia |> pull(life_exp)`$ and fertility rate $y = `r bosnia |> pull(fert_rate)`$. The residual in this case is $`r bosnia |> pull(fert_rate)` - `r bosnia |> pull(fert_rate_hat)` = `r bosnia |> pull(residual)`$, which we mark with an `r if_else(!is_latex_output(), "blue", "")` arrow in the top-left plot.
1. The addition of Chad, which had a life expectancy $x = `r chad |> pull(life_exp)`$ and fertility rate $y = `r chad |> pull(fert_rate)`$. The residual in this case is $`r chad |> pull(fert_rate)` - `r chad |> pull(fert_rate_hat)` = `r chad |> pull(residual)`$, which we mark with a new `r if_else(!is_latex_output(), "blue", "")` arrow in the top-right plot.
1. The addition of India, which had a life expectancy $x = `r india |> pull(life_exp)`$ and fertility rate $y = `r india |> pull(fert_rate)`$. The residual in this case is $`r india |> pull(fert_rate)` - `r india |> pull(fert_rate_hat)` = `r india |> pull(residual)`$, which we mark with a new `r if_else(!is_latex_output(), "blue", "")` arrow in the bottom-left plot.
1. The addition of the Solomon Islands, which had a life expectancy $x = `r solomon |> pull(life_exp)`$ and fertility rate $y = `r solomon |> pull(fert_rate)`$. The residual in this case is $`r solomon |> pull(fert_rate)` - `r solomon |> pull(fert_rate_hat)` = `r solomon |> pull(residual)`$, which we mark with a new `r if_else(!is_latex_output(), "blue", "")` arrow in the bottom-right plot.

Now say we repeated this process of computing residuals for all `r n_demo_ch5` countries with complete information, then we squared all the residuals, and then we summed them. We call this quantity the *sum of squared residuals*\index{sum of squared residuals}; it is a measure of the _lack of fit_ of a model. Larger values of the sum of squared residuals indicate a bigger lack of fit. This corresponds to a worse-fitting model.

If the regression line fits all the points perfectly, then the sum of squared residuals is 0. This is because if the regression line fits all the points perfectly, then the fitted value $\widehat{y}$ equals the observed value $y$ in all cases, and hence the residual $y - \widehat{y}$ = 0 in all cases, and the sum of even a large number of 0's is still 0. 

Furthermore, of all possible lines we can draw through the cloud of `r n_demo_ch5` points, the regression line minimizes this value. In other words, the regression and its corresponding fitted values $\widehat{y}$ minimizes the sum of the squared residuals:

$$
\sum_{i=1}^{n}(y_i - \widehat{y}_i)^2
$$

We use our data-wrangling tools to compute the sum of squared residuals exactly:

```{r regression-create-demographics_model, eval=FALSE}
# Fit regression model and regression points
demographics_model <- lm(fert_rate ~ life_exp, data = UN_data_ch5)
regression_points <- get_regression_points(demographics_model)

# Compute sum of squared residuals
regression_points |>
  mutate(squared_residuals = residual^2) |>
  summarize(sum_of_squared_residuals = sum(squared_residuals))
```

```{r regression-create-demographics_model-alt, echo=FALSE}
# Fit regression model:
demographics_model <- lm(fert_rate ~ life_exp, data = UN_data_ch5)

# Get regression points:
regression_points <- get_regression_points(demographics_model)

# Compute sum of squared residuals
SSR <- regression_points |>
  mutate(squared_residuals = residual^2) |>
  summarize(sum_of_squared_residuals = sum(squared_residuals)) |> 
  pull()
SSR
```

Any other straight line drawn in the figure would yield a sum of squared residuals greater than `r SSR`. This is a mathematically guaranteed fact that you can prove using calculus and linear algebra. That's why alternative names for the linear regression line are the *best-fitting line* and the *least-squares line*. 

Why do we square the residuals (i.e., the arrow lengths)? So that both positive and negative deviations of the same amount are treated equally. (That being said, while taking the absolute value of the residuals would also treat both positive and negative deviations of the same amount equally, squaring the residuals is used for reasons related to calculus: taking derivatives and minimizing functions. To learn more we suggest you consult a textbook on mathematical statistics.)


```{block, type="learncheck", purl=FALSE}
\vspace{-0.15in}
**_Learning check_**
\vspace{-0.1in}
```

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** Note in Figure \@ref(fig:three-lines) there are 3 points marked with dots and:

* The "best" fitting solid regression line `r if_else(is_latex_output(), "", "in blue")`
* An arbitrarily chosen dotted `r if_else(is_latex_output(), "", "red")` line 
* Another arbitrarily chosen dashed `r if_else(is_latex_output(), "", "green")` line

```{r three-lines, fig.cap="Regression line and two others.", out.width="60%", echo=FALSE, purl=FALSE, message=FALSE}
example <- tibble(
  x = c(0, 0.5, 1),
  y = c(2, 1, 3)
)

ggplot(example, aes(x = x, y = y)) +
  geom_smooth(method = "lm", se = FALSE, fullrange = TRUE) +
  geom_hline(yintercept = 2.5, col = "red", linetype = "dotted", size = 1) +
  geom_abline(
    intercept = 2, slope = -1, col = "forestgreen",
    linetype = "dashed", linewidth = 1
  ) +
  geom_point(size = 4)
```

Compute the sum of squared residuals by hand for each line. Show that the regression line `r if_else(is_latex_output(), "", "in blue")` has the smallest value of these three lines.

```{block, type="learncheck", purl=FALSE}
\vspace{-0.25in}
\vspace{-0.25in}
```


### get_regression_x() functions {#underthehood}

Recall in this chapter we introduced a wrapper function from the `moderndive` package:

- `get_regression_points()` that returns point-by-point information from a regression model in Subsection \@ref(model1points).

What is going on behind the scenes with the <!-- `get_regression_table()` and --> `get_regression_points()` function? We mentioned in Subsection \@ref(model1table) that this was an example of a *wrapper function*. Such functions take other pre-existing functions and "wrap" them into single functions that hide the user from their inner workings. This way all the user needs to worry about is what the inputs look like and what the outputs look like. In this subsection, we'll "get under the hood" of these functions and see how the "engine" of these wrapper functions works.

The `get_regression_points()` function is a wrapper function, returning information about the individual points involved in a regression model like the fitted values, observed values, and the residuals. `get_regression_points()` \index{R packages!moderndive!get\_regression\_points()} uses the `augment()` \index{R packages!broom!augment()} function in the [`broom` package](https://broom.tidyverse.org/) <!-- instead of the `tidy()` function as with `get_regression_table()` --> to produce the data shown in Table \@ref(tab:regpoints-augment). Additionally, it uses `clean_names()` \index{R packages!janitor!clean\_names()} from the [`janitor` package](https://github.com/sfirke/janitor) [@R-janitor] to clean up the variable names.


```{r regression-load-packages-alt, eval=FALSE}
library(broom)
library(janitor)
demographics_model |>
  augment() |>
  mutate_if(is.numeric, round, digits = 3) |>
  clean_names() |>
  select(-c("std_resid", "hat", "sigma", "cooksd", "std_resid"))
```
```{r regpoints-augment, echo=FALSE, purl=FALSE, message=FALSE, warning=FALSE}
library(broom)
library(janitor)
demographics_model |>
  augment() |>
  mutate_if(is.numeric, round, digits = 3) |>
  clean_names() |>
  select(-c("std_resid", "hat", "sigma", "cooksd", "std_resid")) |>
  slice(1:10) |>
  kbl(
    digits = 3,
    caption = "Regression points using augment() from broom package",
    booktabs = TRUE,
    linesep = ""
  ) |>
  kable_styling(
    font_size = ifelse(is_latex_output(), 10, 16),
    latex_options = c("HOLD_position")
  )
```

In this case, it outputs only the variables of interest to students learning regression: the outcome variable $y$ (`fert_rate`), all explanatory/predictor variables (`life_exp`), all resulting `fitted` values $\hat{y}$ used by applying the equation of the regression line to `life_exp`, and the `resid`ual $y - \hat{y}$.

If you are even more curious about how these and other wrapper functions work, take a look at the source code for these functions on [GitHub](https://github.com/moderndive/moderndive/blob/master/R/regression_functions.R).





## Conclusion {#reg-conclusion}

### Additional resources {#additional-resources-basic-regression}

```{r regression-conditional-text-dup1, echo=FALSE, results="asis", purl=FALSE}
if (is_latex_output()) {
  cat("Solutions to all *Learning checks* can be found in the Appendices of the online version of the book. The Appendices start at <https://moderndive.com/v2/appendixa>.")
}
```

```{r regression-conditional-text-v2, echo=FALSE, purl=FALSE, results="asis"}
# For generating the R script files at the end of relevant chapters
generate_r_file_link <- function(file) {
  if (is_latex_output()) {
    cat(glue::glue("An R script file of all R code used in this chapter is available at <https://www.moderndive.com/v2/scripts/{file}>."))
  } else {
    cat(glue::glue("An R script file of all R code used in this chapter is available [here](scripts/{file})."))
  }
}
generate_r_file_link("05-regression.R")
```

As we suggested in Subsection \@ref(model1EDA), interpreting coefficients that are not close to the extreme values of -1, 0, and 1 can be somewhat subjective. To help develop your sense of correlation coefficients, we suggest you play the 80s-style video game called, "Guess the Correlation," at <http://guessthecorrelation.com/> previewed in Figure \@ref(fig:guess-the-correlation).

(ref:guess-corr) Preview of "Guess the Correlation" game.

```{r guess-the-correlation, echo=FALSE, fig.cap="(ref:guess-corr)", purl=FALSE, out.width="70%", purl=FALSE}
include_graphics("images/copyright/guess_the_correlation.png")
```


### What's to come?

In this chapter, you've studied the term _simple linear regression_, where you fit models that only have one explanatory variable. In Chapter \@ref(multiple-regression), we'll study *multiple regression*, where our regression models can now have more than one explanatory variable moving a little bit more advanced than the basic form of simple linear regression! In particular, we'll consider two scenarios: regression models with one numerical and one categorical explanatory variable and regression models with two numerical explanatory variables. This will allow you to construct more sophisticated and more powerful models, all in the hopes of better explaining your outcome variable $y$.