Chapter 2 Simple Linear Regression

2.1 Getting started

Putting text here

2.2 Foundation

Putting text here

2.3 Inference

Putting text here

2.4 Prediction

2.5 Checking conditions

2.6 Partioning variability

2.7 Derivation for slope and intercept

This document contains the mathematical details for deriving the least-squares estimates for slope (${\beta }_1$) and intercept (${\beta }_0$). We obtain the estimates, ${\hat{\beta }}_1$ and ${\hat{\beta }}_0$ by finding the values that minimize the sum of squared residuals ().

$$SSR=\sum _{i=1}^n[y_i-{\hat{y}}_i{]}^2=[y_i-({\hat{\beta }}_0+{\hat{\beta }}_1{x}_i){]}^2=[y_i-({\hat{\beta }}_0-{\hat{\beta }}_1{x}_i{]}^2$$

Recall that we can find the values of ${\hat{\beta }}_1$ and ${\hat{\beta }}_0$ that minimize () by taking the partial derivatives of () and setting them to 0. Thus, the values of ${\hat{\beta }}_1$ and ${\hat{\beta }}_0$ that minimize the respective partial derivative also minimize the sum of squared residuals. The partial derivatives are

$$\begin{array}{cc}& \frac{\partial \text{SSR}}{\partial {\hat{\beta }}_1}=-2\sum _{i=1}^n{x}_i(y_i-{\hat{\beta }}_0-{\hat{\beta }}_1{x}_i)\\ & \frac{\partial \text{SSR}}{\partial {\hat{\beta }}_0}=-2\sum _{i=1}^n(y_i-{\hat{\beta }}_0-{\hat{\beta }}_1{x}_i)\end{array}$$

Let’s begin by deriving ${\hat{\beta }}_0$.

$$\begin{array}{cc}\frac{\partial \text{SSR}}{\partial {\hat{\beta }}_0}& =-2\sum _{i=1}^n(y_i-{\hat{\beta }}_0-{\hat{\beta }}_1{x}_i)=0\\ & \Rightarrow -\sum _{i=1}^n(y_i+{\hat{\beta }}_0+{\hat{\beta }}_1{x}_i)=0\\ & \Rightarrow -\sum _{i=1}^n{y}_i+n{\hat{\beta }}_0+{\hat{\beta }}_1\sum _{i=1}^n{x}_i=0\\ & \Rightarrow n{\hat{\beta }}_0=\sum _{i=1}^n{y}_i-{\hat{\beta }}_1\sum _{i=1}^n{x}_i\\ & \Rightarrow {\hat{\beta }}_0=\frac{1}{n}(\sum _{i=1}^n{y}_i-{\hat{\beta }}_1\sum _{i=1}^n{x}_i)\\ & \Rightarrow {\hat{\beta }}_0=\overline{y}-{\hat{\beta }}_1\overline{x}\end{array}$$

Now, we can derive ${\hat{\beta }}_1$ using the ${\hat{\beta }}_0$ we just derived

$$\begin{array}{cc}& \frac{\partial \text{SSR}}{\partial {\hat{\beta }}_1}=-2\sum _{i=1}^n{x}_i(y_i-{\hat{\beta }}_0-{\hat{\beta }}_1{x}_i)=0\\ & \Rightarrow -\sum _{i=1}^n{x}_i{y}_i+{\hat{\beta }}_0\sum _{i=1}^n{x}_i+{\hat{\beta }}_1\sum _{i=1}^n{x}_i^2=0\\ \text{(Fill in}{\hat{\beta }}_0\text{)}& \Rightarrow -\sum _{i=1}^n{x}_i{y}_i+(\overline{y}-{\hat{\beta }}_1\overline{x})\sum _{i=1}^n{x}_i+{\hat{\beta }}_1\sum _{i=1}^n{x}_i^2=0\\ & \Rightarrow (\overline{y}-{\hat{\beta }}_1\overline{x})\sum _{i=1}^n{x}_i+{\hat{\beta }}_1\sum _{i=1}^n{x}_i^2=\sum _{i=1}^n{x}_i{y}_i\\ & \Rightarrow \overline{y}\sum _{i=1}^n{x}_i-{\hat{\beta }}_1\overline{x}\sum _{i=1}^n{x}_i+{\hat{\beta }}_1\sum _{i=1}^n{x}_i^2=\sum _{i=1}^n{x}_i{y}_i\\ & \Rightarrow n\overline{y}\overline{x}-{\hat{\beta }}_{1}n{\overline{x}}^2+{\hat{\beta }}_1\sum _{i=1}^n{x}_i^2=\sum _{i=1}^n{x}_i{y}_i\\ & \Rightarrow {\hat{\beta }}_1\sum _{i=1}^n{x}_i^2-{\hat{\beta }}_{1}n{\overline{x}}^2=\sum _{i=1}^n{x}_i{y}_i-n\overline{y}\overline{x}\\ & \Rightarrow {\hat{\beta }}_1(\sum _{i=1}^n{x}_i^2-n{\overline{x}}^2)=\sum _{i=1}^n{x}_i{y}_i-n\overline{y}\overline{x}\\ & {\hat{\beta }}_1=\frac{\sum _{i=1}^n{x}_i{y}_i-n\overline{y}\overline{x}}{\sum _{i=1}^n{x}_i^2-n{\overline{x}}^2}\end{array}$$

To write ${\hat{\beta }}_1$ in a form that’s more recognizable, we will use the following:

$$\sum x_i{y}_i-n\overline{y}\overline{x}=\sum (x-\overline{x})(y-\overline{y})=(n-1)\text{Cov}(x,y)$$

$$\sum x_i^2-n{\overline{x}}^2-\sum (x-\overline{x}{)}^2=(n-1)s_x^2$$

where $\text{Cov}(x,y)$ is the covariance of $x$ and $y$, and $s_x^2$ is the sample variance of $x$ ($s_x$ is the sample standard deviation).

Thus, applying () and (), we have

$$\begin{array}{cc}{\hat{\beta }}_1& =\frac{\sum _{i=1}^n{x}_i{y}_i-n\overline{y}\overline{x}}{\sum _{i=1}^n{x}_i^2-n{\overline{x}}^2}\\ & =\frac{\sum _{i=1}^n(x-\overline{x})(y-\overline{y})}{\sum _{i=1}^n(x-\overline{x}{)}^2}\\ & =\frac{(n-1)\text{Cov}(x,y)}{(n-1)s_x^2}\\ & =\frac{\text{Cov}(x,y)}{s_x^2}\end{array}$$

The correlation between $x$ and $y$ is $r=\frac{\text{Cov}(x,y)}{s_x{s}_y}$. Thus, $\text{Cov}(x,y)=r{s}_x{s}_y$. Plugging this into (), we have

$${\hat{\beta }}_1=\frac{\text{Cov}(x,y)}{s_x^2}=r\frac{s_y{s}_x}{s_x^2}=r\frac{s_y}{s_x}$$