10-5-2025 Regression

#stat #stat380 #math

Regression

Regression is the process of finding a model that best fits a set of data.
For example

Prediction

Predict the most accurate answer such as the price of the car

Inference

Understand the association between variables

Simple linear regression

A model for predicting a quantitative response (Y) based on a single predictor variable (X).

R^2

The proportion of total variation in the response (y) explained by the least-squares regression line. For example How much of the up and down of a Camry can be explained by just it’s mileage.

Dummy variables

A technique to convert categorial data into numerical indicators (0s and 1s) for a model.

Model	Is_camry	Is_tacoma
Camry	1	0
Tacoma	0	1
Volt	0	0
Using the combination of 0’s and 1’s we can mathematically represent categories for a regression model

Multiple linear regression

This is when we have more than one predictor variable (X1, X2, …, Xp) to predict a quantitative response (Y).

Clustering

This is an unsupervised learning technique that involved finding subgroups in a dataset where there is no supervising output, which matches the problem description

$\hat{Y}$

$\hat{Y}$ (or $\hat{y}$ ) is the predicted or estimated value of the response. For Simple Linear Regression (SLR), the estimated regression equation is ${\hat{y}}_{i} = a + b x_{i}$ .
Prediction problems fall under the category of Supervised Learning.

Steps to get MSE

M S E = \frac{1}{n} \sum_{i = 1}^{n} {(y_{i} - {\hat{y}}_{i})}^{2}

The necessary order of operations for calculation is:

Obtain predictions ( $\hat{y}$ ).
Find the residuals ( $y - \hat{y}$ ).
Square the residuals.
Sum the squared residuals (this is the Residual Sum of Squares, RSS).
Divide the sum by $n$ (or multiply by $1 / n$ ).

(Note: Steps 4 and 5 can be combined by finding the mean of the squared residuals).

Practice

Since the notation $\hat{Y} = \hat{f} (X)$ is the core of prediction, let's focus on using an existing model to generate predictions ( ${\hat{y}}_{i}$ ) and then evaluate those predictions using the Mean Squared Error (MSE).

We will use the Simple Linear Regression (SLR) example from your sources for Camry vehicles.

R Exercise: Predicting Price and Calculating MSE

We will use the estimated regression equation for Camry's price based on mileage: $$ \hat{y} = 21.312 - 0.133x_{\text{miles}} $$

The goal of this exercise is to calculate the MSE, which is defined as: $$ MSE = \frac{1}{n} \sum_{i=1}^{n}\left(y_{i}-\hat{y}_{i}\right)^{2} $$

Your Task: Imagine you have a new set of 4 Camry vehicles. Use the equation above to find the predicted price ( ${\hat{y}}_{i}$ ) for each car, and then calculate the MSE for this small dataset.


Car	Miles ( $x_{i}$ ) (in thousands)	Observed Price ( $y_{i}$ ) (in thousands)	Predicted Price ( ${\hat{y}}_{i}$ )	Residual ( $y_{i} - {\hat{y}}_{i}$ )	Squared Residual
1	50	14.5	?	?	?
2	100	6.1	?	?	?
3	75	11.0	?	?	?
4	20	18.9	?	?	?

Step 1: Calculate the Predicted Price ( ${\hat{y}}_{i}$ ) for Car 1.
Predictions: $\hat{y_{1}} = 21.312 - 0.133 (50)_{miles} = 14.67$
Residuals: $y_{1} - \hat{y_{1}} = 14.5 - 14.67 = - 0.17$ .
Square the residuals $R^{2}$ : $(- 0.17)^{2} = 0.0289$

Step 2: Calculate the Predicted Price ( ${\hat{y}}_{i}$ ) for Car 2.
Predictions: $\hat{y_{1}} = 21.312 - 0.133 (100)_{miles} = 8.02$
Residuals: $y_{1} - \hat{y_{1}} = 14.5 - 8.02 = 6.48$ .
Square the residuals $R^{2}$ : $(6.84)^{2} = 41.9904$

Step 3: Calculate the Predicted Price ( ${\hat{y}}_{i}$ ) for Car 3.
Predictions: $\hat{y_{1}} = 21.312 - 0.133 (75)_{miles} = 11.345$
Residuals: $y_{1} - \hat{y_{1}} = 14.5 - 11.345 = 3.155$ .
Square the residuals $R^{2}$ : $(6.84)^{2} = 9.954025$

Step 4: Calculate the Predicted Price ( ${\hat{y}}_{i}$ ) for Car 4.
Predictions: $\hat{y_{1}} = 21.312 - 0.133 (20)_{miles} = 18.66$
Residuals: $y_{1} - \hat{y_{1}} = 14.5 - 18.66 = - 4.16$ .
Square the residuals $R^{2}$ : $(6.84)^{2} = 17.3056$

Step 5: Calculate the MSE

M S E = \frac{0.0289 + 41.9904 + 9.954025 + 17.3056}{4} = 17.31975625