Linear Regression Algorithm: ML Algorithm Complete Guide

Introduction and Core Concepts

Linear regression is one of the most fundamental and widely used supervised learning algorithms in machine learning and statistics. At its core, the linear regression algorithm attempts to model the relationship between a dependent variable (target) and one or more independent variables (features) by fitting a linear equation to observed data. Despite its simplicity, it serves as the cornerstone for understanding more complex algorithms and remains highly effective for a vast range of predictive tasks where relationships are approximately linear.

The central assumption of the linear regression algorithm is that the target variable can be expressed as a weighted linear combination of the input features plus an error term. In its simplest form, simple linear regression models the relationship between a single feature x and target y as y = b0 + b1 * x + ε, where b0 is the intercept, b1 is the slope or coefficient, and ε represents irreducible error or noise. For multiple features, this extends to y = b0 + b1*x1 + b2*x2 + ... + bn*xn + ε. The goal of the algorithm is to learn the optimal values for the coefficients (b0 ... bn) that minimize the difference between predicted and actual values.

Why does the linear regression algorithm still matter in an era of deep learning and ensemble methods? First, it offers unparalleled interpretability. Each coefficient directly quantifies the expected change in the target for a one-unit change in the corresponding feature, holding other features constant. This makes it invaluable for inference, policy decisions, and regulated industries like finance and healthcare. Second, it is computationally efficient, fast to train, and requires relatively little data to produce stable estimates. Third, it provides a strong baseline; if a complex model cannot significantly outperform linear regression, the added complexity may not be justified. Understanding its assumptions, strengths, and limitations is essential for any machine learning practitioner.

Key Terminology

To master the linear regression algorithm, you must be familiar with its core vocabulary. The hypothesis function represents the model's prediction, while the cost function, most commonly Mean Squared Error (MSE), measures how well the model fits the data. The process of finding the best coefficients is called optimization, typically performed via Ordinary Least Squares (OLS), Gradient Descent, or the Normal Equation. Residuals are the differences between observed and predicted values, and analyzing their distribution is key to validating model assumptions.

Linear regression is categorized as a parametric, discriminative model for regression tasks. It is parametric because it assumes a fixed form with a finite number of parameters (the coefficients). It is also considered a white-box model because its internal logic is fully transparent. This contrasts with non-parametric models like k-Nearest Neighbors or black-box models like deep neural networks, where the mapping from inputs to outputs is far less interpretable.

How It Works

The mechanism of the linear regression algorithm revolves around finding the line (or hyperplane in higher dimensions) that best fits the training data according to a defined loss function. The most common loss function is the Sum of Squared Errors (SSE) or its mean, MSE. The algorithm seeks to minimize J(b) = (1/2m) * Σ(y_pred(i) - y_true(i))^2, where m is the number of training examples. Squaring the errors penalizes larger deviations more heavily and makes the function differentiable, enabling efficient optimization.

There are two primary analytical approaches to find the optimal coefficients. The first is the Normal Equation, a closed-form solution derived from calculus: θ = (X^T * X)^(-1) * X^T * y, where X is the feature matrix (with an added column of ones for the intercept), y is the target vector, and θ is the vector of coefficients. This method directly computes the global minimum without iteration and is highly effective for small to medium datasets. However, it requires computing the inverse of X^T * X, which can be computationally expensive and numerically unstable if features are highly correlated.

The second and more scalable approach is Gradient Descent, an iterative optimization algorithm. It starts with random initial coefficients and repeatedly updates them in the direction opposite to the gradient of the cost function. The update rule for each coefficient bj is: bj := bj - α * (∂J/∂bj), where α is the learning rate. For linear regression with MSE, the gradient is (1/m) * Σ((y_pred - y_true) * xj). By choosing an appropriate learning rate and iterating until convergence, gradient descent will converge to the global minimum because the MSE cost function for linear regression is convex, meaning it has only one global minimum and no local minima to get trapped in.

Core Assumptions

The statistical validity and predictive power of the linear regression algorithm depend on four key assumptions. Violating these assumptions does not necessarily prevent the model from making predictions, but it can make those predictions unreliable and the coefficient interpretations invalid. First is Linearity: the relationship between features and target is linear. Second is Independence: residuals are independent of each other, with no autocorrelation, which is especially critical for time-series data.

Third is Homoscedasticity: residuals have constant variance at every level of the independent variables. If variance increases with the predicted value (a funnel shape in a residual plot), the model suffers from heteroscedasticity. Fourth is Normality: for inference purposes, residuals should be approximately normally distributed. Additionally, there should be little or no multicollinearity, where independent variables are highly correlated with each other, as this inflates variance of coefficient estimates and makes them unstable.

Cost Function and Optimization Landscape

The choice of MSE as the cost function is not arbitrary. Under the assumption that errors are normally distributed, minimizing MSE is equivalent to maximizing the likelihood of the data (Maximum Likelihood Estimation). The convex, bowl-shaped nature of the MSE surface for linear regression guarantees that any local minimum is the global minimum. This is a significant advantage over non-convex models like neural networks, where optimization is far more challenging. Regularization terms can be added to this cost function to handle overfitting and multicollinearity, leading to Ridge and Lasso variants.

Implementation

Implementing the linear regression algorithm from scratch provides deep insight into its inner workings, while using established libraries like scikit-learn offers production-ready efficiency and robustness. Below are four distinct implementations covering the core approaches: gradient descent from scratch, the normal equation, a full scikit-learn pipeline with preprocessing, and regularized variants.

The first example demonstrates batch gradient descent for simple linear regression using only NumPy. This implementation is educational and shows every step: hypothesis computation, gradient calculation, and parameter updates. It is ideal for understanding the iterative learning process.

import numpy as np
import matplotlib.pyplot as plt

# Generate synthetic data: y = 3 + 2*x + noise
np.random.seed(42)
X = 2 * np.random.rand(100, 1)
y = 3 + 2 * X + np.random.randn(100, 1) * 0.5

# Add bias term (x0 = 1) to feature matrix
X_b = np.c_[np.ones((100, 1)), X]  # shape (100, 2)

# Hyperparameters
learning_rate = 0.1
n_iterations = 1000
m = len(X_b)

# Initialize parameters randomly
theta = np.random.randn(2, 1)  # [b0, b1]

# Batch Gradient Descent
for iteration in range(n_iterations):
    # Compute predictions: y_pred = X_b dot theta
    y_pred = X_b.dot(theta)
    # Compute error
    error = y_pred - y
    # Compute gradients: (1/m) * X_b.T dot error
    gradients = (1/m) * X_b.T.dot(error)
    # Update parameters: theta = theta - alpha * gradients
    theta = theta - learning_rate * gradients
    # Optional: compute MSE for monitoring
    if iteration % 200 == 0:
        mse = np.mean(error**2)
        print(f"Iteration {iteration}, MSE: {mse:.4f}, Theta: {theta.ravel()}")

print(f"Final learned parameters: Intercept (b0) = {theta[0][0]:.3f}, Slope (b1) = {theta[1][0]:.3f}")
# Expected close to b0=3, b1=2

The second example implements the closed-form Normal Equation. This approach requires no learning rate or iterations and finds the optimal solution in one step via matrix operations. It is mathematically elegant but computationally intensive for large feature spaces due to matrix inversion.

import numpy as np

# Using same X_b and y from previous example
# Normal Equation: theta_best = (X^T * X)^(-1) * X^T * y

# Compute theta using normal equation
# Use np.linalg.inv for inverse, or np.linalg.pinv for pseudo-inverse (more stable)
XT_X = X_b.T.dot(X_b)
XT_y = X_b.T.dot(y)

# Check if matrix is invertible (determinant != 0)
# For numerical stability, use pseudo-inverse which handles singular matrices
theta_best = np.linalg.inv(XT_X).dot(XT_y)
# More robust alternative:
# theta_best = np.linalg.pinv(X_b).dot(y)

print(f"Normal Equation Solution: {theta_best.ravel()}")
# Predict for new data point x=1.5
X_new = np.array([[1, 1.5]])  # [bias, feature]
y_predict = X_new.dot(theta_best)
print(f"Prediction for x=1.5: {y_predict[0][0]:.3f}")

# Compare condition number to detect multicollinearity
cond_number = np.linalg.cond(XT_X)
print(f"Condition number of X^T*X: {cond_number:.2f} (high value > 1000 indicates multicollinearity)")

The third example shows the industry-standard approach using scikit-learn. This pipeline includes proper train-test splitting, feature scaling, and model evaluation. Scikit-learn's LinearRegression uses an efficient solver (typically SVD) that is more numerically stable than a naive normal equation implementation.

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error

# Multiple Linear Regression Example: House price prediction
# Features: [size_sqft, bedrooms, age_years]
np.random.seed(0)
n_samples = 500
size = np.random.normal(1500, 400, n_samples)
bedrooms = np.random.randint(1, 6, n_samples)
age = np.random.randint(0, 40, n_samples)
# True model: price = 50000 + 120*size + 10000*bedrooms - 800*age + noise
price = 50000 + 120*size + 10000*bedrooms - 800*age + np.random.randn(n_samples)*15000

X_multi = np.column_stack((size, bedrooms, age))
y_multi = price

# Split data
X_train, X_test, y_train, y_test = train_test_split(X_multi, y_multi, test_size=0.2, random_state=42)

# Feature scaling is not strictly required for Normal Equation but helps Gradient Descent and interpretation
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Train model
model = LinearRegression()
model.fit(X_train_scaled, y_train)

print(f"Coefficients: {model.coef_}")
print(f"Intercept: {model.intercept_:.2f}")
print(f"Feature names importance: size={model.coef_[0]:.2f}, bedrooms={model.coef_[1]:.2f}, age={model.coef_[2]:.2f}")

# Evaluate
y_pred = model.predict(X_test_scaled)
print(f"MSE: {mean_squared_error(y_test, y_pred):.2f}")
print(f"RMSE: {np.sqrt(mean_squared_error(y_test, y_pred)):.2f}")
print(f"MAE: {mean_absolute_error(y_test, y_pred):.2f}")
print(f"R^2 Score: {r2_score(y_test, y_pred):.4f}")

# Check if prediction improves over baseline (mean predictor)
baseline_mse = mean_squared_error(y_test, [np.mean(y_train)] * len(y_test))
print(f"Baseline MSE (predict mean): {baseline_mse:.2f}")

The fourth example addresses overfitting and multicollinearity through regularization. Ridge (L2) and Lasso (L1) regression add a penalty term to the cost function, shrinking coefficients and performing feature selection. This is crucial when dealing with high-dimensional data or correlated features.

from sklearn.linear_model import Ridge, Lasso, ElasticNet
from sklearn.model_selection import cross_val_score

# Simulate high-dimensional, correlated data
np.random.seed(1)
X_corr = np.random.randn(100, 10)
# Make features 2 and 3 highly correlated with feature 0
X_corr[:, 2] = X_corr[:, 0] * 0.9 + np.random.randn(100) * 0.1
X_corr[:, 3] = X_corr[:, 0] * 0.85 + np.random.randn(100) * 0.1
y_corr = 3*X_corr[:,0] + 2*X_corr[:,1] + np.random.randn(100)*0.5

# Ridge Regression (L2 penalty): shrinks coefficients, handles multicollinearity
# Cost = MSE + alpha * sum(theta_j^2)
ridge = Ridge(alpha=1.0)  # alpha is regularization strength
ridge.fit(X_corr, y_corr)
print(f"Ridge coefficients (alpha=1.0): {ridge.coef_.round(2)}")
print(f"Ridge R^2 (CV 5-fold): {cross_val_score(ridge, X_corr, y_corr, cv=5).mean():.4f}")

# Lasso Regression (L1 penalty): shrinks and can zero out coefficients (feature selection)
# Cost = MSE + alpha * sum(|theta_j|)
lasso = Lasso(alpha=0.1)
lasso.fit(X_corr, y_corr)
print(f"Lasso coefficients (alpha=0.1): {lasso.coef_.round(2)}")
print(f"Lasso selected {np.sum(lasso.coef_ != 0)} out of 10 features")

# ElasticNet combines L1 and L2
elastic = ElasticNet(alpha=0.1, l1_ratio=0.5)  # l1_ratio=0.5 balances L1 and L2
elastic.fit(X_corr, y_corr)
print(f"ElasticNet coefficients: {elastic.coef_.round(2)}")

# Finding best alpha via cross-validation
alphas = [0.01, 0.1, 1.0, 10.0, 100.0]
for a in alphas:
    r = Ridge(alpha=a)
    score = cross_val_score(r, X_corr, y_corr, cv=5, scoring='neg_mean_squared_error').mean()
    print(f"Alpha {a}: Avg Neg MSE = {score:.3f}")

Step-by-Step Example

Let us walk through a concrete, manual example of the linear regression algorithm using a tiny dataset to make every calculation transparent. Suppose we want to predict a student's exam score (y) based on hours studied (x). We have five observations: (1, 2), (2, 4), (3, 5), (4, 4), (5, 6). Our goal is to find the best-fit line y = b0 + b1*x using Ordinary Least Squares.

Step 1: Compute the means. The mean of x, denoted x_bar, is (1+2+3+4+5)/5 = 3.0. The mean of y, y_bar, is (2+4+5+4+6)/5 = 4.2. Step 2: Compute the slope b1. The formula for the slope is b1 = Σ((xi - x_bar)*(yi - y_bar)) / Σ((xi - x_bar)^2). We calculate the numerator: (1-3)*(2-4.2)=(-2)*(-2.2)=4.4; (2-3)*(4-4.2)=(-1)*(-0.2)=0.2; (3-3)*(5-4.2)=0*0.8=0; (4-3)*(4-4.2)=1*(-0.2)=-0.2; (5-3)*(6-4.2)=2*1.8=3.6. Sum = 4.4+0.2+0-0.2+3.6 = 8.0. The denominator: (-2)^2=4; (-1)^2=1; 0^2=0; 1^2=1; 2^2=4. Sum = 10. Therefore, b1 = 8.0 / 10 = 0.8.

Step 3: Compute the intercept b0. The formula is b0 = y_bar - b1 * x_bar = 4.2 - 0.8*3.0 = 4.2 - 2.4 = 1.8. Our fitted line is y = 1.8 + 0.8*x. Step 4: Make predictions and compute residuals. For x=1, y_pred=2.6, residual=-0.6; x=2, y_pred=3.4, residual=0.6; x=3, y_pred=4.2, residual=0.8; x=4, y_pred=5.0, residual=-1.0; x=5, y_pred=5.8, residual=0.2. Notice the sum of residuals is approximately zero, a property of OLS.

Step 5: Evaluate the model. Compute SSE = Σ(residual^2) = 0.36+0.36+0.64+1.0+0.04 = 2.4. Compute Total Sum of Squares SST = Σ((yi - y_bar)^2) = (2-4.2)^2=4.84 +0.04+0.64+0.04+3.24 = 8.8. Then R-squared = 1 - SSE/SST = 1 - 2.4/8.8 = 0.727. This means 72.7% of the variance in exam scores is explained by hours studied. The Mean Squared Error is SSE/m = 2.4/5 = 0.48. For a new student who studies 6 hours, we predict y = 1.8 + 0.8*6 = 6.6. This manual walkthrough reveals exactly what the algorithm optimizes and how each statistic is derived.

Visualizing the Fit

If we plotted these five points, we would see the line y = 1.8 + 0.8*x passing through the center of the data cloud, minimizing the sum of squared vertical distances from each point to the line. The residuals are those vertical distances. Gradient descent would arrive at the same solution iteratively: starting from, say, b0=0, b1=0, it would compute gradients and take small steps downhill on the MSE surface until it converges to b0=1.8, b1=0.8. The normal equation would solve it instantly via matrix algebra.

Complexity Analysis

Understanding the computational complexity of the linear regression algorithm is critical for choosing the right solver and scaling to large datasets. Complexity differs dramatically between the closed-form and iterative methods, and between training and inference phases.

For the Normal Equation, the dominant cost is computing (X^T * X) and its inverse. If X is an m x n matrix (m samples, n features), computing X^T * X takes O(m * n^2) time, and inverting the resulting n x n matrix takes O(n^3) time using standard Gaussian elimination. The overall training time complexity is O(m * n^2 + n^3). Space complexity is O(n^2) to store X^T * X and O(m * n) to store X. This method is very fast when n is small (e.g., < 1000) but becomes infeasible when n is large (e.g., 10,000+ features) due to the cubic term.

For Batch Gradient Descent, each iteration computes predictions and gradients over the entire dataset, costing O(m * n) time per iteration. If it takes k iterations to converge, total training time is O(k * m * n). Space complexity is O(m * n) to store the data plus O(n) for parameters. Stochastic Gradient Descent (SGD) updates parameters using a single random example per iteration, costing O(n) per iteration, while Mini-batch Gradient Descent uses a batch of size b, costing O(b * n) per iteration. These iterative methods scale much better to large m (millions of rows) and large n, and they can be used for online learning.

Prediction (inference) is extremely cheap for all variants. Once theta is learned, predicting for one new example requires a dot product O(n), and for m_new examples, O(m_new * n). This makes linear regression ideal for low-latency production systems like web servers, ad bidding, and real-time databases where fast scoring is required. Model size is just O(n) coefficients, making it memory efficient.

In practice, the choice depends on data dimensions. For n < 1000 and m < 100,000, the Normal Equation (via SVD) is often fastest and most accurate. For m in the millions or n in the thousands, iterative methods with feature scaling are preferred. Libraries like scikit-learn automatically choose an appropriate solver (e.g., 'svd', 'cholesky', 'sag') based on data shape and sparsity.

Variations and Optimizations

While vanilla linear regression is powerful, many real-world problems violate its assumptions or require additional constraints. Several important variations and optimizations extend its applicability and robustness.

Polynomial Regression is a direct extension that models non-linear relationships while remaining linear in the parameters. By adding polynomial features like x^2, x^3, or interaction terms x1*x2, the model can fit curves. For example, y = b0 + b1*x + b2*x^2 is still solved with the same OLS machinery, just with an expanded feature matrix. However, high-degree polynomials can lead to severe overfitting and numerical instability, so they must be combined with regularization and cross-validation.

Regularized Regression methods add a penalty to the cost function to prevent overfitting and handle multicollinearity. Ridge Regression (L2 regularization) adds α * Σ(bj^2), which shrinks coefficients toward zero but keeps all features. It is particularly effective when features are correlated, as it distributes weight among them. Lasso Regression (L1 regularization) adds α * Σ(|bj|), which can shrink some coefficients exactly to zero, performing automatic feature selection. Elastic Net combines both penalties: α * (l1_ratio * L1 + (1-l1_ratio) * L2), offering a balance between Ridge and Lasso and is ideal when there are groups of correlated features.

Other optimizations address specific data issues. Weighted Least Squares gives different weights to different observations, useful when heteroscedasticity is present or some data points are more reliable. Robust Regression (e.g., Huber loss, RANSAC) reduces sensitivity to outliers by using a loss function that is less punitive than squared error for large residuals. Bayesian Linear Regression places a prior distribution over the coefficients, providing uncertainty estimates for predictions rather than just point estimates. For large-scale systems, optimizations like feature hashing, online gradient descent, and distributed computation via MapReduce or Spark allow linear regression to train on terabytes of data across clusters.

Real-World Applications

The linear regression algorithm is far from just an academic exercise; it powers critical systems across web infrastructure, finance, healthcare, and engineering. Its speed, interpretability, and low resource requirements make it a go-to choice for production environments.

In web servers and cloud infrastructure, linear regression is used for capacity planning and auto-scaling. By modeling the relationship between incoming requests per second (RPS), CPU utilization, and response latency, engineers can predict when to spin up additional servers. For example, a model like latency = b0 + b1*RPS + b2*CPU + b3*memory_usage can forecast performance degradation. Similarly, in databases, query optimizers use linear models to estimate query execution time and cost based on features like table size, number of joins, and index selectivity, helping choose the most efficient execution plan.

In networks and advertising technology, linear regression drives real-time bidding and click-through rate (CTR) prediction. While deep models may be used for final ranking, lightweight linear models often serve as fast filters to score millions of ad impressions per second with sub-millisecond latency. Features include user history, ad position, time of day, and device type. The predicted CTR directly determines bid price: bid = base_price * predicted_CTR.

Beyond tech infrastructure, applications are extensive. In finance, it is used for risk modeling, CAPM (Capital Asset Pricing Model) to predict asset returns, and algorithmic trading. In real estate, the classic house price prediction model uses square footage, location, and age. In healthcare, it models the effect of dosage on blood pressure or predicts hospital length of stay. In e-commerce, it forecasts demand and sales based on price, seasonality, and marketing spend, enabling inventory optimization. Its interpretability is crucial in regulated domains where stakeholders must explain why a prediction was made.

Comparison with Alternatives

Choosing the right algorithm requires understanding trade-offs. The linear regression algorithm excels in interpretability and speed but assumes linearity. When relationships are highly non-linear or involve complex interactions, other algorithms may be more accurate, albeit at the cost of complexity and transparency.

Decision Trees partition the feature space into axis-aligned rectangles and predict the average target in each leaf. They handle non-linearity and feature interactions naturally and require no feature scaling, but they are prone to overfitting and are unstable. Random Forests and Gradient Boosted Trees (XGBoost, LightGBM) ensemble many trees to improve accuracy and robustness, often outperforming linear regression on tabular data, but they are less interpretable and slower to train. Support Vector Regression (SVR) with non-linear kernels can model complex curves and is effective in high dimensions, but it scales poorly to large datasets and requires careful tuning of kernel and regularization parameters.

k-Nearest Neighbors (k-NN) is a non-parametric, instance-based method that predicts based on the average of the k closest training examples. It makes no assumptions about functional form and adapts to local patterns, but it is computationally expensive at prediction time (O(m*n)), sensitive to irrelevant features, and suffers from the curse of dimensionality. Neural Networks, especially deep learning models, can approximate any function given enough data and are state-of-the-art for unstructured data like images and text, but they require large datasets, extensive tuning, and are black boxes.

In practice, linear regression is often the first model to try. Its performance establishes a baseline. If residual plots show clear non-linear patterns or R-squared is low, it signals the need for polynomial features or a more flexible model. For many business problems with structured data and a need for explainability, regularized linear regression remains the final production model.

Common Mistakes and Best Practices

Even though the linear regression algorithm is simple to implement, subtle mistakes can lead to misleading results and poor generalization. Awareness of these pitfalls is what separates a novice from a reliable practitioner.

A frequent mistake is ignoring the assumptions and skipping diagnostic checks. Fitting a line to non-linear data will produce systematic errors; always plot residuals vs. fitted values and check for patterns. A funnel shape indicates heteroscedasticity, while a curved pattern suggests non-linearity requiring transformation or polynomial features. Another common error is data leakage, where information from the test set influences training, such as scaling the entire dataset before splitting or including a feature that is derived from the target. Always fit preprocessing (scalers, imputers) only on the training set and then transform the test set.

Multicollinearity is often overlooked. When two features are highly correlated (e.g., house size in sqft and number of rooms), coefficient estimates become unstable and their p-values unreliable. Check the correlation matrix and Variance Inflation Factor (VIF); VIF > 5-10 indicates problematic collinearity. Solutions include dropping one correlated feature, combining them, or using Ridge regression. Overfitting with too many features, especially polynomial terms, is another trap. More features will always increase training R-squared, but may hurt test performance. Use adjusted R-squared, cross-validation, and regularization to penalize complexity.

Best practices ensure robust models. First, always visualize data with scatter plots and pair plots before modeling. Second, scale features when using gradient descent or regularization, as they are sensitive to feature magnitude. Third, handle outliers thoughtfully: investigate whether they are errors or genuine rare events; consider robust regression or log transformation rather than blindly deleting them. Fourth, use proper evaluation: never evaluate on training data alone; use train-test split or k-fold cross-validation and report RMSE, MAE, and R-squared together, as R-squared alone can be misleading. Finally, document and interpret coefficients in the original units after scaling, and communicate uncertainty via confidence intervals and prediction intervals, not just point predictions.

Checklist Before Deployment

• Validated linearity, homoscedasticity, and normality of residuals via plots and tests (e.g., Breusch-Pagan, Q-Q plot).
• Checked VIF for multicollinearity and addressed it if needed.
• Performed cross-validation and compared against a baseline (mean predictor).
• Ensured no data leakage in preprocessing pipeline.
• Tested model latency and memory footprint for production constraints.

Interview Questions

Linear regression is a staple of machine learning interviews because it tests both theoretical understanding and practical judgment. Below are common questions with concise, interview-ready answers.

1. What are the assumptions of linear regression and what happens if they are violated?

The four main assumptions are linearity, independence of errors, homoscedasticity (constant variance), and normality of residuals, plus low multicollinearity. If linearity is violated, predictions will be systematically biased; you can detect this via residual plots and fix it with polynomial features or transformations. Heteroscedasticity makes standard errors unreliable, affecting confidence intervals and hypothesis tests, but predictions remain unbiased; use Weighted Least Squares or transform the target. Autocorrelated errors (common in time series) violate independence and underestimate standard errors. Non-normal residuals affect p-values but not coefficient estimates. Multicollinearity inflates variance of coefficients, making them unstable and hard to interpret; use Ridge or drop correlated features.

2. What is the difference between R-squared and Adjusted R-squared? Why is Adjusted R-squared important?

R-squared (coefficient of determination) measures the proportion of variance in the target explained by the model: R² = 1 - SSE/SST. It ranges from 0 to 1 (can be negative on test data) and always increases or stays the same when you add more features, even if they are irrelevant, which can encourage overfitting. Adjusted R-squared penalizes adding unnecessary features: Adjusted R² = 1 - [(1-R²)*(m-1)/(m-n-1)], where m is samples and n is features. It only increases if the new feature improves the model more than would be expected by chance. Therefore, Adjusted R-squared is a better metric for comparing models with different numbers of predictors and for feature selection.

3. Explain the Bias-Variance Tradeoff in the context of Ridge and Lasso. How do you choose alpha?

Ordinary Least Squares has low bias but can have high variance, especially with many or correlated features, leading to overfitting. Ridge (L2) and Lasso (L1) introduce bias by shrinking coefficients toward zero, which reduces variance and improves generalization. As alpha (regularization strength) increases, bias increases and variance decreases. Lasso can also reduce variance by performing feature selection (setting some coefficients to zero). The optimal alpha balances this tradeoff and is chosen via cross-validation: try a grid of alpha values (e.g., 0.001 to 100 on a log scale), compute cross-validated MSE or R² for each, and pick the alpha with the best validation performance. Typically, you would use RidgeCV or LassoCV in scikit-learn or GridSearchCV.

4. Why do we use Gradient Descent instead of the Normal Equation for large datasets?

The Normal Equation requires computing (X^T * X)^(-1), which costs O(n^3) time and O(n^2) space, where n is the number of features. When n is large (e.g., >10,000), this inversion becomes computationally prohibitive and numerically unstable. It also requires O(m*n^2) to compute X^T*X, which is expensive when m (samples) is in the millions. Gradient Descent, especially Stochastic and Mini-batch variants, costs only O(n) or O(b*n) per iteration and scales linearly with both m and n, making it feasible for large-scale and online learning. Additionally, gradient descent can be used with regularization and is more memory efficient, as it does not need to store the n x n covariance matrix.

5. How would you detect and handle outliers in linear regression?

Outliers can disproportionately influence OLS because squared error penalizes large residuals heavily. Detection methods include visualizing scatter plots and residual plots, computing standardized residuals (>3 indicates outlier), leverage (hat values), and influence measures like Cook's Distance, which combines residual and leverage. To handle them, first investigate whether the outlier is a data entry error, a rare but valid event, or a different population. Options include: correcting errors, removing only if justified and documented, transforming the target (e.g., log), using robust regression (Huber loss or RANSAC) which downweights large residuals, or using regularization. Blindly deleting outliers without investigation is a poor practice and can bias results.