Polynomial feature transformation allows a [[Linear Classifier]] to model non-linear relationships by transforming the input data into a higher-dimensional feature space. The transformation maps each observation $x \in \mathbb R^d$ to a higher-dimensional feature [[Vector Operations|Vector]] $\phi(x) \in \mathbb{R}^p$, where $p > d$. The linear classifier then operates on the transformed feature vectors as if they were the original inputs. ## Quadratic Transformation Assume we have classifier $h(x; \theta, \theta_0)$ where $x \in \mathbb R$. Thus we have a 1-dimensional input, that needs to be classified into $y \in \{-1,1\}$. $ h(x; \theta, \theta_0) = \text{sign}(\theta \cdot x+\theta_0) $ **Problem:** The observations might not be linearly separable in the original space. ![[feature-transform-1.png|center|300]] **Solution:** To make the data linearly separable, we map the 1-dimensional $x$ into a 2-dimensional feature vector $\phi(x)$. Consequently also our $\theta$ parameters get expanded to match with the higher dimensionality of new input features. $ x \mapsto \phi(x) = \begin{bmatrix}x\phantom{^2}\\x^2\end{bmatrix} \qquad \theta \mapsto \theta =\begin{bmatrix} \theta_1 \\ \theta_2\end{bmatrix} $ The classifier now writes as follows, and allows linear separability. $ \begin{align} h(x; \theta, \theta_0) &= \text{sign}(\theta \cdot \phi(x)+\theta_0) \\ &= \text{sign}(\theta_1*x_1+ \theta_2*x_2+ \theta_0) \end{align} $ ![[feature-transform-2.png|center|300]] ## Code Example: Polynomial Regression Below is an example of fitting polynomial [[Univariate Linear Regression|Linear Regression]] models to data using `scikit-learn`. It showcases that the more polynomial terms we add, the better we can fit the training data. However this bears the risk of overfitting. *Step 1:* Create a function that fits a polynomial regression and plots it: ```python import matplotlib.pyplot as plt import numpy as np from sklearn.linear_model import LinearRegression from sklearn.preprocessing import PolynomialFeatures # Function to fit and plot polynomial regression def fit_and_plot_poly(X, Y, degree): # Get up to the n-th polynomial of X feature_mapping = PolynomialFeatures(degree=degree, include_bias=False) X_poly = feature_mapping.fit_transform(X) # Fit linear regression lin_reg = LinearRegression() lin_reg.fit(X_poly, Y) # Predict for discrete steps in data range X_plot = np.linspace(X.min(), X.max(), 1000).reshape(-1, 1) X_plot_poly = feature_mapping.transform(X_plot) Y_plot = lin_reg.predict(X_plot_poly) # Return plot plt.plot(X_plot, Y_plot, label=f'Degree {degree}') ``` *Step 2:* Generate random data points for $x$ and $y$. ```python # Number of data points n = 50 # Create a synthetic dataset np.random.seed(0) X = np.random.rand(n, 1) y = 2 * X + 1 + 0.5 * np.random.randn(n, 1) ``` *Step 3:* Plot the fit of different polynomial degrees: ```python fig = plt.figure(figsize=(12,7)) # Plot the original dataset plt.scatter(X, y, color='grey', label='Data points', s=3) # Fit and plot polynomial regressions with different degrees degrees = [1, 5, 10, 25] for degree in degrees: fit_and_plot_poly(X, y, degree) # Configure and show the plot plt.xlabel('X') plt.ylabel('y') plt.legend() plt.title('Linear Regression with Polynomial Features of Different Degrees') plt.show() ``` The resulting plot shows how polynomial transformations of increasing degree fit the data: ![[feature-transform-3.png|center|500]]