Dimensionality Reduction

Overview

In the study of machine learning, we frequently encounter datasets characterized by a vast number of features or variables. While a rich feature set can provide detailed information, it also introduces significant computational and statistical challenges, a phenomenon often referred to as the "curse of dimensionality." High-dimensional spaces are counter-intuitively sparse, making data analysis difficult. Furthermore, models trained on such data are prone to overfitting, and the computational resources required for processing and training can become prohibitive. This chapter addresses the critical task of transforming data from a high-dimensional space into a feature space of lower dimensionality, while retaining as much of the meaningful properties of the original data as possible.

We shall explore the techniques of dimensionality reduction, which are indispensable tools in the modern data analyst's toolkit. These methods not only serve as a vital preprocessing step to improve the performance and efficiency of subsequent learning algorithms but also aid in data compression and visualization. Our primary focus will be on feature extraction, a process that creates new, smaller sets of features by combining the original ones. For the GATE examination, a thorough understanding of these concepts is paramount, as questions frequently test the theoretical underpinnings and practical application of core techniques. Mastery of this topic will provide a foundational advantage in solving complex problems related to data preprocessing, model optimization, and interpretation.

---

Chapter Contents

| # | Topic | What You'll Learn |
|---|-------|-------------------|
| 1 | Goals of Dimensionality Reduction | Motivations for simplifying high-dimensional data. |
| 2 | Principal Component Analysis (PCA) | Finding orthogonal axes of maximum variance. |

---

Learning Objectives

---

We now turn our attention to Goals of Dimensionality Reduction...
## Part 1: Goals of Dimensionality Reduction

Introduction

In the domain of machine learning and data analysis, we frequently encounter datasets with an exceedingly large number of features or dimensions. Such "high-dimensional" data, while potentially rich in information, presents significant challenges for modeling and interpretation. The phenomenon known as the "Curse of Dimensionality" describes how, as the number of dimensions increases, the volume of the feature space grows exponentially, causing the available data to become sparse. This sparsity makes it difficult for algorithms to discern meaningful patterns, increases computational costs, and elevates the risk of model overfitting.

Dimensionality reduction encompasses a set of techniques designed to transform data from a high-dimensional space into a lower-dimensional space. The core objective is to retain the most meaningful properties of the original data while discarding noise and redundancy. This process is not merely about selecting a subset of features but often involves creating new, composite features (a process known as feature extraction). Understanding the fundamental goals of this process is critical for its effective application in building robust and efficient machine learning pipelines.

---

The Principal Goals of Dimensionality Reduction

We can identify four primary motivations for applying dimensionality reduction techniques. While often interconnected, each goal addresses a distinct challenge posed by high-dimensional data.

#
## 1. Mitigating the Curse of Dimensionality

As the dimensionality $D$ of a feature space increases, the volume of that space grows exponentially. Consequently, a fixed number of data points become increasingly sparse. This sparsity makes it difficult for many machine learning algorithms, particularly those based on distance metrics (like k-Nearest Neighbors), to perform effectively because the concept of "neighborhood" becomes less meaningful.

By reducing the number of dimensions from $D$ to $d$, we project the data into a smaller, denser space. In this lower-dimensional space, the distances between data points become more meaningful again, allowing algorithms to identify clusters and patterns more reliably.

Worked Example:

Problem: Consider a dataset with $N=1000$ points uniformly distributed in a 10-dimensional unit hypercube ($[0,1]^{10}$). We wish to estimate the local density in a small hyper-neighborhood of side length $l=0.1$. What fraction of the total data space does this neighborhood occupy? Contrast this with a 2-dimensional case.

Solution:

Step 1: Define the volume of the total space and the neighborhood. The total space is a unit hypercube, so its volume is $1^D = 1$. The volume of the hyper-neighborhood is $l^D$.

Step 2: Calculate the fractional volume for the 10-dimensional case ($D=10$).

This incredibly small volume implies that any given neighborhood will contain virtually no data points, illustrating data sparsity.

Step 3: Calculate the fractional volume for the 2-dimensional case ($D=2$).

In the 2D case, the neighborhood occupies 1% of the total space, a much more substantial region where we can expect to find neighboring points.

Answer: The neighborhood in the 10D space occupies only $10^{-10}$ of the total volume, whereas in 2D it occupies $0.01$. This demonstrates how dimensionality reduction to a lower-dimensional space makes the data significantly denser.

#
## 2. Improving Computational Efficiency

The computational complexity of many machine learning algorithms is a direct function of the number of features $D$. Training models on data with thousands or millions of features can be prohibitively expensive in terms of both time and memory.

Reducing the dimensionality from $D$ to $d$ (where $d \ll D$) can lead to dramatic improvements in performance. For instance, an algorithm with a complexity of $O(D^2)$ will see its computational cost reduced by a factor of $(D/d)^2$, making model training and prediction significantly faster.

#
## 3. Reducing Model Complexity and Overfitting

High-dimensional data often contains redundant or irrelevant features (noise). Models trained on such data are susceptible to overfitting, where the model learns the noise specific to the training set rather than the underlying signal. This results in a model that performs well on training data but poorly on unseen data.

Dimensionality reduction acts as a form of regularization. By compressing the data into a lower-dimensional representation, it forces the model to focus on the most prominent patterns and structures, effectively filtering out noise. This leads to simpler models that often generalize better.

#
## 4. Enabling Data Visualization and Interpretation

A fundamental goal of data analysis is to gain insight into the structure of the data. However, human perception is limited to two or three dimensions. It is impossible to directly visualize a dataset with, for example, 50 features.

Dimensionality reduction techniques can project high-dimensional data onto a 2D or 3D space, allowing for direct visualization. Scatter plots of the reduced data can reveal clusters, manifolds, outliers, and other intrinsic structures that would be impossible to observe in the original high-dimensional space. This is an indispensable tool for exploratory data analysis.

High-Dimensional Space ($\mathbb{R}^3$)
Low-Dimensional Space ($\mathbb{R}^2$)





X1
X2
X3

Z1
Z2

Projection

---

Problem-Solving Strategies

---

Common Mistakes

---

Practice Questions

:::question type="MSQ" question="Which of the following are valid reasons for applying dimensionality reduction techniques to a high-dimensional dataset before training a machine learning model?" options=["To increase the training time of the model, thereby ensuring a more thorough search of the hypothesis space.","To reduce the risk of the model overfitting to the noise in the training data.","To enable the visualization of the data's intrinsic structure in a 2D or 3D plot.","To decrease the memory requirements for storing the dataset and the model."] answer="To reduce the risk of the model overfitting to the noise in the training data.,To enable the visualization of the data's intrinsic structure in a 2D or 3D plot.,To decrease the memory requirements for storing the dataset and the model." hint="Consider the four primary goals discussed: computational efficiency, overfitting reduction, visualization, and mitigating data sparsity. Evaluate each option against these goals." solution="Option A is incorrect; a key goal is to decrease, not increase, training time. Option B is correct; reducing dimensions can act as a form of regularization, helping the model to generalize better and avoid overfitting. Option C is correct; this is a primary use case for exploratory data analysis. Option D is correct; fewer features directly translate to lower memory (storage) and computational (time) costs."
:::

:::question type="NAT" question="The time complexity of an algorithm is given by $O(N \cdot D^3)$, where $N$ is the number of samples and $D$ is the number of dimensions. If a dataset with 1000 dimensions is projected down to 100 dimensions, by what factor is the computational time reduced? (Assume $N$ is constant)." answer="1000" hint="Calculate the ratio of the original complexity to the new complexity. The factor of reduction is the inverse of this ratio." solution="
Step 1: Define the original and new complexities.
Let $D_{old} = 1000$ and $D_{new} = 100$.
The original time complexity is proportional to $D_{old}^3$.

The new time complexity is proportional to $D_{new}^3$.

Step 2: Calculate the reduction factor. The reduction factor is the ratio of the old time to the new time.

Step 3: Substitute the given values.

Result: The computational time is reduced by a factor of 1000.
"
:::

:::question type="MCQ" question="A data scientist observes that their k-Nearest Neighbors (k-NN) classifier performs poorly on a dataset with 500 features. They note that the distances between most pairs of points are very similar. This problem is a direct consequence of:" options=["Overfitting","The curse of dimensionality","High computational cost","Lack of data for visualization"] answer="The curse of dimensionality" hint="Think about how distance metrics behave in very high-dimensional spaces. When all points are approximately equidistant, neighborhood-based algorithms fail." solution="The phenomenon where distances between points become less meaningful and tend to converge in high-dimensional spaces is a classic symptom of the curse of dimensionality. This makes it difficult for distance-based algorithms like k-NN to distinguish between 'near' and 'far' neighbors. Overfitting and high computational cost are also problems of high-dimensional data, but the specific issue described (failure of distance metrics) points directly to the curse of dimensionality."
:::

---

Summary

---

What's Next?

---

---

Part 2: Principal Component Analysis (PCA)

Introduction

Principal Component Analysis (PCA) stands as a cornerstone of unsupervised learning, specifically within the domain of dimensionality reduction. In the analysis of high-dimensional datasets, we often encounter issues such as multicollinearity, computational inefficiency, and the "curse of dimensionality," which can impede the performance of subsequent machine learning models. PCA provides an elegant mathematical solution to this problem by transforming a set of possibly correlated variables into a smaller set of uncorrelated variables called principal components.

The fundamental objective of this technique is to identify the directions, or principal components, along which the variation in the data is maximal. It achieves this by projecting the data onto a lower-dimensional subspace while preserving as much of the original data's variance as possible. This new subspace is defined by a set of orthogonal axes that are ordered by the amount of variance they explain. Consequently, PCA is not merely a data compression tool; it is a powerful technique for feature extraction, data visualization, and noise filtering, making it an indispensable topic for the GATE examination.

---

Key Concepts

#
## 1. The Goal of PCA: Maximizing Variance

The central intuition behind PCA is that the directions in which the data varies the most are the directions that contain the most information. Conversely, directions with very little variance might be considered noise or redundant information. PCA systematically finds these directions of maximum variance.

Let us consider a dataset of points in a two-dimensional plane. If these points form an elliptical cloud, we can intuitively see that there is one axis along which the points are most spread out, and an orthogonal axis along which the spread is smaller. The first axis is the first principal component (PC1), and the second is the second principal component (PC2).

PC1
PC2
Original Feature 2
Original Feature 1

In the diagram above, PC1 captures the primary direction of variance, while PC2, being orthogonal to PC1, captures the remaining variance. By projecting the data points onto the PC1 axis, we can represent the data in one dimension while retaining most of its original structure.

#
## 2. The Covariance Matrix

To mathematically identify these directions of variance, we must first quantify the relationships between the different features (dimensions) in our data. The covariance matrix serves this exact purpose. For a dataset with $d$ features, the covariance matrix is a $d \times d$ symmetric matrix where the element $(i, j)$ is the covariance between the $i$-th and $j$-th features. The diagonal elements represent the variance of each feature.

A critical prerequisite for PCA is that the data must be mean-centered. That is, for each feature, we subtract its mean from all observations. Let our dataset be represented by a matrix $X$ of size $n \times d$, where $n$ is the number of observations and $d$ is the number of features. If we assume $X$ is already mean-centered (i.e., the mean of each column is zero), the covariance matrix $C$ can be computed efficiently.

#
## 3. Eigen-Decomposition of the Covariance Matrix

The core of PCA lies in the eigen-decomposition of the covariance matrix $C$. The eigenvectors of $C$ provide the directions of the principal components, and the corresponding eigenvalues specify the amount of variance along those directions.

Let $u$ be an eigenvector and $\lambda$ be its corresponding eigenvalue for the covariance matrix $C$. They satisfy the characteristic equation:

For a $d \times d$ covariance matrix, we will find $d$ eigenvalue-eigenvector pairs. These eigenvectors, which represent the principal components, are orthogonal to each other. We sort these pairs in descending order based on their eigenvalues.

• The First Principal Component (PC1) is the eigenvector $u_1$ corresponding to the largest eigenvalue $\lambda_1$.
• The Second Principal Component (PC2) is the eigenvector $u_2$ corresponding to the second-largest eigenvalue $\lambda_2$.
• And so on, up to the $d$-th principal component.

A profoundly important result, and one frequently tested in GATE, is the relationship between an eigenvalue and the variance of the data projected onto its corresponding eigenvector.

#
## 4. The PCA Algorithm: A Step-by-Step Procedure

Let us formalize the process into a clear algorithm.

Given: A dataset $X$ of size $n \times d$.
Goal: Reduce dimensionality from $d$ to $k$ (where $k < d$).

Step 1: Mean-Center the Data
For each feature (column) $j=1, \dots, d$, compute its mean $\mu_j$. Then, for each data point $x_{ij}$, update it as $x_{ij} \leftarrow x_{ij} - \mu_j$. This ensures that the transformed dataset has a zero mean.

Step 2: Compute the Covariance Matrix
Using the mean-centered data matrix $X'$, calculate the $d \times d$ covariance matrix:

Step 3: Compute Eigenvectors and Eigenvalues
Perform eigen-decomposition on the covariance matrix $C$ to find its $d$ eigenvalues $\lambda_1, \lambda_2, \dots, \lambda_d$ and corresponding eigenvectors $u_1, u_2, \dots, u_d$.

Step 4: Sort Eigenvectors
Sort the eigenvalues in descending order: $\lambda_1 \ge \lambda_2 \ge \dots \ge \lambda_d$. Rearrange the corresponding eigenvectors according to this new order. The vector $u_1$ is now PC1, $u_2$ is PC2, and so forth.

Step 5: Select Principal Components and Form the Projection Matrix
Choose the top $k$ eigenvectors to form the new feature space. These $k$ vectors form the projection matrix $W$, which is a $d \times k$ matrix where each column is a principal component:

Step 6: Project the Data
Transform the original mean-centered data $X'$ onto the new $k$-dimensional subspace by multiplying it with the projection matrix $W$:

The resulting matrix $Z$ is of size $n \times k$ and represents the original data in the lower-dimensional space.

#
## 5. Explained Variance

To decide on a suitable value for $k$ (the number of dimensions to keep), we often examine the "explained variance." The total variance in the dataset is the sum of all eigenvalues of the covariance matrix (which is also the trace of the matrix).

Worked Example:

Problem:
Consider the following 2x2 covariance matrix $C$ for a dataset:

Find the principal components and their corresponding variances (eigenvalues).

Solution:

Step 1: Find the eigenvalues of $C$.
We solve the characteristic equation $\det(C - \lambda I) = 0$.

The eigenvalues are $\lambda_1 = 5.0$ and $\lambda_2 = 0.8$.

Step 2: Identify the variance along each principal component.
The largest eigenvalue corresponds to the first principal component.
Variance along PC1 = $\lambda_1 = 5.0$.
Variance along PC2 = $\lambda_2 = 0.8$.

Step 3: Find the eigenvector for $\lambda_1 = 5.0$ (PC1).
We solve $(C - \lambda_1 I)u_1 = 0$.

This gives the equation $-2.1x + 2.1y = 0$, which simplifies to $x=y$. A possible eigenvector is $\begin{pmatrix} 1 \\ 1 \end{pmatrix}$. Normalizing it to a unit vector, we get:

Step 4: Find the eigenvector for $\lambda_2 = 0.8$ (PC2).
We solve $(C - \lambda_2 I)u_2 = 0$.

This gives the equation $2.1x + 2.1y = 0$, which simplifies to $x=-y$. A possible eigenvector is $\begin{pmatrix} 1 \\ -1 \end{pmatrix}$. Normalizing it, we get:

Answer:
The first principal component is the direction $u_1 = \begin{pmatrix} 1/\sqrt{2} \\ 1/\sqrt{2} \end{pmatrix}$ with a variance of $5.0$. The second principal component is the direction $u_2 = \begin{pmatrix} 1/\sqrt{2} \\ -1/\sqrt{2} \end{pmatrix}$ with a variance of $0.8$.

---

Problem-Solving Strategies

---

Common Mistakes

---

Practice Questions

:::question type="NAT" question="A mean-centered dataset in $\mathbb{R}^4$ has a covariance matrix whose eigenvalues are 25, 16, 4, and 1. What percentage of the total variance is captured by the first two principal components? (Round off to two decimal places)" answer="89.13" hint="First, calculate the total variance by summing all eigenvalues. Then, sum the variance captured by the first two components (the two largest eigenvalues) and express it as a percentage of the total." solution="
Step 1: Identify the eigenvalues for the first two principal components.
The eigenvalues are given as 25, 16, 4, and 1. They are already sorted in descending order.
The largest eigenvalue is $\lambda_1 = 25$.
The second-largest eigenvalue is $\lambda_2 = 16$.

Step 2: Calculate the total variance.
The total variance is the sum of all eigenvalues.

Step 3: Calculate the variance captured by the first two principal components.
This is the sum of the first two eigenvalues.

Step 4: Compute the percentage of total variance.

Result:
Rounding to two decimal places, the percentage is 89.13.
"
:::

:::question type="MCQ" question="In the context of Principal Component Analysis, what do the eigenvectors of the data's covariance matrix represent?" options=["The magnitude of variance in the data","The directions of maximum variance","The projected data points in the new subspace","The mean of the dataset"] answer="The directions of maximum variance" hint="Recall the fundamental goal of PCA and the mathematical objects used to achieve it." solution="
The core idea of PCA is to find a new set of orthogonal axes (or directions) that align with the greatest variance in the data. The mathematical procedure for finding these directions involves the eigen-decomposition of the covariance matrix. The eigenvectors of this matrix point in these exact directions of maximal variance. The corresponding eigenvalues quantify the amount of variance along these directions. Therefore, the eigenvectors represent the directions of maximum variance.
"
:::

:::question type="MSQ" question="Let the covariance matrix of a 2D dataset be $C = \begin{pmatrix} 5 & 0 \\ 0 & 2 \end{pmatrix}$. Which of the following statements is/are correct?" options=["The first principal component is aligned with the direction vector $\begin{pmatrix} 1 \\ 0 \end{pmatrix}$.","The variance of the data along the second principal component is 5.","The total variance in the dataset is 7.","The principal components are orthogonal."] answer="The first principal component is aligned with the direction vector $\begin{pmatrix} 1 \\ 0 \end{pmatrix}$.,The total variance in the dataset is 7.,The principal components are orthogonal." hint="For a diagonal covariance matrix, the eigenvalues are the diagonal entries and the eigenvectors are the standard basis vectors. Check each statement based on this." solution="
The given covariance matrix is a diagonal matrix:

For a diagonal matrix, the eigenvalues are the diagonal elements, and the eigenvectors are the standard basis vectors.
The eigenvalues are $\lambda_1 = 5$ and $\lambda_2 = 2$.
The eigenvector corresponding to $\lambda_1 = 5$ is $u_1 = \begin{pmatrix} 1 \\ 0 \end{pmatrix}$. This is the first principal component (PC1).
The eigenvector corresponding to $\lambda_2 = 2$ is $u_2 = \begin{pmatrix} 0 \\ 1 \end{pmatrix}$. This is the second principal component (PC2).

Let's evaluate the options:

• "The first principal component is aligned with the direction vector $\begin{pmatrix} 1 \\ 0 \end{pmatrix}$." This is correct. The eigenvector for the largest eigenvalue ($\lambda_1=5$) is indeed $\begin{pmatrix} 1 \\ 0 \end{pmatrix}$.


• "The variance of the data along the second principal component is 5." This is incorrect. The variance along PC2 is given by the second-largest eigenvalue, which is $\lambda_2 = 2$.


• "The total variance in the dataset is 7." This is correct. The total variance is the sum of the eigenvalues: $5 + 2 = 7$.


• "The principal components are orthogonal." This is correct. The eigenvectors of a symmetric matrix (like a covariance matrix) corresponding to distinct eigenvalues are always orthogonal. Here, $u_1^\mathsf{T} u_2 = (1)(0) + (0)(1) = 0$.

Therefore, the correct statements are the first, third, and fourth options.
"
:::

:::question type="NAT" question="The covariance matrix of a mean-centered dataset has three eigenvalues: $\lambda_1=50$, $\lambda_2=30$, and $\lambda_3=20$. If $u_2$ is the unit eigenvector corresponding to the second principal component, what is the value of the expression $\frac{1}{n} \sum_{i=1}^n (u_2^\mathsf{T} x^{(i)})^2$?" answer="30" hint="Recognize the given expression. It is the formula for the variance of the data projected onto the direction vector $u_2$. This variance is equal to the eigenvalue corresponding to $u_2$." solution="
Step 1: Identify the expression.
The expression $\frac{1}{n} \sum_{i=1}^n (u_2^\mathsf{T} x^{(i)})^2$ represents the variance of the dataset projected onto the direction defined by the vector $u_2$.

Step 2: Relate the expression to PCA concepts.
In PCA, the vector $u_2$ is the second principal component, which is the eigenvector corresponding to the second-largest eigenvalue, $\lambda_2$.

Step 3: Apply the core theorem of PCA.
A fundamental property of PCA is that the variance of the data projected onto a principal component (eigenvector) is equal to its corresponding eigenvalue.
Therefore, the value of the expression is equal to $\lambda_2$.

Step 4: Substitute the given value.
The eigenvalues are given as $\lambda_1=50$, $\lambda_2=30$, and $\lambda_3=20$.
The second-largest eigenvalue is $\lambda_2 = 30$.

Result:
The value of the expression is 30.
"
:::

---

Summary

---

What's Next?

---

Chapter Summary

---

Chapter Review Questions

:::question type="MCQ" question="A 2D dataset has a covariance matrix given by $S = \begin{pmatrix} 5 & 2 \\ 2 & 2 \end{pmatrix}$. Which of the following vectors represents the direction of the first principal component?" options=["$\begin{pmatrix} 2 \\ 1 \end{pmatrix}$","$\begin{pmatrix} 1 \\ -2 \end{pmatrix}$","$\begin{pmatrix} 1 \\ 1 \end{pmatrix}$","$\begin{pmatrix} -1 \\ -2 \end{pmatrix}$"] answer="A" hint="The first principal component is the eigenvector of the covariance matrix corresponding to the largest eigenvalue." solution="The principal components are the eigenvectors of the covariance matrix $S$. We find the eigenvalues by solving the characteristic equation $\det(S - \lambda I) = 0$.

The eigenvalues are $\lambda_1 = 6$ and $\lambda_2 = 1$. The first principal component corresponds to the largest eigenvalue, $\lambda_1 = 6$. We find its corresponding eigenvector $v_1 = \begin{pmatrix} x \\ y \end{pmatrix}$ by solving $(S - \lambda_1 I)v_1 = 0$.


This gives the equation $-x + 2y = 0$, or $x = 2y$. A vector satisfying this condition is $\begin{pmatrix} 2 \\ 1 \end{pmatrix}$. Thus, this is the direction of the first principal component."
:::

:::question type="NAT" question="For a dataset whose features have a covariance matrix $S = \begin{pmatrix} 13 & -4 \\ -4 & 7 \end{pmatrix}$, what percentage of the total variance is explained by the first principal component?" answer="75" hint="The variance explained by a principal component is given by its corresponding eigenvalue. The total variance is the sum of all eigenvalues." solution="First, we find the eigenvalues of the covariance matrix $S$. The characteristic equation is $\det(S - \lambda I) = 0$.

The eigenvalues are $\lambda_1 = 15$ and $\lambda_2 = 5$. The total variance in the data is the sum of the eigenvalues (which is also the trace of the covariance matrix):

The first principal component corresponds to the largest eigenvalue, $\lambda_1 = 15$. The percentage of variance it explains is:

"
:::

:::question type="MCQ" question="Which of the following statements regarding Principal Component Analysis (PCA) is FALSE?" options=["PCA is sensitive to the scale of the input features, and standardization is often a required preprocessing step.","The principal components are linear combinations of the original features and are mutually orthogonal.","PCA is a supervised learning technique as it requires labeled data to find the directions of maximum variance.","The variance of the data projected onto the $k$-th principal component is equal to the $k$-th largest eigenvalue of the data's covariance matrix."] answer="C" hint="Consider whether PCA utilizes target labels ($y$ values) during its computational process." solution="Let us evaluate each statement:

• A: True. If one feature has a much larger variance than others, it will dominate the first principal component. Therefore, standardizing features to have zero mean and unit variance is a standard and necessary step before applying PCA.


• B: True. By definition, each principal component is a weighted linear combination of the original features. The principal components (which are the eigenvectors of the symmetric covariance matrix) are orthogonal to each other.


• C: False. PCA is an unsupervised learning algorithm. It only analyzes the relationships and variance within the feature set ($X$) and does not use any class labels or target variables ($y$). Its objective is to find the best representation of the input data, not to predict an output.


• D: True. This is a fundamental property of PCA. The eigenvalue $\lambda_k$ corresponding to the $k$-th eigenvector (principal component) quantifies the amount of variance in the data along that component's direction.

Therefore, the false statement is C."
:::

:::question type="NAT" question="Consider the following 2D dataset with 3 data points, which has already been mean-centered: $X = \begin{pmatrix} -1 & -2 \\ 0 & 0 \\ 1 & 2 \end{pmatrix}$. Calculate the element $S_{12}$ (the covariance between the first and second feature) of the sample covariance matrix, defined as $S = \frac{1}{n-1}X^T X$." answer="2" hint="Compute the matrix product $X^T X$ first, then scale by the appropriate factor, where $n$ is the number of data points." solution="We are given the mean-centered data matrix $X$ and the number of samples $n=3$. The formula for the sample covariance matrix is $S = \frac{1}{n-1}X^T X$.

First, we find the transpose of $X$:

Next, we compute the matrix product $X^T X$:



Finally, we scale this matrix by $\frac{1}{n-1} = \frac{1}{3-1} = \frac{1}{2}$:

The element $S_{12}$ is the entry in the first row and second column, which is 2."
:::

---

What's Next?