Singular Value Decomposition and Principal Component Analysis Notes for GATE DA

Singular Value Decomposition and Principal Component Analysis notes for GATE DA: 9 study cards covering concepts, formulas, shortcuts and exam traps, plus solved practice questions.

Chapter Roadmap: SVD and PCA

Orientation

Chapter Roadmap: SVD and PCA

Matrix Decompositions, with the current stop on PCA eigenvalues and the direction of maximum variance.

Current topic: PCA Eigenvalues and Maximum Variance Direction

Core idea: covariance eigenvalues measure variance along principal directions.

Goal: identify the direction that captures maximum variance.

Exam focus: direct use of the largest eigenvalue and the quadratic form.

Next topic: Rank-One Matrices and Singular Values

Core idea: outer products create simple matrices with clean spectral structure.

Goal: connect singular values to rank-one constructions.

Exam focus: singular values of structured matrices.

The Goal: Finding the Direction of Maximum Spread

Concept

The Goal: Finding the Direction of Maximum Spread

Geometric Meaning

A centered data cloud can be spread differently along different directions. Principal Component Analysis asks:

Which unit direction keeps the largest spread when all points are projected onto it?

That direction is called the first principal component.

Algebraic Translation

If $Σ$ is the covariance matrix of centered data, then the best direction is an eigenvector of $Σ$ .

Σ v = λ v

The direction with the largest variance is the eigenvector corresponding to

λ_{m a x}

Key Mapping

Geometry	Linear Algebra
Direction of projection	Unit vector $u$
Spread along that direction	$u^{T} Σ u$
Best direction	Eigenvector for $λ_{m a x}$
Variance kept there	$λ_{m a x}$

Why Eigenvalues Equal Variance

Concept

Why Eigenvalues Equal Variance

Starting Point

For a covariance matrix $Σ$ , suppose $u$ is an eigenvector:

Σ u = λ u

Multiply by the Transpose

Left-multiply by $u^{T}$ :

u^{T} Σ u = λ u^{T} u

Use Unit Length

Since $u$ is a unit vector,

u^{T} u = 1

Hence,

u^{T} Σ u = λ

Interpretation

The expression $u^{T} Σ u$ is the variance of data projected onto $u$ .

If $u$ is an eigenvector, projected variance equals its eigenvalue.
If $u$ is the eigenvector for $λ_{m a x}$ , projected variance is maximized.
Sorting eigenvalues sorts directions by captured variance.

λ_{1} \geq λ_{2} \geq \dots \geq λ_{d}

The first eigenvector gives the direction of maximum variance.

Core Result

u^{T} u = 1 max u^{T} Σ u = λ_{m a x}

Solving Maximum-Variance Problems Step by Step

Method

Solving Maximum-Variance Problems Step by Step

Step	Action	Reason
1	Check whether data is centered	The clean variance identity needs zero mean
2	Identify $Σ$ or its eigenvalues	The answer usually comes from the spectrum
3	Recognize the projection expression	$\frac{1}{n} \sum (u^{T} x^{(i)})^{2} = u^{T} Σ u$
4	Use the maximum-variance condition	If $u$ is optimal, $u^{T} Σ u = λ_{m a x}$
5	Compute $λ_{m a x}$	Use the given matrix, sequence, or formula

Eigenvalues given directly

Pick the largest one.

Eigenvalues given by a formula

Analyze the formula and find where it is maximum.

Covariance matrix given explicitly

Solve

det (Σ - λ I) = 0

then choose the largest eigenvalue.

Data matrix given

If $X$ is an $n \times d$ matrix with centered rows as observations, then

Σ = \frac{1}{n} X^{T} X

Then find its largest eigenvalue.

Singular Value Decomposition and Principal Component Analysis: Solved Questions with Step-by-Step Explanations (10 Problems)

Question 1 · Linear Algebra MCQ

In Principal Component Analysis (PCA), the first principal component is defined as the direction that:

A.
Minimizes the reconstruction error of the data
B.
Maximizes the variance of the projected data
C.
Minimizes the trace of the covariance matrix
D.
Maximizes the determinant of the covariance matrix

Correct Answer:

Step-by-Step Solution

Key idea: This is a definition-based question about the primary objective of PCA.

Step 1: Recall that PCA seeks to find orthogonal directions (principal components) that capture the most information in the data.

Step 2: Information in this context is measured by variance. The first principal component is specifically the unit vector $u$ that maximizes the variance of the data when projected onto it.

Answer: Maximizes the variance of the projected data.

Question 2 · Linear Algebra MCQ

If $v$ is an eigenvector of the covariance matrix $C$ with corresponding eigenvalue $λ$ , what is the variance of the data along the direction $v$ (assuming $v$ is normalized)?

A.
$λ^{2}$
B.
$λ$
C.
$λ$
D.
$1/ λ$

Correct Answer:

Step-by-Step Solution

Key idea: This question connects the spectral decomposition of the covariance matrix to statistical variance.

Step 1: The variance along a direction $v$ is $v^{T} C v$ .

Step 2: Since $v$ is an eigenvector, $C v = λ v$ .

Step 3: Substitute this into the variance formula: $v^{T} (λ v) = λ (v^{T} v)$ .

Step 4: Since $v$ is normalized, $v^{T} v = 1$ . Thus, the variance is $λ$ .

Answer: $λ$

Question 3 · Linear Algebra MCQ

In Principal Component Analysis, if a unit vector $u$ is constrained to be orthogonal to the first principal component, what is the maximum possible variance of the data projected onto $u$ ?

A.
The largest eigenvalue of the covariance matrix
B.
The sum of all eigenvalues
C.
The second largest eigenvalue of the covariance matrix
D.
Zero

Correct Answer:

Step-by-Step Solution

Key idea: This is a comparison question, recognizable because it asks for the maximum variance under an orthogonality constraint to the first PC.

Why it applies: The variational characterization of eigenvalues (Courant-Fischer theorem) states that the maximum variance in the subspace orthogonal to the first $k - 1$ principal components is the $k$ -th largest eigenvalue.

Step 1: The first principal component corresponds to the largest eigenvalue $λ_{1}$ .

Step 2: We are looking for the maximum variance in the subspace orthogonal to this first component.

Step 3: By definition of PCA, the direction that maximizes variance in this orthogonal subspace is the second principal component.

Step 4: The variance along the second principal component is the second largest eigenvalue, $λ_{2}$ .

Answer: The correct option is C.

Trap: Assuming the maximum variance is still the largest eigenvalue, ignoring the orthogonality constraint.

Question 4 · Linear Algebra MCQ

Consider the following assertion and reason:

Assertion (A): If $u = [34]$ and $M = u u^{T}$ , the sum of the singular values of $M$ is 25.

Reason (R): The only non-zero singular value of $u u^{T}$ is constructed by taking the squared Euclidean norm of $u$ .

A.
Both A and R are true and R is the correct explanation of A
B.
Both A and R are true but R is NOT the correct explanation of A
C.
A is true but R is false
D.
A is false but R is true

Correct Answer:

Step-by-Step Solution

Key idea: For a symmetric rank-one matrix $M = u u^{T}$ , the single non-zero singular value is $∥ u ∥^{2}$ . The sum of all singular values is just this single value.

Step 1: Evaluate Reason (R).

The non-zero singular value of $u u^{T}$ is indeed $∥ u ∥^{2}$ . So, R is true.

Step 2: Evaluate Assertion (A).

Calculate $∥ u ∥^{2}$ for $u = [34]$ .

$∥ u ∥^{2} = 3^{2} + 4^{2} = 9 + 16 = 25$ .

Since there is only one non-zero singular value, the sum of the singular values is 25. So, A is true.

Step 3: Check the link.

R provides the exact formula needed to compute the value in A. Thus, R is the correct explanation for A.

Answer: Both A and R are true and R is the correct explanation of A

Question 5 · Linear Algebra MCQ

Consider the following assertion and reason:

Assertion (A): If $u = 221$ and $M = u u^{T}$ , the sum of the singular values of $M$ is $3$ .

Reason (R): The only non-zero singular value of $u u^{T}$ equals $∥ u ∥^{2}$ .

A.
Both A and R are true and R is the correct explanation of A
B.
Both A and R are true but R is NOT the correct explanation of A
C.
A is true but R is false
D.
A is false but R is true

Correct Answer:

Step-by-Step Solution

Key idea: This is an assertion-reason question on the special case $M = u u^{T}$ . The non-zero singular value is $∥ u ∥^{2}$ (the squared norm), not $∥ u ∥$ (the norm).

Exam route: Evaluate R first (it states a general fact), then evaluate A by computation.

Step 1: Evaluate Reason (R).

For $M = u u^{T}$ , the eigenvalues are $∥ u ∥^{2}, 0, 0, \dots, 0$ . Since $M$ is symmetric positive semidefinite, singular values equal eigenvalues. So the only non-zero singular value is $∥ u ∥^{2}$ . R is true.

Step 2: Evaluate Assertion (A).

Compute $∥ u ∥^{2}$ for $u = 221$ .

$∥ u ∥^{2} = 2^{2} + 2^{2} + 1^{2} = 4 + 4 + 1 = 9$ .

The sum of the singular values is $9 + 0 + 0 = 9$ .

Assertion A claims the sum is $3$ . This is false.

Step 3: Determine the relationship.

A is false, R is true.

Wrong path: If you compute $∥ u ∥ = 4 + 4 + 1 = 9 = 3$ and use that as the singular value, you would get the sum as $3$ and conclude A is true. This is the unit mismatch trap: confusing $∥ u ∥$ (length) with $∥ u ∥^{2}$ (squared length). The correct singular value for $u u^{T}$ is the squared length.

Generalization: For $u u^{T}$ , always use $∥ u ∥^{2}$ for the singular value, not $∥ u ∥$ . For a general outer product $a b^{T}$ , use $∥ a ∥ ∥ b ∥$ .

Verification: $M u = u (u^{T} u) = u ∥ u ∥^{2} = 9 u$ . So $u$ is an eigenvector with eigenvalue $9$ . Since $M$ is positive semidefinite, the singular value is $9$ . The sum of singular values is $9$ , not $3$ . Confirmed.

Answer: A is false but R is true

Question 6 · Linear Algebra MCQ

Let $C$ be the covariance matrix of a dataset, and let $w$ be a unit vector. Which expression represents the variance of the data when projected onto the direction $w$ ?

A.
$w^{T} C w$
B.
$w^{T} C^{- 1} w$
C.
$tr (C w)$
D.
$det (C)$

Correct Answer:

Step-by-Step Solution

Key idea: This question tests the algebraic formula for projected variance using the covariance matrix.

Step 1: The projection of a data point $x$ onto a unit vector $w$ is the scalar $w^{T} x$ .

Step 2: The variance of these projections is given by the quadratic form $w^{T} C w$ .

Answer: $w^{T} C w$

Question 7 · Linear Algebra MCQ

A centered dataset in $R^{d}$ has a covariance matrix with eigenvalues given by $λ_{i} = 80 \cdot (1/2)^{i}$ for $i = 1, 2, \dots, d$ . What is the maximum possible variance of the data when projected onto any single unit direction?

A.
80
B.
20
C.
40
D.
10

Correct Answer:

Step-by-Step Solution

Key idea: This is a sequence evaluation question, recognizable because it gives a formula for eigenvalues and asks for the maximum variance.

Why it applies: The maximum variance in any direction is the largest eigenvalue of the covariance matrix.

Step 1: The eigenvalues are given by $λ_{i} = 80 \cdot (1/2)^{i}$ for $i = 1, 2, \dots, d$ .

Step 2: Since $(1/2)^{i}$ is a decreasing sequence, the largest eigenvalue occurs at the smallest index, $i = 1$ .

Step 3: Calculate $λ_{1} = 80 \cdot (1/2)^{1} = 40$ .

Step 4: The maximum possible variance of the data when projected onto any single unit direction is exactly this largest eigenvalue.

Answer: The correct option is C.

Trap: Evaluating the sequence at $i = 0$ (which would give 80) or summing the series. The problem explicitly specifies $i \geq 1$ .

Question 8 · Linear Algebra MCQ

To find the direction $u$ that maximizes the variance $u^{T} C u$ , we must optimize this quadratic form subject to which constraint?

A.
$u^{T} C u = 1$
B.
$det (C) = 1$
C.
$u^{T} u = 1$
D.
$tr (C) = 1$

Correct Answer:

Step-by-Step Solution

Key idea: This is an optimization constraint question, recognizable because it asks for the condition under which the variance is maximized.

Why it applies: Maximizing a quadratic form requires a constraint, otherwise the variance could be made arbitrarily large by scaling the vector.

Step 1: We want to maximize the variance $u^{T} C u$ .

Step 2: If we scale $u$ by a constant $k$ , the variance becomes $k^{2} u^{T} C u$ , which can be infinitely large.

Step 3: To prevent this trivial scaling, we must restrict the length of $u$ .

Step 4: The standard constraint in PCA is that $u$ must be a unit vector, which is written as $u^{T} u = 1$ or $∥ u ∥^{2} = 1$ .

Answer: The correct option is C.

Trap: Confusing the objective function ( $u^{T} C u$ ) with the constraint. The constraint is strictly on the norm of the vector $u$ , not on the variance itself.

Question 9 · Linear Algebra NAT

A dataset has a covariance matrix with eigenvalues $50$ , $30$ , and $20$ . What proportion of the total variance is captured by the first two principal components?

Correct Answer:

0.8

Step-by-Step Solution

Key idea: This is a direct formula question, recognizable because it asks for the proportion of variance explained by a subset of principal components.

Why it applies: The proportion of variance explained by the first $k$ components is the sum of their eigenvalues divided by the sum of all eigenvalues (total variance).

Step 1: Identify the eigenvalues: $λ_{1} = 50$ , $λ_{2} = 30$ , $λ_{3} = 20$ .

Step 2: Calculate the total variance, which is the sum of all eigenvalues: $50 + 30 + 20 = 100$ .

Step 3: Calculate the variance captured by the first two principal components: $50 + 30 = 80$ .

Step 4: Compute the proportion: $80/100 = 0.8$ .

Answer: 0.8

Trap: Forgetting to divide by the total variance, or only taking the first eigenvalue.

Question 10 · Linear Algebra MCQ

Let $a$ and $b$ be vectors in $R^{3}$ such that $∥ a ∥ \leq 2$ and $∥ b ∥ \leq 3$ . What is the maximum possible value of the largest singular value of the matrix $A = a b^{T}$ ?

A.
$6$
B.
$5$
C.
$9$
D.
$36$

Correct Answer:

Step-by-Step Solution

Key idea: The largest (and only non-zero) singular value of $A = a b^{T}$ is $σ_{1} = ∥ a ∥∥ b ∥$ . To maximize this, we maximize the product of the norms.

Step 1: Identify the bounds on the norms.

$∥ a ∥ \leq 2$ and $∥ b ∥ \leq 3$ .

Step 2: Maximize the product.

The maximum value of $∥ a ∥∥ b ∥$ occurs when both norms are at their maximum possible values.

$max (σ_{1}) = 2 \times 3 = 6$ .

Answer: 6