chapter

Singular Value Decomposition and Principal Component Analysis Short Notes for GATE DA

Singular Value Decomposition and Principal Component Analysis short notes for GATE DA: 8 study cards covering concepts, formulas, shortcuts and exam traps, pl

singular value decomposition and principal component analysis short notes

Worked Example: Eigenvalues Given by a Decreasing Formula

Trap: Total Variance Versus Maximum Directional Variance

Exam Patterns: How Maximum Variance Questions Appear

#note-card-c008 * { box-sizing: border-box; } #note-card-c008 .c008-badge { display: inline-flex; padding: 7px 12px; border: 1px solid var(--color-border); border-radius: 999px; color: var(--color-muted); background: transparent; background: color-mix(in srgb, var(--color-primary) 8%, transparent); font-size: 12px; letter-spacing: 0.08em; text-transform: uppercase; } #note-card-c008 h2 { margin: 14px 0 0; font-size: clamp(28px, 4.4vw, 40px); line-height: 1.1; letter-spacing: -0.02em; } #note-card-c008 .c008-grid { display: grid; grid-template-columns: repeat(3, minmax(0, 1fr)); gap: 14px; margin-top: 22px; } #note-card-c008 .c008-panel { padding: 16px; border: 1px solid var(--color-border); border-radius: 18px; background: transparent; background: color-mix(in srgb, var(--color-text) 3%, transparent); min-width: 0; } #note-card-c008 .c008-panel h3 { margin: 0 0 10px; font-size: 18px; } #note-card-c008 p { margin: 8px 0 0; } #note-card-c008 .c008-math { margin-top: 10px; padding: 12px; border: 1px solid var(--color-border); border-radius: 12px; overflow-x: auto; background: transparent; background: color-mix(in srgb, var(--color-bg) 60%, transparent); } #note-card-c008 .c008-guide { margin-top: 16px; padding: 18px; border: 1px solid var(--color-border); border-left: 3px solid var(--color-primary); border-radius: 18px; background: transparent; background: color-mix(in srgb, var(--color-primary) 7%, transparent); } #note-card-c008 ul { margin: 12px 0 0; padding-left: 20px; display: grid; gap: 9px; } @media (max-width: 920px) { #note-card-c008 .c008-grid { grid-template-columns: 1fr; } } Pattern Exam Patterns: How Maximum Variance Questions Appear Pattern A: Direct Maximum-Variance Evaluation You see an expression like where is the maximum-variance direction. Then the answer is Pattern B: Explained Variance You are asked how much total variance is retained by the first directions. Use If the question asks for the smallest that retains a given fraction, compute cumulative ratios until the threshold is crossed. Pattern C: Norm Connection For a symmetric positive semidefinite covariance matrix, So if the largest variance direction is linked to a spectral norm question, the needed value is still the largest eigenvalue. Quick Decision Guide If is the best direction, answer is . If is an arbitrary unit vector, evaluate . If all directions are involved, use . If retained information is asked, use a ratio of eigenvalue sums.

5 more cards in this chapter

Try a question

Answer it here to see how it works. Nothing is recorded until you sign in.

Question 1

Level 1: Warm-up

In Principal Component Analysis (PCA), the first principal component is defined as the direction that:

Question 2

Level 1: Warm-up

If v is an eigenvector of the covariance matrix C with corresponding eigenvalue λ, what is the variance of the data along the direction v (assuming v is normalized)?

Question 3

Level 1: Warm-up

In Principal Component Analysis, if a unit vector u is constrained to be orthogonal to the first principal component, what is the maximum possible variance of the data projected onto u ?

Question 4

Level 1: Warm-up

Consider the following assertion and reason: Assertion (A): If u = [34] and M = u u^{T}, the sum of the singular values of M is 25. Reason (R): The only non-zero singular value of u u^{T} is constructed by taking the squared Euclidean norm of u .

Question 5

Level 1: Warm-up

Consider the following assertion and reason: Assertion (A): If u = 221 and M = u u^{T}, the sum of the singular values of M is 3 . Reason (R): The only non-zero singular value of u u^{T} equals ∥ u ∥^{2} .

Question 6

Level 1: Warm-up

Let C be the covariance matrix of a dataset, and let w be a unit vector. Which expression represents the variance of the data when projected onto the direction w ?

Question 7

Level 1: Warm-up

A centered dataset in R^{d} has a covariance matrix with eigenvalues given by λ_{i} = 80 \cdot (1/2)^{i} for i = 1, 2, \dots, d . What is the maximum possible variance of the data when projected onto any single unit direction?

Question 8

Level 1: Warm-up

To find the direction u that maximizes the variance u^{T} C u, we must optimize this quadratic form subject to which constraint?

Question 9

Level 1: Warm-up

A dataset has a covariance matrix with eigenvalues 50, 30, and 20 . What proportion of the total variance is captured by the first two principal components?

Question 10

Level 1: Warm-up

Let a and b be vectors in R^{3} such that ∥ a ∥ \leq 2 and ∥ b ∥ \leq 3 . What is the maximum possible value of the largest singular value of the matrix A = a b^{T} ?

Free preview ends here

Login to view the complete short notes

Creating an account is free. You get the rest of this chapter, step-by-step solutions, and a study plan built around the topics you are actually weak at.

Why MastersUp

Personalised first. High quality throughout.

Most platforms hand everyone the same content. Here the content moves with your performance, topic by topic.

Built around you, not around a syllabus PDF

Every answer you give moves your topic-level intelligence rate. The next question, the next revision card and tomorrow's plan all change with it.

Revision that hits your weak spots

We only revise topics you have actually attempted and are still below the safe bar on — never the same chapter on repeat.

Questions calibrated to the real exam

Each question carries a measured toughness. You are served a rung above your current level, so practice keeps stretching you.

Notes written for recall, not for volume

Full lesson cards for first study, curated short-note cards for the last mile — with derivations, traps and exam patterns marked.

One place for everything

Notes, chapter practice, previous-year questions, test series and full-length papers — all feeding one picture of your preparation.

Honest progress

No vanity streaks. Progress here means chapters mastered and accuracy that held up on harder questions.

Unlock the whole course

Full notes and short notes, the complete question bank with worked solutions, mock tests, full-length papers, and an adaptive plan that rebuilds itself as you improve.

View all plans Start free

Singular Value Decomposition and Principal Component Analysis Short Notes for GATE DA

Singular Value Decomposition and Principal Component Analysis short notes for GATE DA: 8 study cards covering concepts, formulas, shortcuts and exam traps, plus solved practice questions.

Worked Example: Eigenvalues Given by a Decreasing Formula

Worked Example

Worked Example: Eigenvalues Given by a Decreasing Formula

Given

A centered dataset in $R^{100}$ has covariance matrix $Σ$ .

The eigenvalues of $Σ$ are

λ_{i} = 100 \cdot 2^{- i}, 1 \leq i \leq 100

Let $u$ be the unit direction of maximum variance.

We need

\frac{1}{n} i = 1 \sum n (u^{T} x^{(i)})^{2}

Recognize the Expression

For centered data,

\frac{1}{n} i = 1 \sum n (u^{T} x^{(i)})^{2} = u^{T} Σ u

Use the Maximum-Variance Direction

Since $u$ is the direction of maximum variance,

u^{T} Σ u = λ_{m a x}

Find the Largest Eigenvalue

The sequence

λ_{i} = 100 \cdot 2^{- i}

decreases as $i$ increases. Therefore, the maximum occurs at $i = 1$ .

λ_{m a x} = λ_{1} = 100 \cdot 2^{- 1}

λ_{m a x} = 100 \cdot \frac{1}{2} = 50

Final Answer

50

Trap: Total Variance Versus Maximum Directional Variance

Trap

Trap: Total Variance Versus Maximum Directional Variance

Trap 1: Sum of Eigenvalues Is Not the Maximum

Quantity	Formula	Meaning
Total variance	$tr (Σ) = \sum_{i} λ_{i}$	Variance spread over all directions
Maximum directional variance	$λ_{m a x}$	Variance along the best single direction
Explained variance ratio	$λ_{k} / \sum_{i} λ_{i}$	Fraction of total variance in direction $k$

If the problem asks for variance along the best direction, use only $λ_{m a x}$ .

If the problem asks for total variance, use the trace.

Trap 2: Forgetting Centering

If the mean is $\overset{x}{ˉ}$ and

Σ = \frac{1}{n} i = 1 \sum n (x^{(i)} - \overset{x}{ˉ}) (x^{(i)} - \overset{x}{ˉ})^{T}

then

\frac{1}{n} i = 1 \sum n (u^{T} x^{(i)})^{2} = u^{T} Σ u + (u^{T} \overset{x}{ˉ})^{2}

The extra term disappears only when $\overset{x}{ˉ} = 0$ .

Safety Check

Data is centered.
$u$ is a unit vector.
$u$ is the maximum-variance direction.
The matrix involved is the covariance matrix.

Exam Patterns: How Maximum Variance Questions Appear

Pattern

Exam Patterns: How Maximum Variance Questions Appear

Pattern A: Direct Maximum-Variance Evaluation

You see an expression like

\frac{1}{n} i = 1 \sum n (u^{T} x^{(i)})^{2}

where $u$ is the maximum-variance direction.

Then the answer is

λ_{m a x}

Pattern B: Explained Variance

You are asked how much total variance is retained by the first $k$ directions.

Use

\frac{\sum _{j = 1}^{k} λ _{j}}{\sum _{j = 1}^{d} λ _{j}}

If the question asks for the smallest $k$ that retains a given fraction, compute cumulative ratios until the threshold is crossed.

Pattern C: Norm Connection

For a symmetric positive semidefinite covariance matrix,

∥Σ ∥_{2} = λ_{m a x}

So if the largest variance direction is linked to a spectral norm question, the needed value is still the largest eigenvalue.

Quick Decision Guide

If $u$ is the best direction, answer is $λ_{m a x}$ .
If $u$ is an arbitrary unit vector, evaluate $u^{T} Σ u$ .
If all directions are involved, use $tr (Σ)$ .
If retained information is asked, use a ratio of eigenvalue sums.

Topic Summary: PCA Eigenvalues and Maximum Variance

Summary

Topic Summary: PCA Eigenvalues and Maximum Variance

Core Identities

For centered data matrix $X$ with observations in rows:

Σ = \frac{1}{n} X^{T} X

Projected variance along unit vector $u$ :

u^{T} Σ u

If $u$ is an eigenvector:

Σ u = λ u

then

u^{T} Σ u = λ

Maximum variance direction:

u^{T} u = 1 max u^{T} Σ u = λ_{m a x}

Total variance:

tr (Σ) = i \sum λ_{i}

Conditions to Remember

Data should be centered for the clean projected-variance identity.
The direction vector should have unit norm.
The covariance matrix is symmetric and positive semidefinite.
Its eigenvalues are real and nonnegative.

Memory Hook

Eigenvalue equals variance along the corresponding eigenvector.

Largest eigenvalue equals maximum variance in one direction.

Singular Value Decomposition and Principal Component Analysis: Solved Questions with Step-by-Step Explanations (10 Problems)

Question 1 · Linear Algebra MCQ

In Principal Component Analysis (PCA), the first principal component is defined as the direction that:

A.
Minimizes the reconstruction error of the data
B.
Maximizes the variance of the projected data
C.
Minimizes the trace of the covariance matrix
D.
Maximizes the determinant of the covariance matrix

Correct Answer:

Step-by-Step Solution

Key idea: This is a definition-based question about the primary objective of PCA.

Step 1: Recall that PCA seeks to find orthogonal directions (principal components) that capture the most information in the data.

Step 2: Information in this context is measured by variance. The first principal component is specifically the unit vector $u$ that maximizes the variance of the data when projected onto it.

Answer: Maximizes the variance of the projected data.

Question 2 · Linear Algebra MCQ

If $v$ is an eigenvector of the covariance matrix $C$ with corresponding eigenvalue $λ$ , what is the variance of the data along the direction $v$ (assuming $v$ is normalized)?

A.
$λ^{2}$
B.
$λ$
C.
$λ$
D.
$1/ λ$

Correct Answer:

Step-by-Step Solution

Key idea: This question connects the spectral decomposition of the covariance matrix to statistical variance.

Step 1: The variance along a direction $v$ is $v^{T} C v$ .

Step 2: Since $v$ is an eigenvector, $C v = λ v$ .

Step 3: Substitute this into the variance formula: $v^{T} (λ v) = λ (v^{T} v)$ .

Step 4: Since $v$ is normalized, $v^{T} v = 1$ . Thus, the variance is $λ$ .

Answer: $λ$

Question 3 · Linear Algebra MCQ

In Principal Component Analysis, if a unit vector $u$ is constrained to be orthogonal to the first principal component, what is the maximum possible variance of the data projected onto $u$ ?

A.
The largest eigenvalue of the covariance matrix
B.
The sum of all eigenvalues
C.
The second largest eigenvalue of the covariance matrix
D.
Zero

Correct Answer:

Step-by-Step Solution

Key idea: This is a comparison question, recognizable because it asks for the maximum variance under an orthogonality constraint to the first PC.

Why it applies: The variational characterization of eigenvalues (Courant-Fischer theorem) states that the maximum variance in the subspace orthogonal to the first $k - 1$ principal components is the $k$ -th largest eigenvalue.

Step 1: The first principal component corresponds to the largest eigenvalue $λ_{1}$ .

Step 2: We are looking for the maximum variance in the subspace orthogonal to this first component.

Step 3: By definition of PCA, the direction that maximizes variance in this orthogonal subspace is the second principal component.

Step 4: The variance along the second principal component is the second largest eigenvalue, $λ_{2}$ .

Answer: The correct option is C.

Trap: Assuming the maximum variance is still the largest eigenvalue, ignoring the orthogonality constraint.

Question 4 · Linear Algebra MCQ

Consider the following assertion and reason:

Assertion (A): If $u = [34]$ and $M = u u^{T}$ , the sum of the singular values of $M$ is 25.

Reason (R): The only non-zero singular value of $u u^{T}$ is constructed by taking the squared Euclidean norm of $u$ .

A.
Both A and R are true and R is the correct explanation of A
B.
Both A and R are true but R is NOT the correct explanation of A
C.
A is true but R is false
D.
A is false but R is true

Correct Answer:

Step-by-Step Solution

Key idea: For a symmetric rank-one matrix $M = u u^{T}$ , the single non-zero singular value is $∥ u ∥^{2}$ . The sum of all singular values is just this single value.

Step 1: Evaluate Reason (R).

The non-zero singular value of $u u^{T}$ is indeed $∥ u ∥^{2}$ . So, R is true.

Step 2: Evaluate Assertion (A).

Calculate $∥ u ∥^{2}$ for $u = [34]$ .

$∥ u ∥^{2} = 3^{2} + 4^{2} = 9 + 16 = 25$ .

Since there is only one non-zero singular value, the sum of the singular values is 25. So, A is true.

Step 3: Check the link.

R provides the exact formula needed to compute the value in A. Thus, R is the correct explanation for A.

Answer: Both A and R are true and R is the correct explanation of A

Question 5 · Linear Algebra MCQ

Consider the following assertion and reason:

Assertion (A): If $u = 221$ and $M = u u^{T}$ , the sum of the singular values of $M$ is $3$ .

Reason (R): The only non-zero singular value of $u u^{T}$ equals $∥ u ∥^{2}$ .

A.
Both A and R are true and R is the correct explanation of A
B.
Both A and R are true but R is NOT the correct explanation of A
C.
A is true but R is false
D.
A is false but R is true

Correct Answer:

Step-by-Step Solution

Key idea: This is an assertion-reason question on the special case $M = u u^{T}$ . The non-zero singular value is $∥ u ∥^{2}$ (the squared norm), not $∥ u ∥$ (the norm).

Exam route: Evaluate R first (it states a general fact), then evaluate A by computation.

Step 1: Evaluate Reason (R).

For $M = u u^{T}$ , the eigenvalues are $∥ u ∥^{2}, 0, 0, \dots, 0$ . Since $M$ is symmetric positive semidefinite, singular values equal eigenvalues. So the only non-zero singular value is $∥ u ∥^{2}$ . R is true.

Step 2: Evaluate Assertion (A).

Compute $∥ u ∥^{2}$ for $u = 221$ .

$∥ u ∥^{2} = 2^{2} + 2^{2} + 1^{2} = 4 + 4 + 1 = 9$ .

The sum of the singular values is $9 + 0 + 0 = 9$ .

Assertion A claims the sum is $3$ . This is false.

Step 3: Determine the relationship.

A is false, R is true.

Wrong path: If you compute $∥ u ∥ = 4 + 4 + 1 = 9 = 3$ and use that as the singular value, you would get the sum as $3$ and conclude A is true. This is the unit mismatch trap: confusing $∥ u ∥$ (length) with $∥ u ∥^{2}$ (squared length). The correct singular value for $u u^{T}$ is the squared length.

Generalization: For $u u^{T}$ , always use $∥ u ∥^{2}$ for the singular value, not $∥ u ∥$ . For a general outer product $a b^{T}$ , use $∥ a ∥ ∥ b ∥$ .

Verification: $M u = u (u^{T} u) = u ∥ u ∥^{2} = 9 u$ . So $u$ is an eigenvector with eigenvalue $9$ . Since $M$ is positive semidefinite, the singular value is $9$ . The sum of singular values is $9$ , not $3$ . Confirmed.

Answer: A is false but R is true

Question 6 · Linear Algebra MCQ

Let $C$ be the covariance matrix of a dataset, and let $w$ be a unit vector. Which expression represents the variance of the data when projected onto the direction $w$ ?

A.
$w^{T} C w$
B.
$w^{T} C^{- 1} w$
C.
$tr (C w)$
D.
$det (C)$

Correct Answer:

Step-by-Step Solution

Key idea: This question tests the algebraic formula for projected variance using the covariance matrix.

Step 1: The projection of a data point $x$ onto a unit vector $w$ is the scalar $w^{T} x$ .

Step 2: The variance of these projections is given by the quadratic form $w^{T} C w$ .

Answer: $w^{T} C w$

Question 7 · Linear Algebra MCQ

A centered dataset in $R^{d}$ has a covariance matrix with eigenvalues given by $λ_{i} = 80 \cdot (1/2)^{i}$ for $i = 1, 2, \dots, d$ . What is the maximum possible variance of the data when projected onto any single unit direction?

A.
80
B.
20
C.
40
D.
10

Correct Answer:

Step-by-Step Solution

Key idea: This is a sequence evaluation question, recognizable because it gives a formula for eigenvalues and asks for the maximum variance.

Why it applies: The maximum variance in any direction is the largest eigenvalue of the covariance matrix.

Step 1: The eigenvalues are given by $λ_{i} = 80 \cdot (1/2)^{i}$ for $i = 1, 2, \dots, d$ .

Step 2: Since $(1/2)^{i}$ is a decreasing sequence, the largest eigenvalue occurs at the smallest index, $i = 1$ .

Step 3: Calculate $λ_{1} = 80 \cdot (1/2)^{1} = 40$ .

Step 4: The maximum possible variance of the data when projected onto any single unit direction is exactly this largest eigenvalue.

Answer: The correct option is C.

Trap: Evaluating the sequence at $i = 0$ (which would give 80) or summing the series. The problem explicitly specifies $i \geq 1$ .

Question 8 · Linear Algebra MCQ

To find the direction $u$ that maximizes the variance $u^{T} C u$ , we must optimize this quadratic form subject to which constraint?

A.
$u^{T} C u = 1$
B.
$det (C) = 1$
C.
$u^{T} u = 1$
D.
$tr (C) = 1$

Correct Answer:

Step-by-Step Solution

Key idea: This is an optimization constraint question, recognizable because it asks for the condition under which the variance is maximized.

Why it applies: Maximizing a quadratic form requires a constraint, otherwise the variance could be made arbitrarily large by scaling the vector.

Step 1: We want to maximize the variance $u^{T} C u$ .

Step 2: If we scale $u$ by a constant $k$ , the variance becomes $k^{2} u^{T} C u$ , which can be infinitely large.

Step 3: To prevent this trivial scaling, we must restrict the length of $u$ .

Step 4: The standard constraint in PCA is that $u$ must be a unit vector, which is written as $u^{T} u = 1$ or $∥ u ∥^{2} = 1$ .

Answer: The correct option is C.

Trap: Confusing the objective function ( $u^{T} C u$ ) with the constraint. The constraint is strictly on the norm of the vector $u$ , not on the variance itself.

Question 9 · Linear Algebra NAT

A dataset has a covariance matrix with eigenvalues $50$ , $30$ , and $20$ . What proportion of the total variance is captured by the first two principal components?

Correct Answer:

0.8

Step-by-Step Solution

Key idea: This is a direct formula question, recognizable because it asks for the proportion of variance explained by a subset of principal components.

Why it applies: The proportion of variance explained by the first $k$ components is the sum of their eigenvalues divided by the sum of all eigenvalues (total variance).

Step 1: Identify the eigenvalues: $λ_{1} = 50$ , $λ_{2} = 30$ , $λ_{3} = 20$ .

Step 2: Calculate the total variance, which is the sum of all eigenvalues: $50 + 30 + 20 = 100$ .

Step 3: Calculate the variance captured by the first two principal components: $50 + 30 = 80$ .

Step 4: Compute the proportion: $80/100 = 0.8$ .

Answer: 0.8

Trap: Forgetting to divide by the total variance, or only taking the first eigenvalue.

Question 10 · Linear Algebra MCQ

Let $a$ and $b$ be vectors in $R^{3}$ such that $∥ a ∥ \leq 2$ and $∥ b ∥ \leq 3$ . What is the maximum possible value of the largest singular value of the matrix $A = a b^{T}$ ?

A.
$6$
B.
$5$
C.
$9$
D.
$36$

Correct Answer:

Step-by-Step Solution

Key idea: The largest (and only non-zero) singular value of $A = a b^{T}$ is $σ_{1} = ∥ a ∥∥ b ∥$ . To maximize this, we maximize the product of the norms.

Step 1: Identify the bounds on the norms.

$∥ a ∥ \leq 2$ and $∥ b ∥ \leq 3$ .

Step 2: Maximize the product.

The maximum value of $∥ a ∥∥ b ∥$ occurs when both norms are at their maximum possible values.

$max (σ_{1}) = 2 \times 3 = 6$ .

Answer: 6

singular value decomposition and principal component analysis short notes

Worked Example: Eigenvalues Given by a Decreasing Formula

Worked Example: Eigenvalues Given by a Decreasing Formula

Given

Recognize the Expression

Use the Maximum-Variance Direction

Find the Largest Eigenvalue

Final Answer

Trap: Total Variance Versus Maximum Directional Variance

Trap: Total Variance Versus Maximum Directional Variance

Trap 1: Sum of Eigenvalues Is Not the Maximum

Trap 2: Forgetting Centering

Safety Check

Exam Patterns: How Maximum Variance Questions Appear

Exam Patterns: How Maximum Variance Questions Appear

Pattern A: Direct Maximum-Variance Evaluation

Pattern B: Explained Variance

Pattern C: Norm Connection

Quick Decision Guide

Try a question

Login to view the complete short notes

Personalised first. High quality throughout.

Built around you, not around a syllabus PDF

Revision that hits your weak spots

Questions calibrated to the real exam

Notes written for recall, not for volume

One place for everything

Honest progress

Unlock the whole course

Singular Value Decomposition and Principal Component Analysis Short Notes for GATE DA

Worked Example: Eigenvalues Given by a Decreasing Formula

Worked Example: Eigenvalues Given by a Decreasing Formula

Given

Recognize the Expression

Use the Maximum-Variance Direction

Find the Largest Eigenvalue

Final Answer

Trap: Total Variance Versus Maximum Directional Variance

Trap: Total Variance Versus Maximum Directional Variance

Trap 1: Sum of Eigenvalues Is Not the Maximum

Trap 2: Forgetting Centering

Safety Check

Exam Patterns: How Maximum Variance Questions Appear

Exam Patterns: How Maximum Variance Questions Appear

Pattern A: Direct Maximum-Variance Evaluation

Pattern B: Explained Variance

Pattern C: Norm Connection

Quick Decision Guide

Topic Summary: PCA Eigenvalues and Maximum Variance

Topic Summary: PCA Eigenvalues and Maximum Variance

Core Identities

Conditions to Remember

Memory Hook

Singular Value Decomposition and Principal Component Analysis: Solved Questions with Step-by-Step Explanations (10 Problems)

Step-by-Step Solution

Step-by-Step Solution

Step-by-Step Solution

Step-by-Step Solution

Step-by-Step Solution

Step-by-Step Solution

Step-by-Step Solution

Step-by-Step Solution

Step-by-Step Solution

Step-by-Step Solution

More short notes in this unit