Singular Value Decomposition and Principal Component Analysis Short Notes for GATE DA: Concepts, Formulas, Worked Examples & Practice

Singular Value Decomposition and Principal Component Analysis short notes for GATE DA: 8 study cards covering concepts, formulas, shortcuts and exam traps, plus solved practice questions.

Worked Example: Eigenvalues Given by a Decreasing Formula

Worked Example

Worked Example: Eigenvalues Given by a Decreasing Formula

Given

A centered dataset in $R^{100}$ has covariance matrix $Σ$ .

The eigenvalues of $Σ$ are

λ_{i} = 100 \cdot 2^{- i}, 1 \leq i \leq 100

Let $u$ be the unit direction of maximum variance.

We need

\frac{1}{n} i = 1 \sum n (u^{T} x^{(i)})^{2}

Recognize the Expression

For centered data,

\frac{1}{n} i = 1 \sum n (u^{T} x^{(i)})^{2} = u^{T} Σ u

Use the Maximum-Variance Direction

Since $u$ is the direction of maximum variance,

u^{T} Σ u = λ_{m a x}

Find the Largest Eigenvalue

The sequence

λ_{i} = 100 \cdot 2^{- i}

decreases as $i$ increases. Therefore, the maximum occurs at $i = 1$ .

λ_{m a x} = λ_{1} = 100 \cdot 2^{- 1}

λ_{m a x} = 100 \cdot \frac{1}{2} = 50

Final Answer

50

Trap: Total Variance Versus Maximum Directional Variance

Trap

Trap: Total Variance Versus Maximum Directional Variance

Trap 1: Sum of Eigenvalues Is Not the Maximum

Quantity	Formula	Meaning
Total variance	$tr (Σ) = \sum_{i} λ_{i}$	Variance spread over all directions
Maximum directional variance	$λ_{m a x}$	Variance along the best single direction
Explained variance ratio	$λ_{k} / \sum_{i} λ_{i}$	Fraction of total variance in direction $k$

If the problem asks for variance along the best direction, use only $λ_{m a x}$ .

If the problem asks for total variance, use the trace.

Trap 2: Forgetting Centering

If the mean is $\overset{x}{ˉ}$ and

Σ = \frac{1}{n} i = 1 \sum n (x^{(i)} - \overset{x}{ˉ}) (x^{(i)} - \overset{x}{ˉ})^{T}

then

\frac{1}{n} i = 1 \sum n (u^{T} x^{(i)})^{2} = u^{T} Σ u + (u^{T} \overset{x}{ˉ})^{2}

The extra term disappears only when $\overset{x}{ˉ} = 0$ .

Safety Check

Data is centered.
$u$ is a unit vector.
$u$ is the maximum-variance direction.
The matrix involved is the covariance matrix.

Exam Patterns: How Maximum Variance Questions Appear

Pattern

Exam Patterns: How Maximum Variance Questions Appear

Pattern A: Direct Maximum-Variance Evaluation

You see an expression like

\frac{1}{n} i = 1 \sum n (u^{T} x^{(i)})^{2}

where $u$ is the maximum-variance direction.

Then the answer is

λ_{m a x}

Pattern B: Explained Variance

You are asked how much total variance is retained by the first $k$ directions.

Use

\frac{\sum _{j = 1}^{k} λ _{j}}{\sum _{j = 1}^{d} λ _{j}}

If the question asks for the smallest $k$ that retains a given fraction, compute cumulative ratios until the threshold is crossed.

Pattern C: Norm Connection

For a symmetric positive semidefinite covariance matrix,

∥Σ ∥_{2} = λ_{m a x}

So if the largest variance direction is linked to a spectral norm question, the needed value is still the largest eigenvalue.

Quick Decision Guide

If $u$ is the best direction, answer is $λ_{m a x}$ .
If $u$ is an arbitrary unit vector, evaluate $u^{T} Σ u$ .
If all directions are involved, use $tr (Σ)$ .
If retained information is asked, use a ratio of eigenvalue sums.

Topic Summary: PCA Eigenvalues and Maximum Variance

Summary

Topic Summary: PCA Eigenvalues and Maximum Variance

Core Identities

For centered data matrix $X$ with observations in rows:

Σ = \frac{1}{n} X^{T} X

Projected variance along unit vector $u$ :

u^{T} Σ u

If $u$ is an eigenvector:

Σ u = λ u

then

u^{T} Σ u = λ

Maximum variance direction:

u^{T} u = 1 max u^{T} Σ u = λ_{m a x}

Total variance:

tr (Σ) = i \sum λ_{i}

Conditions to Remember

Data should be centered for the clean projected-variance identity.
The direction vector should have unit norm.
The covariance matrix is symmetric and positive semidefinite.
Its eigenvalues are real and nonnegative.

Memory Hook

Eigenvalue equals variance along the corresponding eigenvector.

Largest eigenvalue equals maximum variance in one direction.

Singular Value Decomposition and Principal Component Analysis: Solved Questions with Step-by-Step Explanations (2 Problems)

Question 1 · Linear Algebra MCQ

In Principal Component Analysis (PCA), the first principal component is defined as the direction that:

A.
Minimizes the reconstruction error of the data
B.
Maximizes the variance of the projected data
C.
Minimizes the trace of the covariance matrix
D.
Maximizes the determinant of the covariance matrix

Correct Answer:

Step-by-Step Solution

Key idea: This is a definition-based question about the primary objective of PCA.

Step 1: Recall that PCA seeks to find orthogonal directions (principal components) that capture the most information in the data.

Step 2: Information in this context is measured by variance. The first principal component is specifically the unit vector $u$ that maximizes the variance of the data when projected onto it.

Answer: Maximizes the variance of the projected data.

Question 2 · Linear Algebra MCQ

If $v$ is an eigenvector of the covariance matrix $C$ with corresponding eigenvalue $λ$ , what is the variance of the data along the direction $v$ (assuming $v$ is normalized)?

A.
$λ^{2}$
B.
$λ$
C.
$λ$
D.
$1/ λ$

Correct Answer:

Step-by-Step Solution

Key idea: This question connects the spectral decomposition of the covariance matrix to statistical variance.

Step 1: The variance along a direction $v$ is $v^{T} C v$ .

Step 2: Since $v$ is an eigenvector, $C v = λ v$ .

Step 3: Substitute this into the variance formula: $v^{T} (λ v) = λ (v^{T} v)$ .

Step 4: Since $v$ is normalized, $v^{T} v = 1$ . Thus, the variance is $λ$ .

Answer: $λ$

More short notes in this unit

chapter

Singular Value Decomposition and Principal Component Analysis Short Notes for GATE DA: Concepts, Formulas, Worked Examples & Practice

Singular Value Decomposition and Principal Component Analysis short notes for GATE DA: 8 study cards covering concepts, formulas, shortcuts and exam traps, pl

A question from this chapter

Question 1

In Principal Component Analysis (PCA), the first principal component is defined as the direction that:

Question 2

If v is an eigenvector of the covariance matrix C with corresponding eigenvalue λ, what is the variance of the data along the direction v (assuming v is normalized)?

Free preview ends here

Login to view the complete short notes

Creating an account is free. You get the rest of this chapter, step-by-step solutions, and a study plan built around the topics you are actually weak at.

Why MastersUp

Personalised first. High quality throughout.

Most platforms hand everyone the same content. Here the content moves with your performance, topic by topic.

Built around you, not around a syllabus PDF

Every answer you give moves your topic-level intelligence rate. The next question, the next revision card and tomorrow's plan all change with it.

Revision that hits your weak spots

We only revise topics you have actually attempted and are still below the safe bar on — never the same chapter on repeat.

Questions calibrated to the real exam

Each question carries a measured toughness. You are served a rung above your current level, so practice keeps stretching you.

Notes written for recall, not for volume

Full lesson cards for first study, curated short-note cards for the last mile — with derivations, traps and exam patterns marked.

One place for everything

Notes, chapter practice, previous-year questions, test series and full-length papers — all feeding one picture of your preparation.

Honest progress

No vanity streaks. Progress here means chapters mastered and accuracy that held up on harder questions.

Unlock the whole course

Full notes and short notes, the complete question bank with worked solutions, mock tests, full-length papers, and an adaptive plan that rebuilds itself as you improve.

View all plans Start free