Singular Value Decomposition and Principal Component Analysis Notes for GATE DA: Concepts, Formulas, Worked Examples & Practice

Singular Value Decomposition and Principal Component Analysis notes for GATE DA: 19 study cards covering concepts, formulas, shortcuts and exam traps, plus solved practice questions.

Chapter Roadmap: SVD and PCA

Orientation

Chapter Roadmap: SVD and PCA

Matrix Decompositions, with the current stop on PCA eigenvalues and the direction of maximum variance.

Current topic: PCA Eigenvalues and Maximum Variance Direction

Core idea: covariance eigenvalues measure variance along principal directions.

Goal: identify the direction that captures maximum variance.

Exam focus: direct use of the largest eigenvalue and the quadratic form.

Next topic: Rank-One Matrices and Singular Values

Core idea: outer products create simple matrices with clean spectral structure.

Goal: connect singular values to rank-one constructions.

Exam focus: singular values of structured matrices.

The Goal: Finding the Direction of Maximum Spread

Concept

The Goal: Finding the Direction of Maximum Spread

Geometric Meaning

A centered data cloud can be spread differently along different directions. Principal Component Analysis asks:

Which unit direction keeps the largest spread when all points are projected onto it?

That direction is called the first principal component.

Algebraic Translation

If $Σ$ is the covariance matrix of centered data, then the best direction is an eigenvector of $Σ$ .

Σ v = λ v

The direction with the largest variance is the eigenvector corresponding to

λ_{m a x}

Key Mapping

Geometry	Linear Algebra
Direction of projection	Unit vector $u$
Spread along that direction	$u^{T} Σ u$
Best direction	Eigenvector for $λ_{m a x}$
Variance kept there	$λ_{m a x}$

Variance Along a Direction as a Quadratic Form

Formula

Variance Along a Direction as a Quadratic Form

Setup

Let

D = {x^{(1)}, x^{(2)}, \dots, x^{(n)}}

where each $x^{(i)} \in R^{d}$ and the data is centered:

i = 1 \sum n x^{(i)} = 0

Let $u$ be a unit vector:

u^{T} u = 1

Projection and Variance

The projection of $x^{(i)}$ onto $u$ is

u^{T} x^{(i)}

Since the data is centered, the variance of the projected values is

\frac{1}{n} i = 1 \sum n (u^{T} x^{(i)})^{2}

Quadratic Form Identity

\frac{1}{n} i = 1 \sum n (u^{T} x^{(i)})^{2} = u^{T} (\frac{1}{n} i = 1 \sum n x^{(i)} (x^{(i)})^{T}) u

The middle matrix is the covariance matrix:

Σ = \frac{1}{n} i = 1 \sum n x^{(i)} (x^{(i)})^{T}

Therefore,

Projected variance along u = u^{T} Σ u

Optimization Form

u^{T} u = 1 max u^{T} Σ u

The maximum value is the largest eigenvalue of $Σ$ .

Why Eigenvalues Equal Variance

Concept

Why Eigenvalues Equal Variance

Starting Point

For a covariance matrix $Σ$ , suppose $u$ is an eigenvector:

Σ u = λ u

Multiply by the Transpose

Left-multiply by $u^{T}$ :

u^{T} Σ u = λ u^{T} u

Use Unit Length

Since $u$ is a unit vector,

u^{T} u = 1

Hence,

u^{T} Σ u = λ

Interpretation

The expression $u^{T} Σ u$ is the variance of data projected onto $u$ .

If $u$ is an eigenvector, projected variance equals its eigenvalue.
If $u$ is the eigenvector for $λ_{m a x}$ , projected variance is maximized.
Sorting eigenvalues sorts directions by captured variance.

λ_{1} \geq λ_{2} \geq \dots \geq λ_{d}

The first eigenvector gives the direction of maximum variance.

Core Result

u^{T} u = 1 max u^{T} Σ u = λ_{m a x}

Singular Value Decomposition and Principal Component Analysis: Solved Questions with Step-by-Step Explanations (2 Problems)

Question 1 · Linear Algebra MCQ

In Principal Component Analysis (PCA), the first principal component is defined as the direction that:

A.
Minimizes the reconstruction error of the data
B.
Maximizes the variance of the projected data
C.
Minimizes the trace of the covariance matrix
D.
Maximizes the determinant of the covariance matrix

Correct Answer:

Step-by-Step Solution

Key idea: This is a definition-based question about the primary objective of PCA.

Step 1: Recall that PCA seeks to find orthogonal directions (principal components) that capture the most information in the data.

Step 2: Information in this context is measured by variance. The first principal component is specifically the unit vector $u$ that maximizes the variance of the data when projected onto it.

Answer: Maximizes the variance of the projected data.

Question 2 · Linear Algebra MCQ

If $v$ is an eigenvector of the covariance matrix $C$ with corresponding eigenvalue $λ$ , what is the variance of the data along the direction $v$ (assuming $v$ is normalized)?

A.
$λ^{2}$
B.
$λ$
C.
$λ$
D.
$1/ λ$

Correct Answer:

Step-by-Step Solution

Key idea: This question connects the spectral decomposition of the covariance matrix to statistical variance.

Step 1: The variance along a direction $v$ is $v^{T} C v$ .

Step 2: Since $v$ is an eigenvector, $C v = λ v$ .

Step 3: Substitute this into the variance formula: $v^{T} (λ v) = λ (v^{T} v)$ .

Step 4: Since $v$ is normalized, $v^{T} v = 1$ . Thus, the variance is $λ$ .

Answer: $λ$

More notes in this unit

chapter

Singular Value Decomposition and Principal Component Analysis Notes for GATE DA: Concepts, Formulas, Worked Examples & Practice

Singular Value Decomposition and Principal Component Analysis notes for GATE DA: 19 study cards covering concepts, formulas, shortcuts and exam traps, plus so

A question from this chapter

Question 1

In Principal Component Analysis (PCA), the first principal component is defined as the direction that:

Question 2

If v is an eigenvector of the covariance matrix C with corresponding eigenvalue λ, what is the variance of the data along the direction v (assuming v is normalized)?

Free preview ends here

Login to view the complete notes

Creating an account is free. You get the rest of this chapter, step-by-step solutions, and a study plan built around the topics you are actually weak at.

Why MastersUp

Personalised first. High quality throughout.

Most platforms hand everyone the same content. Here the content moves with your performance, topic by topic.

Built around you, not around a syllabus PDF

Every answer you give moves your topic-level intelligence rate. The next question, the next revision card and tomorrow's plan all change with it.

Revision that hits your weak spots

We only revise topics you have actually attempted and are still below the safe bar on — never the same chapter on repeat.

Questions calibrated to the real exam

Each question carries a measured toughness. You are served a rung above your current level, so practice keeps stretching you.

Notes written for recall, not for volume

Full lesson cards for first study, curated short-note cards for the last mile — with derivations, traps and exam patterns marked.

One place for everything

Notes, chapter practice, previous-year questions, test series and full-length papers — all feeding one picture of your preparation.

Honest progress

No vanity streaks. Progress here means chapters mastered and accuracy that held up on harder questions.

Unlock the whole course

Full notes and short notes, the complete question bank with worked solutions, mock tests, full-length papers, and an adaptive plan that rebuilds itself as you improve.

View all plans Start free