Linear Classifiers, Discriminant Analysis and Margin-Based Methods Notes for GATE DA: Concepts, Formulas, Worked Examples & Practice

Linear Classifiers, Discriminant Analysis and Margin-Based Methods notes for GATE DA: 60 study cards covering concepts, formulas, shortcuts and exam traps, plus solved practice questions.

Chapter Roadmap: Linear Classifiers and Margin Methods

1. Fisher Discriminant and Distance-Based Classifiers

Intuition of class separation, scatter matrices, and optimal projection. Current Topic

2. Fisher Discriminant: Mathematical Optimization

Deriving the weight vector using generalized eigenvalue problems.

3. Linear Separability and Perceptron Updates

Understanding linearly separable data and mistake-driven weight updates.

4. Perceptron Algorithm and Convergence

Formal algorithm steps, learning rate, and convergence theorem.

5. Support Vector Machines: Margins and Vectors

Hard margin, soft margin, and geometric intuition of support vectors.

6. SVM Optimization and Practical Application

Primal and dual formulations, hinge loss, and kernel trick basics.

The Core Idea: Why Fisher Discriminant?

The Goal of Fisher Linear Discriminant (FLD)

When reducing dimensions for classification, maximizing total variance (like PCA) can be disastrous. The direction of maximum variance might be orthogonal to the direction that best separates the classes.

Fisher's insight was to frame dimensionality reduction as a supervised optimization problem. For a binary classification task, we seek a projection vector $w$ that maps $d$ -dimensional data $x$ to a scalar $y = w^{T} x$ , such that:

The projected class means are far apart (maximize between-class scatter).
The projected points within each class are tightly clustered (minimize within-class scatter).

This creates a linear decision boundary in the original space that is optimally tuned for separating the two classes.

The Scatter Matrices: $S_W$ and $S_B$

Let the dataset have two classes, $C_{1}$ and $C_{2}$ , with $N_{1}$ and $N_{2}$ samples respectively. Let $μ_{1}$ and $μ_{2}$ be the mean vectors of the two classes, and $μ$ be the overall mean.

1. Within-Class Scatter Matrix (

S_{W}

)

S_{W} = i = 1 \sum 2 x \in C_{i} \sum (x - μ_{i}) (x - μ_{i})^{T}

Measures the compactness of each class. Sometimes defined with a normalization factor like

\frac{1}{N _{i}}

, making it the average within-class covariance.

2. Between-Class Scatter Matrix (

S_{B}

)

S_{B} = (μ_{1} - μ_{2}) (μ_{1} - μ_{2})^{T}

Measures the separation between the class means. In the general multi-class case,

S_{B} = \sum_{i = 1}^{C} N_{i} (μ_{i} - μ) (μ_{i} - μ)^{T}

Both

S_{W}

and

S_{B}

are symmetric, positive semi-definite matrices of size

d \times d

Fisher's Criterion Function

When we project the data using $w$ , the scalar means become $\tilde{μ}_{i} = w^{T} μ_{i}$ , and the projected scatter matrices become $w^{T} S_{W} w$ and $w^{T} S_{B} w$ .

Fisher's Criterion

J (w) = \frac{Between-class scatter}{Within-class scatter} = \frac{w ^{T} S _{B} w}{w ^{T} S _{W} w}

This is a classic Rayleigh Quotient. Our objective is to find the optimal weight vector $w^{*}$ that maximizes $J (w)$ :

w^{*} = ar g w max \frac{w ^{T} S _{B} w}{w ^{T} S _{W} w}

More notes in this unit

chapter

Linear Classifiers, Discriminant Analysis and Margin-Based Methods Notes for GATE DA: Concepts, Formulas, Worked Examples & Practice

Linear Classifiers, Discriminant Analysis and Margin-Based Methods notes for GATE DA: 60 study cards covering concepts, formulas, shortcuts and exam traps, pl

Free preview ends here

Login to view the complete notes

Creating an account is free. You get the rest of this chapter, step-by-step solutions, and a study plan built around the topics you are actually weak at.

Why MastersUp

Personalised first. High quality throughout.

Most platforms hand everyone the same content. Here the content moves with your performance, topic by topic.

Built around you, not around a syllabus PDF

Every answer you give moves your topic-level intelligence rate. The next question, the next revision card and tomorrow's plan all change with it.

Revision that hits your weak spots

We only revise topics you have actually attempted and are still below the safe bar on — never the same chapter on repeat.

Questions calibrated to the real exam

Each question carries a measured toughness. You are served a rung above your current level, so practice keeps stretching you.

Notes written for recall, not for volume

Full lesson cards for first study, curated short-note cards for the last mile — with derivations, traps and exam patterns marked.

One place for everything

Notes, chapter practice, previous-year questions, test series and full-length papers — all feeding one picture of your preparation.

Honest progress

No vanity streaks. Progress here means chapters mastered and accuracy that held up on harder questions.

Unlock the whole course

Full notes and short notes, the complete question bank with worked solutions, mock tests, full-length papers, and an adaptive plan that rebuilds itself as you improve.

View all plans Start free