Neural Networks and Activation Functions Short Notes for GATE DA: Concepts, Formulas, Worked Examples & Practice

Neural Networks and Activation Functions short notes for GATE DA: 4 study cards covering concepts, formulas, shortcuts and exam traps, plus solved practice questions.

Summary: ReLU Quick Revision

Core ReLU Facts

Definition

R (x) = max (0, x)

Derivative

R^{'} (x) = 1

x > 0

, and

0

x < 0

At $x = 0$

Non-differentiable; practical subgradient is

0

1

Primary Advantage

Prevents vanishing gradients for positive activations; computationally efficient.

Primary Trap

"Dying ReLU" — neurons stuck in the negative region output zero and receive zero gradients, permanently halting their learning.

Quick Revision Checklist: ReLU

• Definition:

R (x) = max (0, x)

• Gradient:

1

for

x > 0

0

for

x < 0

. (Subgradient at

x = 0

• Pros: Computationally highly efficient, mitigates vanishing gradients for positive activations, induces sparsity.

• Cons: Dying ReLU problem, not strictly differentiable at

x = 0

• Action: Always apply the max function immediately after computing the weighted sum at each layer before passing to the next layer.

Quick Revision: Parameters and Equivalence

Concept	Rule / Formula
Layer Weights	$n_{in} \times n_{o u t}$
Layer Biases	$n_{o u t}$ (if used), else $0$
Total Parameters	$\sum (Weights + Biases)$
ReLU Equivalence	$R e LU (c \cdot z) = c \cdot R e LU (z)$ only if $c > 0$
Equivalence Strategy	Scale in by $c > 0$ , scale out by $1/ c$

Quick Revision Checklist: Architecture and Parameters

• Input Layer: Always has

0

parameters.

• Layer Transition Formula: From size

N

to size

M

, parameters =

(N \times M) + M

(if bias is included).

• Total Network Parameters: Sum of parameters across all adjacent layer transitions.

• Critical Check: Always verify if the problem states "no bias" or "bias is zero" before adding the

+ M

term.

• Equivalence Principle: Permuting neurons in a hidden layer (along with their corresponding weights) yields a functionally equivalent network.

More short notes in this unit

chapter

Neural Networks and Activation Functions Short Notes for GATE DA: Concepts, Formulas, Worked Examples & Practice

Neural Networks and Activation Functions short notes for GATE DA: 4 study cards covering concepts, formulas, shortcuts and exam traps, plus solved practice qu

Free preview ends here

Login to view the complete short notes

Creating an account is free. You get the rest of this chapter, step-by-step solutions, and a study plan built around the topics you are actually weak at.

Why MastersUp

Personalised first. High quality throughout.

Most platforms hand everyone the same content. Here the content moves with your performance, topic by topic.

Built around you, not around a syllabus PDF

Every answer you give moves your topic-level intelligence rate. The next question, the next revision card and tomorrow's plan all change with it.

Revision that hits your weak spots

We only revise topics you have actually attempted and are still below the safe bar on — never the same chapter on repeat.

Questions calibrated to the real exam

Each question carries a measured toughness. You are served a rung above your current level, so practice keeps stretching you.

Notes written for recall, not for volume

Full lesson cards for first study, curated short-note cards for the last mile — with derivations, traps and exam patterns marked.

One place for everything

Notes, chapter practice, previous-year questions, test series and full-length papers — all feeding one picture of your preparation.

Honest progress

No vanity streaks. Progress here means chapters mastered and accuracy that held up on harder questions.

Unlock the whole course

Full notes and short notes, the complete question bank with worked solutions, mock tests, full-length papers, and an adaptive plan that rebuilds itself as you improve.

View all plans Start free