Neural Networks and Activation Functions Notes for GATE DA: Concepts, Formulas, Worked Examples & Practice

Neural Networks and Activation Functions notes for GATE DA: 37 study cards covering concepts, formulas, shortcuts and exam traps, plus solved practice questions.

Chapter Roadmap: Neural Networks and Activation Functions

Chapter Roadmap

Neural Networks and Activation Functions

1. ReLU Activation Properties and Gradients (Current)

Definition and mechanics of the Rectified Linear Unit
Why it solves the vanishing gradient problem
The derivative and the "Dying ReLU" trap

2. ReLU Activation Properties and Gradients (Advanced)

Handling the limitations of standard ReLU
Variants like Leaky ReLU and their gradient flows

3. Neural Network Architecture, Parameters and Equivalence

Building blocks of multi-layer perceptrons
Exact methods for counting weights and biases

4. Neural Network Architecture, Parameters and Equivalence (Advanced)

Functional equivalence between different network topologies
Complex architectural reasoning for exam problems

What you will master

By the end of this chapter, you will be able to compute gradients through activation functions, diagnose training failures, and precisely calculate the parameter count of any given feed-forward network.

ReLU Activation Properties and Gradients

Neural Networks > ReLU Activation

ReLU Activation Properties and Gradients

ReLU is the engine that made training deep neural networks computationally feasible and stable, replacing older functions like Sigmoid and Tanh.

01 Piecewise mathematical definition

02 Mitigating vanishing gradients

03 Derivative and subgradient rules

04 Identifying the "Dying ReLU" trap

The ReLU Function Definition

The Rectified Linear Unit (ReLU)

The ReLU function is a piecewise linear function defined as:

R (x) = max (0, x)

This can be written explicitly as:

R (x) = {x 0 if x > 0 if x \leq 0

Key Characteristic

It is non-linear overall (allowing the network to learn complex patterns), but linear in the positive region, which makes gradient computation trivial.

Why ReLU? Solving Vanishing Gradients

Solving the Vanishing Gradient Problem

Older activation functions like Sigmoid ( $σ (x)$ ) and Tanh ( $tanh (x)$ ) saturate at extreme values. Their derivatives approach zero as the absolute value of $x$ grows large.

Sigmoid

ReLU

In deep networks, backpropagation multiplies these derivatives across many layers via the Chain Rule. If each layer contributes a small fraction, the gradient exponentially decays to zero, halting learning.

The ReLU Advantage

For $x > 0$ , the derivative of ReLU is exactly $1$ .

\frac{\partial R ( x )}{\partial x} = 1 for x > 0

This ensures that the gradient signal flows backward through active neurons without attenuation, enabling the training of very deep architectures.

More notes in this unit

chapter

Neural Networks and Activation Functions Notes for GATE DA: Concepts, Formulas, Worked Examples & Practice

Neural Networks and Activation Functions notes for GATE DA: 37 study cards covering concepts, formulas, shortcuts and exam traps, plus solved practice questio

Free preview ends here

Login to view the complete notes

Creating an account is free. You get the rest of this chapter, step-by-step solutions, and a study plan built around the topics you are actually weak at.

Why MastersUp

Personalised first. High quality throughout.

Most platforms hand everyone the same content. Here the content moves with your performance, topic by topic.

Built around you, not around a syllabus PDF

Every answer you give moves your topic-level intelligence rate. The next question, the next revision card and tomorrow's plan all change with it.

Revision that hits your weak spots

We only revise topics you have actually attempted and are still below the safe bar on — never the same chapter on repeat.

Questions calibrated to the real exam

Each question carries a measured toughness. You are served a rung above your current level, so practice keeps stretching you.

Notes written for recall, not for volume

Full lesson cards for first study, curated short-note cards for the last mile — with derivations, traps and exam patterns marked.

One place for everything

Notes, chapter practice, previous-year questions, test series and full-length papers — all feeding one picture of your preparation.

Honest progress

No vanity streaks. Progress here means chapters mastered and accuracy that held up on harder questions.

Unlock the whole course

Full notes and short notes, the complete question bank with worked solutions, mock tests, full-length papers, and an adaptive plan that rebuilds itself as you improve.

View all plans Start free