Neural Networks

Overview

In our preceding studies of machine learning, we have primarily concerned ourselves with models that assume a specific underlying structure in the data, such as linearity. We now advance to a class of models inspired by biological neural systems, which are capable of learning highly complex and non-linear relationships directly from data. Neural networks form the foundational basis of modern deep learning and represent a significant paradigm shift in how we approach problems of prediction and classification. Their power lies in their hierarchical structure, where simple computational units are organized into layers to learn progressively more abstract features.

This chapter is designed to provide a rigorous and principled introduction to the core concepts of neural networks, with a specific focus on the architectures most relevant to the GATE examination. A thorough command of these fundamentals is indispensable, as questions frequently test not only the conceptual understanding of network architecture but also the computational mechanics of information flow. We will systematically dissect the components of a neuron, the arrangement of these neurons into layers, and the mechanism by which these networks process input to produce an output. Our objective is to build a firm theoretical and practical foundation for tackling problems related to these powerful models.

We shall begin by examining the simplest of these architectures, the Feed-Forward Neural Network, to establish the core principles of network computation. Subsequently, we will extend this framework to the Multi-Layer Perceptron (MLP), introducing the concepts of hidden layers and non-linear activation functions. It is this extension that endows neural networks with the ability to approximate any continuous function, making them a universal tool for machine learning tasks.

---

Chapter Contents

| # | Topic | What You'll Learn |
|---|-------|-------------------|
| 1 | Feed-Forward Neural Network | The fundamental architecture and signal propagation. |
| 2 | Multi-Layer Perceptron (MLP) | Introducing hidden layers and non-linear activation. |

---

Learning Objectives

---

We now turn our attention to Feed-Forward Neural Network...
## Part 1: Feed-Forward Neural Network

Introduction

The Feed-Forward Neural Network (FFNN), also known as a Multi-Layer Perceptron (MLP), represents a foundational architecture in the study of neural networks. These networks are characterized by a unidirectional flow of information, where data moves from the input layer, through one or more hidden layers, to the output layer without forming any cycles. This acyclic graph structure distinguishes them from recurrent neural networks. FFNNs are universal function approximators, meaning that a sufficiently large network can approximate any continuous function to an arbitrary degree of accuracy.

In the context of the GATE examination, a firm understanding of FFNNs is paramount. This includes the mechanics of how an input signal is processed to produce an output, a process known as forward propagation, and the method by which the network's parameters (weights and biases) are optimized, which relies on calculating gradients via backpropagation. Questions frequently test the computational aspects of these processes, demanding both conceptual clarity and procedural accuracy. We shall explore the mathematical underpinnings of these networks, focusing on the principles necessary for solving competitive examination problems.

---

Key Concepts

#
## 1. The Artificial Neuron

The fundamental processing unit of a neural network is the artificial neuron, or node. It is a mathematical function conceived as a model of a biological neuron.

A neuron receives one or more inputs, computes their weighted sum, adds a bias term, and passes this result through an activation function. Let us consider a neuron that receives $n$ inputs denoted by the vector $\mathbf{x} = [x_1, x_2, \dots, x_n]^T$. Each input $x_i$ is associated with a weight $w_i$. The neuron also has a bias term, $b$.

First, we compute the net input, $z$, which is the affine transformation of the inputs:

Next, the net input $z$ is passed through a non-linear activation function, $\phi(z)$, to produce the neuron's output, $a$:

The bias term $b$ allows the activation function to be shifted to the left or right, which can be critical for successful learning. The weights $\mathbf{w}$ and bias $b$ are the learnable parameters of the neuron.

$x_1$

$w_1$

$x_2$

$w_2$

$x_n$

$w_n$
...

Σ

ϕ

$z$


bias ($b$)

$a = \phi(z)$

#
## 2. Activation Functions

The activation function introduces non-linearity into the network, enabling it to learn complex patterns that a purely linear model could not. Without non-linear activation functions, a deep neural network would be mathematically equivalent to a single-layer linear model.

#
### Rectified Linear Unit (ReLU)

The most commonly used activation function in modern neural networks is the Rectified Linear Unit, or ReLU.

A critical property for backpropagation is the derivative of the activation function. The derivative of ReLU is straightforward:

The derivative is undefined at $z=0$, but in practice, it is typically set to $0$ or $1$. For GATE problems, this discontinuity is rarely the focus; the key is that the gradient is $1$ for positive inputs and $0$ for negative inputs.

#
## 3. Forward Propagation

Forward propagation is the process of computing the output of the neural network, given a set of inputs and parameters (weights and biases). The calculation proceeds layer by layer, from the input layer to the output layer.

Let us denote the activation of neuron $j$ in layer $l$ as $a_j^{(l)}$. The net input to this neuron is $z_j^{(l)}$. The weight connecting neuron $k$ in layer $l-1$ to neuron $j$ in layer $l$ is $w_{jk}^{(l)}$, and the bias of neuron $j$ in layer $l$ is $b_j^{(l)}$.

The computation for a single neuron is:

This process is repeated for all neurons in a layer, and then for all subsequent layers until the final output is produced. For the input layer, the activations $a_k^{(0)}$ are simply the input features $x_k$.

Worked Example:

Problem: Consider a simple network with 2 input neurons, one hidden layer with 2 neurons, and one output neuron. All neurons use the ReLU activation function. The biases are all 0.

• Inputs: $x_1 = 1, x_2 = -2$.


• Weights from input to hidden layer: $w_{11}^{(1)} = 2, w_{12}^{(1)} = -1, w_{21}^{(1)} = 3, w_{22}^{(1)} = 1$.


• Weights from hidden to output layer: $w_{11}^{(2)} = 4, w_{12}^{(2)} = -3$.

Calculate the final output of the network.

Solution:

Let $h_1, h_2$ be the outputs of the two hidden neurons and $y$ be the final output.

Step 1: Calculate the net inputs to the hidden layer neurons ($z_1^{(1)}, z_2^{(1)}$).

Step 2: Apply the ReLU activation function to find the outputs of the hidden layer ($h_1, h_2$).

Step 3: Calculate the net input to the output neuron ($z_1^{(2)}$).

Step 4: Apply the ReLU activation function to find the final output ($y$).

Answer: The final output of the network is $13$.

---

#
## 4. Backpropagation and Gradient Calculation

Backpropagation is the algorithm used to train neural networks. It efficiently computes the gradient of the loss function with respect to the network's weights. At its core, backpropagation is a practical application of the chain rule from calculus. For GATE, questions often focus on finding the partial derivative of the output with respect to a specific weight.

Let us consider finding the derivative of the final output $y$ with respect to a weight $w_{ij}$ connecting neuron $i$ to neuron $j$. The key is to trace the influence of $w_{ij}$ on $y$. The weight $w_{ij}$ first affects the net input $z_j$ of neuron $j$, which in turn affects its activation $a_j$, which then propagates through the network to affect the final output $y$.

Using the chain rule, we can express this relationship:

Let's break down each term:

• $\frac{\partial z_j}{\partial w_{ij}}$: The net input $z_j = \sum_k w_{jk} a_k + b_j$. The derivative with respect to one specific weight $w_{ij}$ is simply the corresponding input activation $a_i$. So, $\frac{\partial z_j}{\partial w_{ij}} = a_i$.


• $\frac{\partial a_j}{\partial z_j}$: This is the derivative of the activation function of neuron $j$, $\phi'(z_j)$. For ReLU, this is either 1 or 0.


• $\frac{\partial y}{\partial a_j}$: This term represents how the activation of neuron $j$ affects the final output $y$. This itself might be a complex chain rule calculation, depending on the network's structure downstream from neuron $j$.

Worked Example:

Problem: Consider the network from PYQ 1. Let the top hidden neuron be $h_1$ and the bottom hidden neuron be $h_2$. The inputs are $u=2, v=3$, and the weights are $a=1, b=1, c=1, d=-1, e=4, f=-1$. The activation function is ReLU. Calculate $\frac{\partial y}{\partial a}$.

Solution:

First, let's write the equations for the network based on the diagram.

• Net input to $h_1$: $z_{h1} = a \cdot u + b \cdot v$


• Output of $h_1$: $h_1 = R(z_{h1})$


• Net input to $h_2$: $z_{h2} = c \cdot u + d \cdot v$


• Output of $h_2$: $h_2 = R(z_{h2})$


• Net input to output neuron: $z_y = e \cdot h_1 + f \cdot h_2$


• Final output: $y = R(z_y)$

We need to compute $\frac{\partial y}{\partial a}$. Using the chain rule, we trace the path from $y$ back to $a$: $y \leftarrow z_y \leftarrow h_1 \leftarrow z_{h1} \leftarrow a$.

Step 1: Perform forward propagation to find the values of all intermediate variables. This is crucial to determine the derivatives of the ReLU functions.

Step 2: Calculate each term in the chain rule expression.

• $\frac{\partial z_{h1}}{\partial a}$: Since $z_{h1} = a \cdot u + b \cdot v$, the derivative with respect to $a$ is $u$.

• $\frac{\partial h_1}{\partial z_{h1}}$: This is the derivative of the ReLU function at $z_{h1}$. Since $z_{h1} = 5 > 0$, the derivative is 1.

• $\frac{\partial z_y}{\partial h_1}$: Since $z_y = e \cdot h_1 + f \cdot h_2$, the derivative with respect to $h_1$ is $e$.

• $\frac{\partial y}{\partial z_y}$: This is the derivative of the ReLU function at $z_y$. Since $z_y = 20 > 0$, the derivative is 1.

Step 3: Multiply the terms together.

Answer: $\frac{\partial y}{\partial a} = 8$.

#
## 5. Network Equivalence and Simplification

Under certain conditions, a complex neural network can be mathematically equivalent to a simpler one. This is an important concept for understanding the expressive power of networks.

A key scenario arises with linear activation functions. If all neurons in a multi-layer network have a linear activation function, $\phi(z) = z$, the entire network collapses into a single linear transformation. The composition of linear functions is another linear function.

A more subtle case, as seen in GATE questions, involves the ReLU function when inputs are constrained. If the net input $z$ to a ReLU neuron is guaranteed to be positive, then $\text{ReLU}(z) = \max(0, z) = z$. In this specific domain, the ReLU function behaves identically to a linear (identity) function. This allows for the simplification of network layers.

Consider two consecutive layers (without bias for simplicity) with weight matrices $W_1$ and $W_2$. If the activation function $\phi$ is linear, the output is $y = \phi(W_2 \phi(W_1 x)) = W_2 (W_1 x) = (W_2 W_1) x$. The two layers are equivalent to a single layer with a weight matrix $W_{equiv} = W_2 W_1$.

---

Problem-Solving Strategies

---

Common Mistakes

---

Practice Questions

:::question type="NAT" question="A neural network has a single hidden layer with one neuron and one output neuron. The input is $x=3$. The weight from input to hidden neuron is $w_1 = 2$. The bias of the hidden neuron is $b_1 = -7$. The weight from the hidden neuron to the output neuron is $w_2 = 5$. The bias of the output neuron is $b_2 = -1$. Both neurons use the ReLU activation function. What is the final output of the network?" answer="0" hint="Perform a forward pass step-by-step. Calculate the output of the hidden neuron first, then use it as input for the output neuron." solution="
Step 1: Calculate the net input to the hidden neuron, $z_1$.

Step 2: Calculate the activation of the hidden neuron, $h_1$.

Step 3: Calculate the net input to the output neuron, $z_2$.

Step 4: Calculate the final output, $y$.

Result: The final output is 0.
"
:::

:::question type="MCQ" question="Consider a neuron with two inputs $x_1=2, x_2=1$ and weights $w_1=3, w_2=-4$. The bias is $b=1$. The activation function is ReLU. If the output of this neuron is $y$, what is the value of the partial derivative $\frac{\partial y}{\partial w_1}$?" options=["0", "1", "2", "3"] answer="2" hint="First, compute the net input $z$ and the output $y$. Then, apply the chain rule: $\frac{\partial y}{\partial w_1} = \frac{\partial y}{\partial z} \cdot \frac{\partial z}{\partial w_1}$. Remember that $\frac{\partial y}{\partial z}$ depends on whether $z$ is positive or negative." solution="
Step 1: Calculate the net input $z$.

Step 2: Calculate the output $y$.

Step 3: Set up the chain rule expression for $\frac{\partial y}{\partial w_1}$.

Step 4: Calculate the components of the chain rule.

The first component is the derivative of the activation function. Since $z=3 > 0$, the derivative of ReLU is 1.

The second component is the derivative of the net input with respect to $w_1$.

Step 5: Compute the final partial derivative.

Result: The value of the partial derivative is 2.
"
:::

:::question type="MSQ" question="Which of the following statements about a standard Feed-Forward Neural Network with ReLU activation in its hidden layers are correct?" options=["The network can model non-linear decision boundaries.", "The derivative of the activation function is constant for all non-zero inputs.", "If all weights and biases are positive, and all inputs are positive, the network behaves as a purely linear model.", "The output of any hidden neuron is always non-negative."] answer="A,C,D" hint="Analyze each property of ReLU and its implications for the network. Consider the definition, derivative, and behavior under specific input conditions." solution="

• A. The network can model non-linear decision boundaries. This is correct. The ReLU function is non-linear (specifically, piecewise linear), and stacking layers with non-linear activations allows the network to approximate complex, non-linear functions.

• B. The derivative of the activation function is constant for all non-zero inputs. This is incorrect. The derivative is $1$ for positive inputs ($z>0$) and $0$ for negative inputs ($z<0$). It is not constant for all non-zero inputs.

• C. If all weights and biases are positive, and all inputs are positive, the network behaves as a purely linear model. This is correct. If inputs, weights, and biases are all positive, the net input $z = \mathbf{w}^T\mathbf{x} + b$ at every neuron will also be positive. For any positive $z$, $\text{ReLU}(z)=z$. Thus, every activation function becomes an identity function, and the entire network collapses into a linear transformation.

• D. The output of any hidden neuron is always non-negative. This is correct. By definition, $\text{ReLU}(z) = \max(0, z)$, so the output is always greater than or equal to zero.

"
:::

:::question type="MCQ" question="A neural network layer is defined by the transformation $h = \text{ReLU}(Wx+b)$, where $W = \begin{pmatrix} 2 & 1 \\ -1 & 3 \end{pmatrix}$, $b = \begin{pmatrix} 1 \\ 1 \end{pmatrix}$, and input $x = \begin{pmatrix} 1 \\ -1 \end{pmatrix}$. What is the output vector $h$?" options=["$\begin{pmatrix} 2 \\ -3 \end{pmatrix}$", "$\begin{pmatrix} 2 \\ 0 \end{pmatrix}$", "$\begin{pmatrix} 1 \\ -4 \end{pmatrix}$", "$\begin{pmatrix} 0 \\ 5 \end{pmatrix}$"] answer="$\begin{pmatrix} 2 \\ 0 \end{pmatrix}$" hint="First, compute the matrix-vector product $Wx$, then add the bias vector $b$ to get the net input vector $z$. Finally, apply the ReLU function element-wise to $z$." solution="
Step 1: Compute the matrix-vector product $Wx$.

Step 2: Add the bias vector $b$ to get the net input vector $z$.

Step 3: Apply the ReLU function element-wise to the vector $z$.

Result: The output vector $h$ is $\begin{pmatrix} 2 \\ 0 \end{pmatrix}$.
"
:::

---

Summary

---

What's Next?

---

---

Part 2: Multi-Layer Perceptron (MLP)

Introduction

The Multi-Layer Perceptron (MLP) represents a foundational architecture in the field of artificial neural networks. While simpler models like the single-layer perceptron are limited to solving linearly separable problems, the MLP overcomes this fundamental limitation by incorporating one or more intermediate, or "hidden," layers between its input and output. This architectural enhancement grants the MLP the capacity to learn complex, non-linear relationships within data.

The true power of the MLP lies in its ability to serve as a universal function approximator. With a sufficient number of hidden neurons and appropriate non-linear activation functions, an MLP can approximate any continuous function to an arbitrary degree of accuracy. This makes it an exceptionally versatile tool for a wide range of supervised learning tasks, including classification and regression. In our study for the GATE examination, a thorough understanding of the MLP's structure, the forward propagation of signals, and the backpropagation algorithm for training is of paramount importance.

---

Key Concepts

#
## 1. From the Single Perceptron to the MLP

To appreciate the necessity of the MLP, we must first consider the limitations of its predecessor, the single-layer perceptron. A single perceptron computes a linear combination of its inputs and applies an activation function. For an input vector $x \in \mathbb{R}^d$, the output $y$ is given by:

Here, $w$ is the weight vector, $b$ is the bias, and $\phi$ is the activation function. If $\phi$ is a step function (like the sign function), the perceptron acts as a linear classifier, defining a hyperplane as its decision boundary.

The critical limitation is that such a model can only classify data that is linearly separable. A classic example of a problem that a single perceptron cannot solve is the XOR problem.

Linearly Separable (AND)





(1,1)
(0,0)

Non-Linearly Separable (XOR)




(1,1)
(0,0)
(1,0)
(0,1)

The MLP overcomes this by stacking layers of neurons. The outputs of one layer become the inputs to the next. This layered composition of non-linear functions allows the MLP to construct complex, non-linear decision boundaries.

Input Layer

x1

x2

xd

Hidden Layer

h1

h2

...

hH

Output Layer

y

#
## 2. Activation Functions

The choice of activation function is critical. If we were to use a linear activation function in the hidden layers, the entire MLP would collapse into an equivalent single-layer linear model, thereby losing its ability to model non-linearity. Therefore, we require non-linear activation functions.

Sigmoid (Logistic):
The sigmoid function maps any real-valued number into the range $(0, 1)$.

Hyperbolic Tangent (tanh):
The tanh function is similar to the sigmoid but maps inputs to the range $(-1, 1)$.

Rectified Linear Unit (ReLU):
The ReLU function is one of the most widely used activation functions in modern neural networks.

#
## 3. The Forward Pass

The forward pass is the process of computing the network's output for a given input vector $x$. We proceed layer by layer, from the input to the output.

Consider an MLP with one hidden layer.
Let:

• $X$ be the input vector.


• $W^{(1)}$ and $b^{(1)}$ be the weight matrix and bias vector for the hidden layer.


• $W^{(2)}$ and $b^{(2)}$ be the weight matrix and bias vector for the output layer.


• $\phi$ be the activation function.

The computation proceeds as follows:

Calculate the pre-activation for the hidden layer ($Z^{(1)}$):

Calculate the activation of the hidden layer ($A^{(1)}$):

Calculate the pre-activation for the output layer ($Z^{(2)}$):

Calculate the final output ($A^{(2)}$ or $\hat{y}$):

(Note: The output layer might use a different activation function, e.g., softmax for multi-class classification).

Worked Example:

Problem:
Consider a simple MLP with 2 input neurons, a hidden layer with 2 neurons, and 1 output neuron. The activation function for all neurons is ReLU. The weights and biases are given as:

$W^{(1)} = \begin{bmatrix} 0.5 & -1.0 \\ 0.8 & 0.2 \end{bmatrix}$, $b^{(1)} = \begin{bmatrix} 0.1 \\ -0.3 \end{bmatrix}$

$W^{(2)} = \begin{bmatrix} 0.7 & -0.4 \end{bmatrix}$, $b^{(2)} = [0.2]$

Calculate the output of the network for the input vector $X = \begin{bmatrix} 2 \\ 3 \end{bmatrix}$.

Solution:

Step 1: Calculate the pre-activation for the hidden layer, $Z^{(1)}$.

Step 2: Apply the ReLU activation function to get the hidden layer's output, $A^{(1)}$.

Step 3: Calculate the pre-activation for the output layer, $Z^{(2)}$.

Step 4: Apply the ReLU activation function to get the final output, $\hat{y}$.

Answer: The final output of the network is $0$.

#
## 4. Backpropagation and Gradient Descent

Training an MLP involves adjusting its weights and biases to minimize a loss function, which measures the discrepancy between the predicted outputs ($\hat{y}$) and the true target values ($y$). The most common algorithm for this is backpropagation combined with an optimization algorithm like gradient descent.

The foundational idea of gradient descent is to update the parameters (weights $w$) in the opposite direction of the gradient of the loss function $L$.

Here, $\eta$ is the learning rate, a hyperparameter that controls the step size.

Backpropagation is an efficient algorithm for computing these gradients, $\frac{\partial L}{\partial w}$, for all weights in the network. It works by applying the chain rule of calculus, starting from the output layer and moving backward through the network.

• First, the gradient of the loss with respect to the output layer's weights is computed.
• Then, this error is "propagated" backward to the previous layer. The gradient for the hidden layer's weights is calculated based on the error signal from the output layer.
• This process continues until the gradients for all weights have been computed.

This method is more complex than the simple update rule of the single-layer perceptron (which only applies to specific loss functions and models), but it is a general mechanism that allows for the training of deep, complex networks.

---

Problem-Solving Strategies

---

Common Mistakes

---

Practice Questions

:::question type="MCQ" question="An MLP has an input layer with 3 neurons, a single hidden layer with 4 neurons, and an output layer with 2 neurons. What are the dimensions of the weight matrix for the hidden layer ($W^{(1)}$) and the output layer ($W^{(2)}$) respectively?" options=["$W^{(1)}: 3 \times 4$, $W^{(2)}: 4 \times 2$","$W^{(1)}: 4 \times 3$, $W^{(2)}: 2 \times 4$","$W^{(1)}: 3 \times 4$, $W^{(2)}: 2 \times 4$","$W^{(1)}: 4 \times 3$, $W^{(2)}: 4 \times 2$"] answer="$W^{(1)}: 4 \times 3$, $W^{(2)}: 2 \times 4$" hint="The dimensions of a weight matrix $W$ connecting layer A to layer B are (number of neurons in B) x (number of neurons in A)." solution="Step 1: Analyze the connection from the input layer to the hidden layer.
The source layer (input) has 3 neurons.
The destination layer (hidden) has 4 neurons.
Therefore, the dimension of the weight matrix $W^{(1)}$ is (destination size) x (source size), which is $4 \times 3$.

Step 2: Analyze the connection from the hidden layer to the output layer.
The source layer (hidden) has 4 neurons.
The destination layer (output) has 2 neurons.
Therefore, the dimension of the weight matrix $W^{(2)}$ is (destination size) x (source size), which is $2 \times 4$.

Result: The dimensions are $W^{(1)}: 4 \times 3$ and $W^{(2)}: 2 \times 4$."
:::

:::question type="NAT" question="A neuron in a hidden layer uses the ReLU activation function. It receives inputs from two neurons with values $x_1 = -2$ and $x_2 = 5$. The corresponding weights are $w_1 = 1.5$ and $w_2 = 0.5$. The bias for this neuron is $b = -0.5$. What is the output of this neuron?" answer="-1.0" hint="Calculate the weighted sum plus bias, $z = w_1x_1 + w_2x_2 + b$, and then apply the ReLU function, $\max(0, z)$." solution="Step 1: Calculate the weighted sum of the inputs.

Step 2: Add the bias to get the pre-activation value, $z$.

Step 3: Apply the ReLU activation function to $z$.

Result: The output of the neuron is 0.
"
:::
:::question type="NAT" question="A neuron in a hidden layer uses the ReLU activation function. It receives inputs from two neurons with values $x_1 = 4$ and $x_2 = -1$. The corresponding weights are $w_1 = 0.5$ and $w_2 = -2$. The bias for this neuron is $b = 1.0$. Calculate the output of this neuron." answer="5.0" hint="Calculate the pre-activation $z = w_1x_1 + w_2x_2 + b$, and then apply the ReLU function, $\text{output} = \max(0, z)$." solution="Step 1: Calculate the weighted sum of inputs.

Step 2: Add the bias to get the pre-activation value $z$.

Step 3: Apply the ReLU activation function.

Result: The output of the neuron is 5.0."
:::

:::question type="MSQ" question="Which of the following statements about activation functions in MLPs are correct?" options=["The ReLU function is linear for all inputs $z < 0$.","The sigmoid function's output is always in the range $[0, 1]$.","The tanh function is zero-centered, meaning its range is symmetric around zero.","Using a linear activation function in all hidden layers allows the MLP to model complex non-linear data."] answer="The ReLU function is linear for all inputs $z < 0$.,The tanh function is zero-centered, meaning its range is symmetric around zero." hint="Evaluate the properties of each activation function mentioned. Recall the output ranges and shapes of their graphs." solution="Option A: The ReLU function is defined as $\max(0, z)$. For all $z < 0$, its output is a constant 0. A constant function is a form of a linear function ($y = 0x + 0$). Thus, this statement is correct.

Option B: The sigmoid function $\sigma(z) = 1 / (1 + e^{-z})$ outputs values in the range $(0, 1)$. It approaches 0 and 1 asymptotically but never strictly reaches them. Therefore, the range is $(0,1)$, not $[0,1]$. This statement is incorrect.

Option C: The tanh function outputs values in the range $(-1, 1)$. This range is symmetric around 0, making it zero-centered. This property can be beneficial for optimization. This statement is correct.

Option D: If all hidden layers use a linear activation function, the composition of these linear functions is itself a linear function. The entire network collapses to a single linear model and cannot learn non-linear patterns. This statement is incorrect.

Result: The correct statements are A and C."
:::

:::question type="MCQ" question="The primary motivation for using Multi-Layer Perceptrons over single-layer perceptrons is their ability to:" options=["Converge faster during training.","Solve non-linearly separable problems.","Require less memory.","Use a simpler weight update rule."] answer="Solve non-linearly separable problems." hint="Consider the fundamental limitation of a single-layer perceptron's decision boundary." solution="A single-layer perceptron can only form a linear decision boundary (a hyperplane). This means it can only solve problems where the classes are linearly separable. The introduction of hidden layers with non-linear activation functions in an MLP allows the model to learn complex, non-linear decision boundaries. This is the key advantage and primary reason for their development and use. While other aspects like convergence speed can vary, the core capability that distinguishes MLPs is their ability to handle non-linear separability."
:::

---

Summary

---

What's Next?

---

Chapter Summary

---

Chapter Review Questions

:::question type="MCQ" question="Consider a Multi-Layer Perceptron (MLP) with 10 neurons in the input layer, a single hidden layer with 8 neurons, and an output layer with 3 neurons for a multi-class classification problem. The hidden layer uses a ReLU activation function and the output layer uses a Softmax function. What is the total number of trainable parameters (weights and biases) in this network?" options=["104", "115", "117", "124"] answer="C" hint="Remember to account for both the weights connecting the layers and the bias term for each neuron in the hidden and output layers." solution="We calculate the parameters for each connection and layer sequentially.

Parameters between Input and Hidden Layer:

- The number of weights connecting the 10 input neurons to the 8 hidden neurons is $10 \times 8 = 80$.
- Each of the 8 neurons in the hidden layer has its own bias term. So, there are 8 biases.
- Total parameters for the hidden layer: $80 + 8 = 88$.

Parameters between Hidden and Output Layer:

- The number of weights connecting the 8 hidden neurons to the 3 output neurons is $8 \times 3 = 24$.
- Each of the 3 neurons in the output layer has its own bias term. So, there are 3 biases.
- Total parameters for the output layer: $24 + 3 = 27$.

Total Trainable Parameters:

- The total number of parameters in the network is the sum of the parameters calculated above.
-
-

Thus, the total number of trainable parameters is 117."
:::

:::question type="NAT" question="A neuron uses the Rectified Linear Unit (ReLU) activation function, defined as $f(z) = \max(0, z)$. The neuron receives two inputs, $x_1 = -3$ and $x_2 = 4$, with corresponding weights $w_1 = 0.5$ and $w_2 = 0.8$. The bias for this neuron is $b = -1.5$. Calculate the output of this neuron." answer="0.2" hint="First, compute the weighted sum of the inputs plus the bias, $z = w_1 x_1 + w_2 x_2 + b$. Then, apply the ReLU activation function to this sum." solution="The process involves two steps: calculating the net input $z$ and then applying the activation function.

Calculate the net input $z$:

The net input is the linear combination of inputs and weights, plus the bias.

Substituting the given values:

Apply the ReLU activation function:

The ReLU function is $f(z) = \max(0, z)$.

The output of the neuron is 0.2.
:::

:::question type="MCQ" question="What is the primary motivation for using the backpropagation algorithm in training neural networks?" options=["To implement a non-linear decision boundary", "To prevent the network from overfitting the training data", "To efficiently compute the gradient of the loss function with respect to the network weights", "To initialize the weights of the network in an optimal manner"] answer="C" hint="Think about the core challenge in using gradient descent for a complex, multi-layered function." solution="The core of training a neural network is to minimize a loss function $E$ by adjusting its weights and biases $w$. Gradient descent requires calculating the partial derivative of the loss function with respect to each weight, $\frac{\partial E}{\partial w}$.

• Option A is incorrect. Non-linear decision boundaries are achieved by using non-linear activation functions, not by the training algorithm itself.
• Option B is incorrect. Preventing overfitting is the role of regularization techniques (e.g., L2 regularization, dropout), not backpropagation.
• Option D is incorrect. Weight initialization is a separate, important step, but it is not the purpose of backpropagation.

• Option C is correct. For a deep network, the loss function is a highly complex, nested function of millions of parameters. Calculating the gradient for each parameter naively would be computationally intractable. Backpropagation is a dynamic programming approach that systematically applies the chain rule of calculus to compute these gradients in a single pass (from output to input), making the training of deep networks feasible. It is fundamentally an algorithm for efficient gradient computation.

:::

:::question type="NAT" question="In a neural network, a particular weight $w$ has a current value of $0.6$. During a training step, the gradient of the Mean Squared Error loss with respect to this weight is calculated to be $\frac{\partial E}{\partial w} = 1.5$. If the learning rate $\eta$ is set to $0.05$, calculate the updated value of the weight after one step of standard gradient descent." answer="0.525" hint="The standard gradient descent update rule is $w_{\text{new}} = w_{\text{old}} - \eta \frac{\partial E}{\partial w}$." solution="We apply the standard gradient descent update rule to find the new value of the weight.

The update rule is given by:

Here, we are given:

• The current weight, $w^{(t)} = 0.6$


• The learning rate, $\eta = 0.05$


• The gradient of the loss with respect to the weight, $\frac{\partial E}{\partial w} = 1.5$

Substituting these values into the formula:

The updated value of the weight after one step is 0.525.
:::

---

What's Next?