Optimization

Overview

In this chapter, we shall explore the principles of optimization, a cornerstone of applied mathematics and a critical tool in engineering and data analysis. At its core, optimization is the process of finding the best possible solution from a set of available alternatives, typically by maximizing or minimizing a specific function. We will build upon the foundational concepts of differential calculus to develop a systematic methodology for identifying these optimal values. The techniques presented herein are not merely abstract mathematical exercises; they form the theoretical basis for solving a wide array of practical problems.

A thorough understanding of optimization is indispensable for success in the GATE examination, particularly within the Data Analytics (DA) syllabus. Many complex problems in machine learning, such as training a model, are fundamentally optimization problems aimed at minimizing a loss or error function. Similarly, questions in engineering disciplines frequently require the determination of maximum efficiency, minimum cost, or optimal design parameters. This chapter will equip you with the analytical tools to translate such problems into a mathematical framework and to solve them rigorously, a skill that is frequently tested and highly valued.

We begin by examining the theory of maxima and minima for functions of a single variable. We will establish the necessary and sufficient conditions for identifying local and global extrema using the first and second derivatives. Subsequently, we will apply these theoretical principles to solve word problems and practical scenarios, focusing on the crucial steps of model formulation and interpretation of results.

---

Chapter Contents

| # | Topic | What You'll Learn |
|---|-------|-------------------|
| 1 | Maxima and Minima | Identifying extreme values using derivative tests. |
| 2 | Optimization Involving a Single Variable | Modeling and solving single-variable optimization problems. |

---

Learning Objectives

---

---

We now turn our attention to Maxima and Minima...

Part 1: Maxima and Minima

Introduction

The study of maxima and minima, a cornerstone of differential calculus, concerns itself with finding the largest and smallest values that a function can attain. This process, often referred to as optimization, is of paramount importance in numerous scientific and engineering disciplines, including data science, where we frequently seek to minimize error functions or maximize likelihood estimates. For the GATE examination, a firm grasp of the methods for identifying and classifying the extreme values of functions of a single variable is essential.

We shall develop a systematic approach to locating these points, known as extrema. Our investigation will begin with the concept of critical points, which are the primary candidates for local maxima and minima. Subsequently, we will introduce two powerful analytical tools—the First Derivative Test and the Second Derivative Test—to classify these critical points. We will also distinguish between local extrema, which represent peaks and valleys in the immediate vicinity of a point, and global extrema, which are the absolute highest and lowest values of a function over its entire domain or a specified interval.

---

Key Concepts

The foundation for finding extrema lies in identifying special points where the function's rate of change is zero or undefined. These are the points where the function might transition from increasing to decreasing, or vice versa.

1. Critical Points

A necessary condition for a differentiable function to have a local extremum at an interior point is that its derivative at that point must be zero. This leads us to the definition of a critical point.

Points where $f'(c) = 0$ are also known as stationary points, as the tangent to the curve at such points is horizontal. It is crucial to understand that while all local extrema (for differentiable functions) occur at critical points, not all critical points correspond to a local extremum. For instance, the function $f(x) = x^3$ has a critical point at $x=0$ since $f'(0)=0$, but this point is neither a local maximum nor a minimum.

2. First Derivative Test

The First Derivative Test provides a method for classifying critical points by observing the sign of the first derivative on either side of the critical point. The sign of $f'(x)$ indicates whether the function $f(x)$ is increasing ($f'(x) > 0$) or decreasing ($f'(x) < 0$).

Local Max
f'(x)>0
f'(x)<0

c

Local Min
f'(x)<0
f'(x)>0

c

Inflection
f'(x)>0
f'(x)>0

c

Let $c$ be a critical point of a continuous function $f$.

If $f'(x)$ changes sign from positive to negative at $c$, then $f$ has a local maximum at $c$.

If $f'(x)$ changes sign from negative to positive at $c$, then $f$ has a local minimum at $c$.

If $f'(x)$ does not change sign at $c$ (that is, $f'(x)$ is positive on both sides of $c$ or negative on both sides), then $f$ has no local extremum at $c$.

3. Second Derivative Test

While the First Derivative Test is robust, the Second Derivative Test is often computationally simpler, provided the second derivative is easy to calculate. It relates the concavity of the function at a stationary point to the nature of the extremum.

Worked Example:

Problem: Find and classify the local extrema of the function $f(x) = 2x^3 - 3x^2 - 12x + 5$.

Solution:

Step 1: Find the first derivative of the function.

Step 2: Find the critical points by setting the first derivative to zero.

The critical points are $x = 2$ and $x = -1$.

Step 3: Find the second derivative to apply the Second Derivative Test.

Step 4: Evaluate the second derivative at each critical point to classify them.

For $x = 2$:

Since $f''(2) = 18 > 0$, the function has a local minimum at $x = 2$.

For $x = -1$:

Since $f''(-1) = -18 < 0$, the function has a local maximum at $x = -1$.

Answer: The function has a local minimum at $x=2$ and a local maximum at $x=-1$.

4. Global Extrema on a Closed Interval

For a continuous function on a closed interval $[a, b]$, the Extreme Value Theorem guarantees the existence of a global maximum and a global minimum. These global extrema can occur either at the critical points within the interval or at the interval's endpoints.

Worked Example:

Problem: Find the global maximum and minimum values of $f(x) = x^3 - 6x^2 + 9x + 1$ on the interval $[0, 4]$.

Solution:

Step 1: Find the first derivative and the critical points.

Set $f'(x) = 0$:

The critical points are $x=1$ and $x=3$. Both lie within the interval $[0, 4]$.

Step 2: Evaluate the function at the critical points.

At $x=1$:

At $x=3$:

Step 3: Evaluate the function at the endpoints of the interval.

At $x=0$:

At $x=4$:

Step 4: Compare the values obtained.

The values are $\{f(1)=5, f(3)=1, f(0)=1, f(4)=5\}$.
The largest value is 5, and the smallest value is 1.

Answer: The global maximum value is 5 (occurring at $x=1$ and $x=4$), and the global minimum value is 1 (occurring at $x=0$ and $x=3$).

5. Convexity and its Implications

The sign of the second derivative provides information about the curvature of the function's graph. A function is convex (or concave up) where $f''(x) > 0$ and concave (or concave down) where $f''(x) < 0$. This has significant implications for optimization.

For example, the function $f(x) = x^2$ has $f''(x) = 2 > 0$ for all $x$. It is strictly convex and has a unique global minimum at $x=0$. The function $f(x) = e^x$ has $f''(x) = e^x > 0$ for all $x$. It is also strictly convex but has no critical points and thus no local or global minimum.

---

---

Problem-Solving Strategies

A structured approach is vital for solving optimization problems under exam conditions.

---

Common Mistakes

Awareness of common pitfalls can significantly improve accuracy.

---

---

Practice Questions

:::question type="MCQ" question="A twice-differentiable function $g(x)$ has a stationary point at $x=c$. If the Taylor series expansion of $g(x)$ around $x=c$ is given by $g(x) = 5 - \frac{1}{2}(x-c)^2 + O((x-c)^4)$, what can be concluded about the point $x=c$?" options=["Local minimum", "Local maximum", "Point of inflection", "Global minimum"] answer="Local maximum" hint="Relate the Taylor series coefficients to the derivatives of the function at the point $c$. Specifically, $g(c)$, $g'(c)$, and $g''(c)$." solution="
Step 1: Recall the general form of a Taylor series expansion of a function $g(x)$ around a point $c$:

Step 2: Compare the given expansion with the general form.
Given: $g(x) = 5 - \frac{1}{2}(x-c)^2 + O((x-c)^4)$

By comparing coefficients:

• The constant term is $g(c) = 5$.


• The coefficient of $(x-c)$ is $g'(c) = 0$. This confirms that $x=c$ is a stationary point.


• The coefficient of $(x-c)^2$ is $\frac{g''(c)}{2!} = -\frac{1}{2}$.

Step 3: Solve for the second derivative.

Step 4: Apply the Second Derivative Test.
We have $g'(c) = 0$ and $g''(c) = -1 < 0$.
According to the Second Derivative Test, when the first derivative is zero and the second derivative is negative, the function has a local maximum at that point.

Result: The point $x=c$ is a local maximum.
Answer: \boxed{\text{Local maximum}}
"
:::

:::question type="NAT" question="Consider the function $f(x) = x + \frac{16}{x}$ for $x > 0$. The local minimum value of the function is ______." answer="8" hint="Find the first derivative, set it to zero to find the critical point, and then use the second derivative test to confirm it's a minimum. Finally, calculate the function's value at that point." solution="
Step 1: Find the first derivative of $f(x)$.

Step 2: Find the critical points by setting $f'(x) = 0$.

Since the domain is $x > 0$, we take the positive root: $x=4$.

Step 3: Find the second derivative to classify the critical point.

Step 4: Evaluate the second derivative at the critical point $x=4$.

Since $f''(4) > 0$, the function has a local minimum at $x=4$.

Step 5: Calculate the local minimum value by evaluating $f(x)$ at $x=4$.

Result: The local minimum value of the function is 8.
Answer: \boxed{8}
"
:::

:::question type="MSQ" question="Let $f(x) = (x-1)^2 (x-3)^2$. Which of the following statements is/are TRUE?" options=["The function has a local maximum at $x=2$","The function has local minima at $x=1$ and $x=3$","The function has three distinct stationary points","The value of the local maximum is 1"] answer="The function has a local maximum at $x=2$,The function has local minima at $x=1$ and $x=3$,The function has three distinct stationary points,The value of the local maximum is 1" hint="Use the product rule to find the derivative $f'(x)$. Factor the derivative to find all stationary points. Then use the second derivative test or the first derivative test to classify each point." solution="
Step 1: Find the first derivative $f'(x)$ using the product rule.

Let's expand $f(x)$ first for easier differentiation:

Step 2: Find the stationary points by setting $f'(x)=0$.

The stationary points are $x=1, x=2, x=3$. Thus, the function has three distinct stationary points.

Step 3: Find the second derivative $f''(x)$ to classify the points.

Step 4: Evaluate $f''(x)$ at each stationary point.

• At $x=1$: $f''(1) = 4(3(1)^2 - 12(1) + 11) = 4(3 - 12 + 11) = 4(2) = 8 > 0$. This is a local minimum.


• At $x=2$: $f''(2) = 4(3(2)^2 - 12(2) + 11) = 4(12 - 24 + 11) = 4(-1) = -4 < 0$. This is a local maximum.


• At $x=3$: $f''(3) = 4(3(3)^2 - 12(3) + 11) = 4(27 - 36 + 11) = 4(2) = 8 > 0$. This is a local minimum.

Step 5: Evaluate the function at the extrema to check the values.

• At $x=1$: $f(1) = (1-1)^2(1-3)^2 = 0$. This is a local minimum value.


• At $x=3$: $f(3) = (3-1)^2(3-3)^2 = 0$. This is a local minimum value.


• At $x=2$: $f(2) = (2-1)^2(2-3)^2 = (1)^2(-1)^2 = 1$. This is the local maximum value.

Conclusion:

• The function has a local maximum at $x=2$. (TRUE)


• The function has local minima at $x=1$ and $x=3$. (TRUE)


• The function has three distinct stationary points ($1, 2, 3$). (TRUE)


• The value of the local maximum is $f(2) = 1$. (TRUE)

All four statements are correct.
Answer: \boxed{\text{All four statements are correct.}}
"
:::

:::question type="NAT" question="The absolute maximum value of the function $f(x) = 2x^3 - 15x^2 + 36x + 1$ on the interval $[1, 5]$ is ______." answer="56" hint="Follow the Closed Interval Method. Find critical points within the interval, then evaluate the function at these points and at the endpoints $x=1$ and $x=5$. The largest value is the answer." solution="
Step 1: Find the first derivative of $f(x)$.

Step 2: Find the critical points by setting $f'(x) = 0$.

The critical points are $x=2$ and $x=3$. Both of these points lie within the interval $[1, 5]$.

Step 3: Evaluate the function at the critical points.

• At $x=2$: $f(2) = 2(2)^3 - 15(2)^2 + 36(2) + 1 = 16 - 60 + 72 + 1 = 29$.


• At $x=3$: $f(3) = 2(3)^3 - 15(3)^2 + 36(3) + 1 = 54 - 135 + 108 + 1 = 28$.

Step 4: Evaluate the function at the endpoints of the interval $[1, 5]$.

• At $x=1$: $f(1) = 2(1)^3 - 15(1)^2 + 36(1) + 1 = 2 - 15 + 36 + 1 = 24$.


• At $x=5$: $f(5) = 2(5)^3 - 15(5)^2 + 36(5) + 1 = 250 - 375 + 180 + 1 = 56$.

Step 5: Compare all the calculated values: $\{29, 28, 24, 56\}$.
The largest value is 56.

Result: The absolute maximum value of the function on the given interval is 56.
Answer: \boxed{56}
"
:::

---

Summary

---

---

What's Next?

---

---

Part 2: Optimization Involving a Single Variable

Introduction

The application of differential calculus to problems of optimization is a cornerstone of mathematical analysis and a frequent subject of inquiry in engineering and data science. At its core, optimization seeks to find the "best" value—either the maximum or the minimum—that a function can attain. In this section, we shall concern ourselves with the optimization of functions of a single variable, denoted $f(x)$. This involves identifying points in the domain of a function where it reaches its highest or lowest values, either locally (in a neighborhood) or globally (over its entire domain).

The methods we develop here are fundamental. They form the basis for more complex, multi-dimensional optimization algorithms that are central to machine learning, such as gradient descent. A firm grasp of how to locate and classify the extrema of a single-variable function is therefore an indispensable prerequisite for advanced studies. We will explore the role of the first and second derivatives in systematically identifying and classifying these points of interest.

---

Key Concepts

The search for extrema begins with identifying candidate points. A crucial insight, often attributed to Fermat, is that at a local maximum or minimum of a differentiable function, the tangent to the curve must be horizontal. This leads directly to the concept of critical points.

1. Critical Points

For a function $f(x)$, the candidates for local extrema occur at points where the derivative is either zero or undefined. These are known as the critical points of the function.

It is essential to understand that while all local extrema of a differentiable function occur at critical points, not all critical points correspond to local extrema. A point where $f'(c) = 0$ might also be a point of inflection, as seen with $f(x) = x^3$ at $x=0$. Therefore, we require further tests to classify these points.

2. The First Derivative Test

The First Derivative Test provides a method for classifying critical points by examining the sign of the first derivative, $f'(x)$, on either side of the critical point $c$. The sign of $f'(x)$ tells us whether the function $f(x)$ is increasing or decreasing.

• If $f'(x) > 0$ on an interval, then $f(x)$ is increasing on that interval.
• If $f'(x) < 0$ on an interval, then $f(x)$ is decreasing on that interval.

By observing the change in sign of $f'(x)$ as we pass through a critical point $c$, we can determine the nature of the extremum at $c$.

Local Max
f'(x) > 0
f'(x) < 0



Local Min
f'(x) < 0
f'(x) > 0

3. The Second Derivative Test

An alternative, and often computationally simpler, method for classifying critical points is the Second Derivative Test. This test uses the sign of the second derivative, $f''(x)$, at the critical point itself. The second derivative provides information about the concavity of the function.

• If $f''(x) > 0$ on an interval, then $f(x)$ is concave up on that interval.
• If $f''(x) < 0$ on an interval, then $f(x)$ is concave down on that interval.

Worked Example:

Problem: Find the local extrema of the function $f(x) = x^3 - 6x^2 + 9x + 1$.

Solution:

Step 1: Find the first derivative of the function to identify critical points.

Step 2: Set the first derivative to zero and solve for $x$.

The critical points are $x=1$ and $x=3$.

Step 3: Find the second derivative to apply the Second Derivative Test.

Step 4: Evaluate the second derivative at each critical point to classify them.

For $x=1$:

Since $f''(1) < 0$, the function has a local maximum at $x=1$.
The maximum value is:

For $x=3$:

Since $f''(3) > 0$, the function has a local minimum at $x=3$.
The minimum value is:

Answer: The function has a local maximum at $(1, 5)$ and a local minimum at $(3, 1)$.

---

---

Problem-Solving Strategies

A systematic approach is essential for solving optimization problems efficiently under exam conditions.

---

Common Mistakes

---

---

Practice Questions

:::question type="MCQ" question="The function $f(x) = 2x^3 - 3x^2 - 36x + 10$ has a local maximum at which value of $x$?" options=["3", "-2", "2", "-3"] answer="-2" hint="Find the critical points by setting $f'(x)=0$. Then use the Second Derivative Test to determine which one corresponds to a local maximum." solution="
Step 1: Find the first derivative.

Step 2: Set the derivative to zero to find critical points.

The critical points are $x=3$ and $x=-2$.

Step 3: Find the second derivative.

Step 4: Test each critical point with the second derivative.
For $x=3$:

This corresponds to a local minimum.

For $x=-2$:

This corresponds to a local maximum.

Result: The local maximum occurs at $x=-2$.
Answer: $\boxed{-2}$
"
:::

:::question type="NAT" question="A rectangular field is to be enclosed by a fence. The field's area must be 800 square meters. One side of the field is along a straight river and needs no fencing. What is the minimum length of fencing (in meters) required for the other three sides?" answer="80" hint="Let the side parallel to the river be of length $x$ and the other two sides be of length $y$. Express the total fencing length as a function of one variable and find its minimum." solution="
Step 1: Define the variables and formulate the equations.
Let $x$ be the length of the side parallel to the river and $y$ be the length of the two other sides.

Step 2: Express the length $L$ as a function of a single variable.
From the area equation, $y = \frac{800}{x}$.
Substitute this into the length equation:

Step 3: Find the derivative of $L(x)$ and set it to zero to find critical points.

Set $L'(x) = 0$:

Step 4: Use the Second Derivative Test to confirm it is a minimum.

At $x=40$:

This confirms a local minimum.

Step 5: Calculate the minimum length of fencing.
When $x=40$, $y = \frac{800}{40} = 20$.

Result: The minimum length of fencing required is 80 meters.
Answer: $\boxed{80}$
"
:::

:::question type="MSQ" question="For the function $f(x) = x^4 - 4x^3 + 10$ on the interval $[-1, 4]$, which of the following statements are true?" options=["The function has a local minimum at x=3.","The function has a local maximum at x=0.","The global maximum value on the interval is 15.","The global minimum value on the interval is -17."] answer="The function has a local minimum at x=3.,The global maximum value on the interval is 15.,The global minimum value on the interval is -17." hint="Find the critical points. Then evaluate the function at the critical points and the interval endpoints to find the global extrema." solution="
Step 1: Find the critical points.

Set $f'(x) = 0$:

The critical points are $x=0$ and $x=3$. Both are within the interval $[-1, 4]$.

Step 2: Classify the critical points.
We use the First Derivative Test.

• For $x < 0$ (e.g., $x=-0.5$), $f'(x) = 4(-0.5)^2(-0.5-3) < 0$.


• For $0 < x < 3$ (e.g., $x=1$), $f'(x) = 4(1)^2(1-3) < 0$.


• For $x > 3$ (e.g., $x=3.5$), $f'(x) = 4(3.5)^2(3.5-3) > 0$.

At $x=0$, the derivative does not change sign (negative to negative). So, $x=0$ is not a local extremum. It is a point of inflection.
At $x=3$, the derivative changes from negative to positive. So, $x=3$ is a local minimum.

Step 3: Evaluate the function at critical points and endpoints to find global extrema.

Step 4: Compare the values.
The values are $\{15, 10, -17, 10\}$.
The global maximum is 15.
The global minimum is -17.

Conclusion:

• "The function has a local minimum at $x=3$." is True.


• "The function has a local maximum at $x=0$." is False.


• "The global maximum value on the interval is 15." is True.


• "The global minimum value on the interval is -17." is True.

Answer: $\boxed{\text{Options 1, 3, and 4 are true.}}$
"
:::

---

Summary

---

What's Next?

---

Chapter Summary

---

Chapter Review Questions

:::question type="MCQ" question="The absolute minimum value of the function $f(x) = 2x^3 - 9x^2 + 12x + 5$ on the interval $[0, 3]$ is:" options=["5","9","10","14"] answer="5" hint="To find the absolute extrema on a closed interval, you must check the function's values at the critical points within the interval as well as at the endpoints of the interval." solution="
To find the absolute minimum value of $f(x) = 2x^3 - 9x^2 + 12x + 5$ on the interval $[0, 3]$, we apply the Closed Interval Method.

Step 1: Find the critical points.
First, we compute the derivative of the function:

Set the derivative to zero to find the critical points:



The critical points are $x=1$ and $x=2$. Both of these points lie within the given interval $[0, 3]$.

Step 2: Evaluate the function at the critical points and endpoints.
We must now evaluate $f(x)$ at the critical points ($x=1, x=2$) and at the endpoints of the interval ($x=0, x=3$).

• At the left endpoint, $x=0$:

• At the first critical point, $x=1$:

• At the second critical point, $x=2$:

• At the right endpoint, $x=3$:

Step 3: Determine the absolute minimum.
We compare the values obtained in Step 2: $\{5, 10, 9, 14\}$.
The smallest value is 5, which occurs at $x=0$. Therefore, the absolute minimum value of the function on the interval $[0, 3]$ is 5.
Answer: $\boxed{5}$
"
:::

:::question type="NAT" question="A wire of length 24 meters is to be bent into the shape of a rectangle. What is the maximum possible area of such a rectangle in square meters?" answer="36" hint="Formulate an objective function for the area in terms of a single variable using the perimeter constraint. Then, find the value of the variable that maximizes this function." solution="
Step 1: Define variables and formulate equations.
Let the length of the rectangle be $l$ and the width be $w$.
The perimeter $P$ is given by the length of the wire, so $P = 24$ m.

The area $A$ of the rectangle is the objective function we wish to maximize:

Step 2: Express the objective function in terms of a single variable.
From the perimeter constraint, we can express $l$ in terms of $w$: $l = 12 - w$.
Substitute this into the area formula:

Step 3: Find the critical points of the area function.
To find the maximum area, we take the derivative of $A(w)$ with respect to $w$ and set it to zero.

Setting the derivative to zero:

Step 4: Verify that this critical point corresponds to a maximum.
We use the Second Derivative Test. The second derivative is:

Since $\frac{d^2A}{dw^2} < 0$, the function has a local maximum at $w=6$. As the area function is a downward-opening parabola, this local maximum is also the global maximum.

Step 5: Calculate the maximum area.
When $w = 6$ m, the length is $l = 12 - w = 12 - 6 = 6$ m.
The rectangle is a square, which is the expected result for maximizing area for a fixed perimeter.
The maximum area is:

Answer: $\boxed{36}$
"
:::

:::question type="MCQ" question="The function $f(x) = x e^{-x}$ has a local maximum at which value of $x$?" options=["-1","0","1","e"] answer="1" hint="Find the first derivative of the function using the product rule, set it to zero to find the critical points, and then use the second derivative test to classify the point." solution="
Step 1: Find the first derivative of the function.
The function is $f(x) = x e^{-x}$. We use the product rule, $(uv)' = u'v + uv'$, where $u=x$ and $v=e^{-x}$.

Step 2: Find the critical points.
Set the first derivative to zero to find the critical points.

Since $e^{-x}$ is always positive for any real $x$, the only way for the product to be zero is if the other term is zero:

Thus, $x=1$ is the only critical point.

Step 3: Use the Second Derivative Test to classify the critical point.
We need to compute the second derivative, $f''(x)$. We differentiate $f'(x) = e^{-x} - x e^{-x}$.

Now, we evaluate the second derivative at the critical point $x=1$:

Since $f''(1) = -1/e < 0$, the function has a local maximum at $x=1$.
Answer: $\boxed{1}$
"
:::

:::question type="NAT" question="A function is defined as $f(x) = x^3 + ax^2 + bx$. It is known to have a local extremum at $x = -1$ and a point of inflection at $x = 1$. What is the value of $b$?" answer="-9" hint="A local extremum implies the first derivative is zero at that point. A point of inflection implies the second derivative is zero at that point. Use these two conditions to create a system of equations to solve for the unknown constants." solution="
The given function is $f(x) = x^3 + ax^2 + bx$.

Step 1: Find the first and second derivatives.

Step 2: Use the condition for the point of inflection.
We are given that there is a point of inflection at $x = 1$. This means that $f''(1) = 0$.

Step 3: Use the condition for the local extremum.
We are given that there is a local extremum at $x = -1$. This means that $f'(-1) = 0$.

Step 4: Solve for $b$.
Now we substitute the value of $a = -3$ that we found in Step 2 into the equation from Step 3.

The value of $b$ is -9.
Answer: $\boxed{-9}$
"
:::

---

What's Next?