This course teaches optimization in machine learning with NumPy and SciPy, covering fundamentals, convex and non-convex techniques, and advanced algorithms.

Optimization for Machine Learning with NumPy and SciPy copy.png

docker.tar.gz

In this course, you will learn about optimization, one of the fundamental pillars of mathematics and machine learning. Machine learning depends heavily on optimization because it allows the model to learn from data and generate precise predictions.

You will begin by introducing optimization. Then, you will learn about optimization basics, including gradients and integrals. Next, you will cover convex optimization. You will then learn how to compute gradient descent for non-convex optimization. Next, you’ll learn how to perform constrained optimization. You will finish the course by studying the miscellaneous methods of optimization, like Newton’s methods, quasi-Newton methods, and conjugate gradient descent.

After completing this course, you’ll have the practical skills to formulate, analyze, and implement optimization algorithms for machine learning using the NumPy and SciPy libraries. This will help you become a highly proficient data scientist or machine learning engineer.

Optimization for Machine Learning with NumPy and SciPy

What is the sparsity of the following matrix?
$$\begin{bmatrix} 3 & 0 & -1 & 2 \\ -5 & 0 & 0 & 1 \\ 0 & 0 & 2 & 1 \\ 1 & 2 & 3 & 4\end{bmatrix}$$

What is the sparsity of the following matrix?
<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mo fence="true">[</mo><mtable rowspacing="0.16em" columnalign="center center center center" columnspacing="1em"><mtr><mtd><mstyle scriptlevel="0" displaystyle="false"><mn>3</mn></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mn>0</mn></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mrow><mo>−</mo><mn>1</mn></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mn>2</mn></mstyle></mtd></mtr><mtr><mtd><mstyle scriptlevel="0" displaystyle="false"><mrow><mo>−</mo><mn>5</mn></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mn>0</mn></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mn>0</mn></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mn>1</mn></mstyle></mtd></mtr><mtr><mtd><mstyle scriptlevel="0" displaystyle="false"><mn>0</mn></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mn>0</mn></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mn>2</mn></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mn>1</mn></mstyle></mtd></mtr><mtr><mtd><mstyle scriptlevel="0" displaystyle="false"><mn>1</mn></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mn>2</mn></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mn>3</mn></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mn>4</mn></mstyle></mtd></mtr></mtable><mo fence="true">]</mo></mrow><annotation encoding="application/x-tex">\begin{bmatrix} 3 &amp; 0 &amp; -1 &amp; 2 \\ -5 &amp; 0 &amp; 0 &amp; 1 \\ 0 &amp; 0 &amp; 2 &amp; 1 \\ 1 &amp; 2 &amp; 3 &amp; 4\end{bmatrix}
</annotation></semantics></math>⎣<svg xmlns="http://www.w3.org/2000/svg" width='0.6667em' height='1.216em' style='width:0.6667em' viewBox='0 0 666.67 1216' preserveAspectRatio='xMinYMin'><path d='M319 0 H403 V1216 H319z M319 0 H403 V1216 H319z'/></svg>⎡​3−501​0002​−1023​2114​⎦<svg xmlns="http://www.w3.org/2000/svg" width='0.6667em' height='1.216em' style='width:0.6667em' viewBox='0 0 666.67 1216' preserveAspectRatio='xMinYMin'><path d='M263 0 H347 V1216 H263z M263 0 H347 V1216 H263z'/></svg>⎤​

$\hat{g}(x, t) = x_1 e^{x_1} + x_2 e^{x_2} - \frac{1}{t} \left ( \log(1- x_1 - x_2) +\log(x_2) +\log(x_1) \right)$

$\hat{g}(x, t) = x_1 e^{x_1} + x_2 e^{x_2} + \frac{1}{t} \left ( \log(x_1 + x_2 -1) +\log(-x_2) +\log(-x_1) \right)$

$\hat{g}(x, t) = x_1 e^{x_1} + x_2 e^{x_2} - t \left ( \log(x_1 + x_2 -1) +\log(-x_2) +\log(-x_1) \right)$

$\hat{g}(x, t) = x_1 e^{x_1} + x_2 e^{x_2} + t \left ( \log(x_1 + x_2 -1) +\log(-x_2) +\log(-x_1) \right)$

Which is the correct log barrier objective of the following constrained optimization problem?

$$\min_x \ g(x) = \ x_1 e^{x_1} + x_2 e^{x_2} \\ \text{ subject to } \\ x_1 + x_2 \leq 1 \\ x_1,x_2 \geq 0.

Which is the correct log barrier objective of the following constrained optimization problem?
<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><munder><mrow><mi>min</mi><mo>⁡</mo></mrow><mi>x</mi></munder><mtext> </mtext><mi>g</mi><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo><mo>=</mo><mtext> </mtext><msub><mi>x</mi><mn>1</mn></msub><msup><mi>e</mi><msub><mi>x</mi><mn>1</mn></msub></msup><mo>+</mo><msub><mi>x</mi><mn>2</mn></msub><msup><mi>e</mi><msub><mi>x</mi><mn>2</mn></msub></msup><mspace linebreak="newline"></mspace><mtext> subject to </mtext><mspace linebreak="newline"></mspace><msub><mi>x</mi><mn>1</mn></msub><mo>+</mo><msub><mi>x</mi><mn>2</mn></msub><mo>≤</mo><mn>1</mn><mspace linebreak="newline"></mspace><msub><mi>x</mi><mn>1</mn></msub><mo separator="true">,</mo><msub><mi>x</mi><mn>2</mn></msub><mo>≥</mo><mn>0.</mn></mrow><annotation encoding="application/x-tex">\min_x \ g(x) = \ x_1 e^{x_1} + x_2 e^{x_2} \\ \text{ subject to } \\ x_1 + x_2 \leq 1 \\ x_1,x_2 \geq 0.
</annotation></semantics></math>xmin​ g(x)= x1​ex1​+x2​ex2​ subject to x1​+x2​≤1x1​,x2​≥0.

Given the two points on the functions $(x_1, y_1) = (1, 3)$ and $(x_2, y_2) = (2, -1)$, what will be the function value of $y_3$ in $(x_3,y_3) = (1.5, ?)$ based on the concept of linear interpolation?

Given the two points on the functions <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><msub><mi>x</mi><mn>1</mn></msub><mo separator="true">,</mo><msub><mi>y</mi><mn>1</mn></msub><mo stretchy="false">)</mo><mo>=</mo><mo stretchy="false">(</mo><mn>1</mn><mo separator="true">,</mo><mn>3</mn><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">(x_1, y_1) = (1, 3)</annotation></semantics></math>(x1​,y1​)=(1,3) and <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><msub><mi>x</mi><mn>2</mn></msub><mo separator="true">,</mo><msub><mi>y</mi><mn>2</mn></msub><mo stretchy="false">)</mo><mo>=</mo><mo stretchy="false">(</mo><mn>2</mn><mo separator="true">,</mo><mo>−</mo><mn>1</mn><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">(x_2, y_2) = (2, -1)</annotation></semantics></math>(x2​,y2​)=(2,−1), what will be the function value of <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>y</mi><mn>3</mn></msub></mrow><annotation encoding="application/x-tex">y_3</annotation></semantics></math>y3​ in <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><msub><mi>x</mi><mn>3</mn></msub><mo separator="true">,</mo><msub><mi>y</mi><mn>3</mn></msub><mo stretchy="false">)</mo><mo>=</mo><mo stretchy="false">(</mo><mn>1.5</mn><mo separator="true">,</mo><mo stretchy="false">?</mo><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">(x_3,y_3) = (1.5, ?)</annotation></semantics></math>(x3​,y3​)=(1.5,?) based on the concept of linear interpolation?

$x_{t+1} = x_t - \frac{f''(x_t)}{f'(x_t)}$

$x_{t+1} = x_t - \frac{f'(x_t)}{f''(x_t)}$

Which is the correct iterative formula of Newton’s method in one dimension? Here, $f'(x)$ and $f''(x)$ represent the first- and second-order derivatives of the objective $f(x)$.

Which is the correct iterative formula of Newton’s method in one dimension? Here, <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>f</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">f&#x27;(x)</annotation></semantics></math>f′(x) and <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>f</mi><mrow><mo mathvariant="normal">′</mo><mo mathvariant="normal">′</mo></mrow></msup><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">f&#x27;&#x27;(x)</annotation></semantics></math>f′′(x) represent the first- and second-order derivatives of the objective <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>f</mi><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">f(x)</annotation></semantics></math>f(x).

$x_{t+1} = x_t - \frac{(3 + 2\log(x))}{(x + 2x\log(x))}$ .

$x_{t+1} = x_t - \frac{(x + x\log(x))}{(3 + 2\log(x))}$ .

$x_{t+1} = x_t - \frac{(x + 2x\log(x))}{(1 + \log(x))}$ .

$x_{t+1} = x_t - \frac{(x + 2x\log(x))}{(3 + 2\log(x))}$ .

Which is the correct update rule for optimizing the function, $f(x) = x^2\log(x)$, using Newton’s method?

Which is the correct update rule for optimizing the function, <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>f</mi><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo><mo>=</mo><msup><mi>x</mi><mn>2</mn></msup><mi>log</mi><mo>⁡</mo><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">f(x) = x^2\log(x)</annotation></semantics></math>f(x)=x2log(x), using Newton’s method?

Evaluate the concepts learned in this chapter via an MCQ quiz.

Introduction to Optimization

Vector Calculus

Convex Optimization

Gradient Descent for Non-Convex Optimization

Constrained Optimization

Miscellaneous Methods

Course Conclusion

Test Your Concepts of Optimization

Training Support Vector Machines (SVMs)

Quiz Yourself on Miscellaneous Methods