The purpose of this lecture is to provide a “crash course” on calculus with multiple variables. The motivation in the context of optimization should be obvious:

• We know from basic calculus that a necessary condition for a point $x$ to minimize a (differentiable) function $f : \R \to \R$ is that $f'(x)=0$
• We will want to generalize this kind of statements to multiple variables, and later also use them algorithmically to design optimization methods

We will (briefly) cover in this lecture:

• Differentiability with multiple variables
• Partial derivatives, directional derivatives and the gradient
• Second-order derivatives and the Hessian
• Multi-variable Taylor theorem

The general approach will be based on reducing the multi-variable case to the single-variable case, and using classical results we know from basic calculus.

Differentiation of Real-valued Functions

Recap: Differentiability in single variable

<aside> 💡 Definition: differentiable, single variable For $f : (a,b) \to \R$ and $x \in (a,b)$, if

$$

\lim_{h\to 0} \frac{f(x+h) - f(x)}{h} \quad\text{exists}, $$

then:

• We say that $f$ is differentiable at $x$, and that the above limit is the derivative of $f$ at $x$.
• We write $f'(x)$ to denote the derivative of $f$ at $x$. </aside>

Our first goal is to generalize this definition to multivariate functions $f : \R^d \to \R$.

Preliminary: Limits of multivariate functions