The purpose of this lecture is to provide a “crash course” on calculus with multiple variables. The motivation in the context of optimization should be obvious:

• We know from basic calculus that a necessary condition for a point $x$ to minimize a (differentiable) function $f : \R \to \R$ is that $f'(x)=0$
• We will want to generalize this kind of statements to multiple variables, and later also use them algorithmically to design optimization methods that find such solutions

We will (briefly) cover in this lecture:

• Differentiability with multiple variables
• Partial derivatives, directional derivatives and the gradient
• Second-order derivatives and the Hessian
• Multi-variable Taylor theorem

The general approach will be based on reducing the multi-variable case to the single-variable case, and using classical results we know from basic calculus.

Differentiation of Real-valued Functions

Recap: Differentiability in single variable

<aside> 💡 Definition: differentiable, single variable For $f : (a,b) \to \R$ and $x \in (a,b)$, if

$$

\lim_{h\to 0} \frac{f(x+h) - f(x)}{h} \quad\text{exists}, $$

then:

• We say that $f$ is differentiable at $x$, and that the above limit is the derivative of $f$ at $x$.
• We write $f'(x)$ to denote the derivative of $f$ at $x$. </aside>

Our first goal is to generalize this definition to multivariate functions $f : \R^d \to \R$.

Preliminary: Limits of multivariate functions