In this lecture we focus on the important concept of convexity in optimization. We will see that:

Recap of convexity, basic definitions and properties
“optimality conditions” give a simple way to verify optimality of solutions
convex optimization problems can be solved (approximated) efficiently
on the negative side, without convexity, even very simple optimization problems can be computationally hard
later in this course: we will also see statistical / out-of-sample guarantees implied by convexity

Context: “classic” optimization

The most basic optimization framework is the focus of traditional “mathematical optimization” / “nonlinear programming”:

<aside> 🚧 Classic “mathematical” optimization

Goal:

$$ \newcommand{\E}{\mathbb E}

\begin{aligned} \min \quad & F(w) \\ \text{s.t. } \quad & w \in W \end{aligned} $$

</aside>

$F : W \to \R$ is the objective, $W \subseteq \R^d$ is feasible domain / constraint set
many variants, differ by assumptions on $F$ and/or $W$ and on form of access to $F$ and $W$ (e.g., gradients of $F$, membership queries of $W$)

In the context of ML—