This lecture:

• Some basic flavors of optimization frameworks in ML
• Basic observations about relations between them

Batch Optimization

The most basic optimization framework is the focus of traditional “mathematical optimization” / “nonlinear programming”:

<aside> 🚧 Classic “mathematical” optimization

Goal:

$$ \newcommand{\E}{\mathbb E}

\begin{aligned} \min \quad & F(w) \\ \text{s.t. } \quad & w \in W \end{aligned} $$

</aside>

• $F : W \to \R$ is the objective, $W \subseteq \R^d$ is feasible domain / constraint set
• many variants, differ by assumptions on $F$ and/or $W$ and on form of access to $F$ and $W$ (e.g., gradients of $F$, membership queries of $W$)

In the context of ML—

• $F$ often measures fit of model $w$ to data
• $W$ constrains “complexity” of model given by parameters $w$

One prominent instance is “Empirical Risk Minimization”, aka finite-sum optimization:

<aside> 🚧 Finite-sum optimization / Empirical Risk Minimization (ERM)

Goal:

$$ \newcommand{\E}{\mathbb E}

\begin{aligned} \min_{w \in W} \; F_S(w) = \frac1n \sum_{i=1}^n f(w,z_i) \end{aligned} $$

given sample $S$ of data samples / examples $z_1, \ldots, z_n \in Z$

</aside>