Convex Functions

1. Basic properties and examples

1.1 Definitions

​ A function $f:\R^n\rarr\R$ is convex if $\mathrm{\textbf{dom}}\ f$ is a convex set and if for all $x,y\in\mathrm{\textbf{dom}}\ f$, and $\theta$ with $0\le\theta\le1$, we have

$$f(\theta x+(1-\theta)y)\le\theta f(x)+(1-\theta)f(y)$$

We say $f$ is concave if $-f$ is convex.

1.2 Extended-value extensions

​ It is often convenient to extend a convex function to all of $\R^n$ by defining its value to be $\infin$ outside its domain. If $f$ is convex, define its extended-value extension $\tilde f: \R^n\rarr\R\cup\{\infin\}$ by

$$\tilde f(x)=\left\{\begin{align}&f(x) &x\in\mathrm{\textbf{dom}}\ f\\ &\infin &x\notin\mathrm{\textbf{dom}}\ f \end{align} \right.$$

$\tilde f(x)$ is also convex.

1.3 First-order conditions

​ Suppose $f$ is differentiable (i.e., its gradient $\nabla f$ exists at each point in $\mathrm{\textbf{dom}}\ f$, which is open). Then $f$ is convex if and only if $\mathrm{\textbf{dom}}\ f$ is convex and

$$f(y)\ge f(x)+\nabla f(x)^T(y-x)$$

holds for all $x,y\in\mathrm{\textbf{dom}}\ f$. Similarly, $f$ is concave if and only if $\mathrm{\textbf{dom}}\ f$ is convex and

$$f(y)\le f(x)+\nabla f(x)^T(y-x)$$

for all $x,y\in\mathrm{\textbf{dom}}\ f$.

1.4 Second-order conditions

​ We now assume $f$ is twice differentiable, that is, its Hessian or second derivative $\nabla^2f$ exists at each point of $\mathrm{\textbf{dom}}\ f$, which is open. Then $f$ is convex if and only if $\mathrm{\textbf{dom}}\ f$ is convex and its Hessian is positive semidefinite: for all $x\in\mathrm{\textbf{dom}}\ f$,

$$\nabla^2f(x)\succeq0$$

The Hessian matrix of function $f$ is defined by

$$H_f= \begin{bmatrix} \displaystyle\frac{\part^2 f}{\part x_1^2} & \displaystyle\frac{\part^2 f}{\part x_1\part x_2} & \cdots & \displaystyle\frac{\part^2 f}{\part x_1\part x_n} \\ \displaystyle\frac{\part^2 f}{\part x_2\part x_1} & \displaystyle\frac{\part^2 f}{\part x_2^2} & \cdots & \displaystyle\frac{\part^2 f}{\part x_2\part x_n} \\ \vdots & \vdots & \ddots & \vdots\\ \displaystyle\frac{\part^2 f}{\part x_n\part x_n} & \displaystyle\frac{\part^2 f}{\part x_n\part x_2} & \cdots & \displaystyle\frac{\part^2 f}{\part x_n^2} \end{bmatrix}$$

For a function on $\R$, this reduces to the simple condition $f''(x)\ge0$, which means that the derivative is nondecreasing. The condition $\nabla^2f(x)\succeq0$ can be interpreted geometrically as the requirement that the graph of the function have positive curvature at $x$.

1.5 Examples

​ Omitted

1.6 Sublevel sets

​ The $\alpha$-sublevel set of a function $f:\R^n\rarr\R$ is defined as

$$C_\alpha =\{x\in\mathrm{\textbf{dom}}\ f |\ f(x)\le\alpha\}$$

Sublevel sets of a convex function are convex, for any value of $\alpha$.

​ Similarly, if $f$ is concave, its $\alpha$-superlevel set, given by $\{x\in\mathrm{\textbf{dom}}\ f\ |\ f(x)\ge\alpha\}$, is a convex set.

1.7 Epigraph

​ The graph of a function $f:\R^n\rarr\R$ is defined as

$$\{(x,f(x))\ |\ x\in \mathrm{\textbf{dom}}\ f\}$$

which is a subset of $\R^{n+1}$. The epigraph of a function $f:\R^n\rarr\R$ is defined as

$$\mathrm{\textbf{epi}}\ f=\{(x,t)\ |\ x\in\mathrm{\textbf{dom}}\ f,f(x)\le t\}$$

A function is convex if and only if its epigraph is a convex set. A function is concave if and only if its hypograph, defined as

$$\mathrm{\textbf{hypo}}\ f=\{(x,t)\ |\ t\le f(t)\}$$

is a convex set.

​ Many results for convex functions can be proved (or interpreted) geometrically using epigraphs, and applying results for convex sets. As an example, consider the first-order condition for convexity:

$$f(y)\ge f(x)+\nabla f(x)^T(y-x)$$

where $f$ is convex and $x,y\in\mathrm{\textbf{dom}}\ f$. We can interpret this basic inequality geometrically in terms of $\mathrm{\textbf{epi}}\ f$. If $(y,t)\in \mathrm{\textbf{epi}}\ f$, then

$$t\ge f(y)\ge f(x)+\nabla f(x)^T(y-x)$$

We can express this as:

$$(y,t)\in\mathrm{\textbf{epi}}\ f\Longrightarrow \begin{bmatrix} \nabla f(x)\\ -1 \end{bmatrix}^T \left( \begin{bmatrix}y\\t \end{bmatrix}- \begin{bmatrix}x\\f(x) \end{bmatrix} \right)\le 0$$

This means that the hyperplane defined by $(\nabla f(x),-1)$ supports $\mathrm{\textbf{epi}}\ f$ at the boundary point $(x,f(x))$.

1.8 Jensen's inequality and extensions

​ The basic inequality

$$f(\theta x+(1-\theta)y) \le\theta f(x)+(1-\theta)f(y)$$

is sometimes called Jensen's inequality.

​ As in the case of convex sets, the inequality extends to infinite sums, integrals, and expected values. For example, if $p(x)\ge0$ on $S\subseteq\mathrm{\textbf{dom}}\ f$, $\int_Sp(x)\mathrm dx=1$, then

$$f\left(\int_Sp(x)x\mathrm dx \right)\le\int_Sf(x)p(x)\mathrm dx$$

​ If $x$ is a random variable such that $x\in\mathrm{\textbf{dom}}\ f$ with probability one, and $f$ is convex, then we have

$$f(\mathrm{\textbf{E}}x)\le \mathrm{\textbf{E}}f(x)$$

​ All of these inequalities are now called Jensen's inequality.

2. Operations that preserve convexity

2.1 Nonnegative weighted sums

​ If $f(x,y)$ is convex in $x$ for every $y\in\mathcal A$, and $w(y)\ge0$ for every $y\in\mathcal A$, then the function $g$ defined as

$$g = \int_{\mathcal A}w(y)f(x,y)\mathrm dy$$

is convex in $x$ (provided that the integral exists).

2.2 Composition with an affine mapping

​ Suppose $f: \R^n\rarr\R,A\in\R^{n\times m},b\in\R^n$ Define $g:\R^m\rarr\R$ by

$$g(x)=f(Ax+b)$$

with $\mathrm{\textbf{dom}}\ g=\{x\ |\ Ax+b\in\mathrm{\textbf{dom}}\ f\}$. If $f$ is convex, then so is $g$.

2.3 Pointwise maximum and supremum

​ If $f_1$ and $f_2$ are convex functions then their pointwise maximum $f$, defined by

$$f(x) = \max\{f_1(x),f_2(x)\}$$

with $\mathrm{\textbf{dom}}\ f=\mathrm{\textbf{dom}}\ f_1\cap\mathrm{\textbf{dom}}\ f_2$ is also convex.

2.4 Composition

​ Suppose $h:\R^k\rarr\R,g:\R^n\rarr\R^k$ and they are convex, then we discuss the convexity of their composition $f=h\circ g:\R^n\rarr\R$, defined as

$$f = h(g(x)),\quad \mathrm{\textbf{dom}}\ f=\{x\in\mathrm{\textbf{dom}}\ g\ |\ g(x)\in\mathrm{\textbf{dom}}\ h\}$$

Omitted.

3. The conjugate function

3.1 Definitions

The supermum of a subset $S$ of a partially ordered set $P$ is the least element in $P$ that is greater than or equal to each element of $S$, if such an element exists.

​ Let $f:\R^n\rarr\R$. The function $f^*:\R^n\rarr\R$ defined as

$$f^*(y)=\sup_{x\in\mathrm{\textbf{dom}}\ f}(y^Tx-f(x))$$

is called the conjugate of the function $f$. The domain of the conjugate function consists of $y\in\R^n$ for which the supremum is finite, i.e., for which the difference $y^Tx-f(x)$ is bounded above on $\mathrm{\textbf{dom}}\ f$.

​ We see immediately that $f^*$ is convex, since it is the pointwise supermum of a family of convex functions of $y$. This is true whether or not $f$ is convex.

3.2 Basic properties

Fenchel's inequality

$$f^*(y)+f(x)\ge y^Tx$$

This is quite obvious.

Conjugate of the conjugate

​ The conjugate of the conjugate of a convex function is the original function.

Differentiable functions ★

​ The conjugate of a differentiable function $f$ is also called the Legendre transform of $f$.

​ Suppose $f$ is convex and differentiable, with $\mathrm{\textbf{dom}}\ f=\R^n$. Any maximizer $x^*$ of $y^Tx-f(x)$ satisfies $y=\nabla f(x^*)$, and conversely, if $x^*$ satisfies $y=\nabla f(x^*)$, then $x^*$ maximizes $y^Tx-f(x)$. Therefore, if $y=\nabla f(x^*)$, we have

$$f^*(y)=x^{*T}\nabla f(x^*)-f(x^*)$$

This allows us to determine $f^*(y)$ for any $f$ for which we can solve the gradient equation $y=\nabla f(z)$ for any $z$.

​ We can express this another way. Let $z\in\R^n$ be arbitrary and define $y=\nabla f(z)$. Then we have

$$f^*(y)=z^T\nabla f(z)-f(z)$$

Scaling and composition with affine transformation

​ Suppose $A\in\R^{n\times n}$ is nonsingular and $b\in\R^n$. Then the conjugate of $g(x)=f(Ax+b)$ is

$$g^*(y)=f^*(A^{-T}y)-b^TA^{-T}y$$

with $\mathrm{\textbf{dom}}\ g^* = A^T\mathrm{\textbf{dom}}\ f^*$.

Sums of independent functions

​ If $f(u,v) = f_1(u)+f_2(v)$, where $f_1$ and $f_2$ are convex functions with conjugates $f_1^*$ and $f_2^*$, respectively, then

$$f^*(w,z)=f_1^*(w)+f^*_2(z)$$

4. Quasiconvex functions

4.1 Definition

​ A function $f:\R^n\rarr\R$ is called quasiconvex if its domain and all its sublevel sets

$$S_\alpha=\{x\in\mathrm{\textbf{dom}}\ f\ |\ f(x)\le\alpha\}$$

for all $\alpha\in\R$, are convex. A function is quasiconcave if $-f$ is quasiconvex, i.e., every superlevel set $\{x\ |\ f(x)\ge\alpha\}$ is convex.

​ A function that is both quasiconvex and quasiconcave is called quasilinear. If a function is quasilinear, then its domain, and every level set $\{x\ |\ f(x)=\alpha\}$ is convex.

​ For a function on $\R$, quasiconvexity requires that each sublevel set to be an interval. All convex functions are quasiconvex, but not all quasiconvex functions are convex, so quasiconvexity is a generalization of convexity.

4.2 Basic properties

​ A function is quasiconvex if and only $\mathrm{\textbf{dom}}\ f$ is convex and for any $x,y\in\mathrm{\textbf{dom}}\ f$ and $0\le\theta\le1$,

$$f(\theta x+(1-\theta y))\le\max\{f(x),f(y)\}$$

Quasiconvex functions on $\R$

​ A continuous function $f:\R\rarr\R$ is quasiconvex if and only if at least one of the following conditions holds:

• $f$ is nondecreasing
• $f$ is nonincreasing
• there is a point $c\in\mathrm{\textbf{dom}}\ f$ such that for $t\le c$ (and $t\in\mathrm{\textbf{dom}}\ f$), $f$ is nonincreasing, and for $t\ge c$ (and $t\in\mathrm{\textbf{dom}}\ f$), $f$ is nondecreasing.

4.3 Differentiable quasiconvex functions

First-order conditions

​ Suppose $f:\R^n\rarr\R$ is differentiable. Then $f$ is quaisconvex if and only if $\mathrm{\textbf{dom}}\ f$ is convex and for all $x,y\in\mathrm{\textbf{dom}}\ f$

$$f(y)\le f(x)\Longrightarrow\nabla f(x)^T(y-x)\le 0\tag{4.1}$$

​ Proof It suffices to proove the result for a function on $\R$; the general result follows by restrication to an arbitrary line.

​ Suppose $f$ is differentiable function on $\R$ and satisfies

$$f(y)\le f(x)\Longrightarrow f'(x)(y-x)\le 0$$

Suppose $f(x_1)\ge f(x_2)$ where $x_1\ne x_2$. We assume $x_2>x_1$, and show that $f(z)\le f(x_1)$ for $z\in[x_1,x_2]$. Suppose there exists a $z\in[x_1,x_2]$ with $f(z)>f(x_1)$. Since $f$ is differentiable, we can choose a $z$ that also satisfies $f'(z)<0$. Because

$$f(x_2)\le f(x_1)<f(z)\Longrightarrow f'(z)(x_2-z)\le 0$$

However, $f(x_1)<f(z)\Longrightarrow f'(z)(x_1-z)\le0$ implies $f'(z)\ge0$, which contradicts $f'(z)<0$.

​ To prove sufficiency, assume $f$ is quasiconvex. Suppose $f(x)\ge f(y)$. By the definition of quasiconvexity $f(x+t(y-x))\le f(x)$ for $0<t\le1$. Diving both sides by $t$, taking limit for $t\rarr0$, we obtain

$$\lim_{t\rarr0}\frac{f(x+t(y-x))-f(x)}t=f'(x)(y-x)\le0$$

​ The condition (4.1) has a simple geometric interpretation when $\nabla f(x)\ne0$. It states that $\nabla f(x)$ defines a supporting hyperplane to the sublevel set $\{y\ |\ f(y)\le f(x)\}$, at the point $x$.

Second-order conditions

​ Now suppose $f$ is twice differentiable. If $f$ is quasiconvex, then for all $x\in\mathrm{\textbf{dom}}\ f$, and all $y\in\R^n$, we have

$$y^T\nabla f(x)=0\Longrightarrow y^T\nabla^2f(x)y\ge0\tag{4.2}$$

For a quasicconvex function on $\R$, this reduces a simple condition

$$f'(x)=0\Longrightarrow f''(x)\ge0$$

For a quasiconvex function on $\R^n$, when $\nabla f(x)\ne0$, the condition (4.2) means that $\nabla^2f(x)$ is positive semidefinite on the $(n-1)$-dimensional subspace $\nabla f(x)^{\perp}$. This implies that $\nabla^2f(x)$ can have at most one negative eigenvalue.

​ As a (partial) converse, if $f$ satisfies

$$y^T\nabla f(x)=0\Longrightarrow y^T\nabla^2f(x)y\ge0$$

for $x\in\mathrm{\textbf{dom}}\ f$ and all $y\in\R^n,y\ne0$, then $f$ is quasiconvex.

4.4 Operations that preserve quasiconvexity

Omitted

4.5 Representation via family of convex functions

​ In the sequel, it will be convenient to represent the sublevel sets of a quasiconvex function $f$ (which are convex) via inequalities of convex functions. We seek a family of convex functions $\phi_t:\R^n\rarr\R$, indexed by $t\in\R$, with

$$f(x)\le t\Longleftrightarrow\phi_t(x)\le0\tag{4.3}$$

​ To see that such a representable always exists, we can take

$$\phi_t(x)=\left\{\begin{align}&0&&f(x)\le t\\&\infin&&\mathrm{otherwise} \end{align}\right.$$

5. Log-concave and log-convex functions

Omitted

6. Convexity with respect to generalized inequalities

6.1 Monotonicity with respect to a generalized inequality

​ Suppose $K\subseteq\R^n$ is a proper cone with associated generalized inequality $\preceq_K$. A function $f:\R^n\rarr\R$ is called K-nondecreasing if

$$x\preceq_K y\Longrightarrow f(x)\le f(y)$$

and K-increasing if

$$x\prec_K y,x\ne y\Longrightarrow f(x)<f(y)$$

Gradient conditions with monotonicity

​ A differentiable function $f$, with convex domain, is $K$-nondecreasing if and only if

$$\nabla f(x)\succeq_{K^*}0$$

for all $x\in\mathrm{\textbf{dom}}\ f$. The converse is not true.

6.2 Convexity with respect to a generalized inequality

​ Suppose $K\subseteq\R^m$ is a proper cone with associated generalized inequality $\preceq_K$. We say $f: \R^n\rarr\R^m$ is $K$-convex if for all $x,y$, and $0\le\theta\le1$,

$$f(\theta x+(1-\theta)y)\preceq_K\theta f(x)+(1-\theta)f(y)$$

Dual characterization of $K$-convexity

​ A function $f$ is $K$-convex if and only for every $w\succeq_{K^*}0$, the function $w^Tf$ is convex.

Differentiable $K$-convex functions

​ A differentiable function $f$ is $K$-convex if and only if its domain is convex, and for all $x,y\in\mathrm{\textbf{dom}}\ f$,

$$f(y)\succeq_Kf(x)+\mathrm Df(x)(y-x)$$

(Here $\mathrm Df(x)\in\R^{m\times n}$ is the derivative or Jacobian matrix of $f$ at $x$).

Suppose $f:\R^n\rarr\R^m$, the Jacobian matrix of $f$ is defined as

$$J= \begin{bmatrix} \displaystyle\frac{\part f_1}{\part x_1}& \cdots& \displaystyle\frac{\part f_1}{\part x_n}\\ \vdots & \ddots &\vdots\\ \displaystyle\frac{\part f_m}{\part x_1}& \cdots& \displaystyle\frac{\part f_m}{\part x_n} \end{bmatrix}$$

Composition theorem

​ If $g:\R^n\rarr\R^p$ is $K$-convex, $h:\R^p\rarr\R$ is convex, and $\tilde h$(the extended-value extensions of $h$) is $K$-nondecreasing, then $h\circ g$ is convex.