Convex Sets

1. Affine and convex sets

1.1 Affine sets

​ A set \(C\in\R^n\) is affine if the line through any two distinct points in \(C\) lies in \(C\), which means \(\forall x_1,x_2\in C,\theta\in\R\), we have \(\theta x_1+(1-\theta)x_2\in C\).

​ An affine set \(C\) can be expressed as

\[C = V+x_0=\{v+x_0\ |\ v\in V\} \]

i.e., as a subspace plus an offset.

​ Every affine set can be expressed as the solution set of a system of linear equations.

\[C = \{x\ |\ Ax=b \}, A\in\R^{m\times n},b\in \R^m \]

​ The set of all affine combinations of points in some set \(C\in\R^n\) is called the affine hull of \(C\), and denoted \(\mathrm{aff}\ C\):

\[\mathrm{aff}\ C = \{\theta_1x_1+\cdots +\theta_kx_k\ |\ x_1,...,x_k\in C, \theta_1+\cdots+\theta_k=1\} \]

• The affine hull of the empty set is the empty set.
• The affine hull of a singleton (a set made of one single element) is the singleton itself.
• The affine hull of a set of two different points is the line through them.
• The affine hull of a set of three different points not on one line is the plane going through them.
• The affine hull of a set of four points not in a plane in \(\R^3\) is the entire space \(\R^3\).

1.2 Affine dimension and relative interior

​ We define the affine dimension of a set \(C\) as the dimension of its affine hull.

​ If the affine dimension of a set \(C\subseteq\R^n\) is less than \(n\), then the set lies in the affine set \(\mathrm{aff}\ C\ne \R^n\).

​ We define the relative interior of the set \(C\), denoted \(\mathrm{relint}\ C\), as its interior relative to \(\mathrm{aff}\ C\):

\[\mathrm{relint}\ C =\{x\in C\ |\ B(x,r)\cap\mathrm{aff}\ C\subseteq C\ \mathrm{for\ some}\ r>0 \} \]

where \(B(x,r) = \{y\ |\ \|y-x\|\le r\}\), the ball of radius \(r\) and center \(x\) in the norm \(\|\cdot\|\).

In mathematics, a normed vector space is a vector space over the real or complex numbers on which a norm is defined.

A norm is a generalization of the intuitive motion of "length" in the physical world. If \(V\) is a vector space over \(K\), where \(K\) is a field equal to \(\C\) or \(\R\), then a norm on \(V\) is a map \(V\rightarrow \R\), typically denoted as \(\|\cdot\|\), satisfying the following four axioms:

1. Non-negativity: for every \(x\in V,\|x\|\ge0\)
2. Positive Definiteness: for every \(x\in V, \|x\|=0\) if and only if \(x\) is the zero vector
3. Absolute Homogeneity: for every \(\lambda\in K,x\in V\), \(\|\lambda x\|=|\lambda|\ \|x\|\)
4. Triangle Inequality: for every \(x\in V, y\in V\), \(\|x+y\|\le\|x\|+\|y\|\)

The relative interior can be equivalently defined as

\[\begin{align} \mathrm{relint}\ C &:= \{x\in C\ |\ \forall\ y\in C, \exist\ \lambda>1, \lambda x+(1-\lambda)y\in C \} \end{align} \]

1.3 Convex sets

​ A set \(C\) is convex if the line segment between any two points in \(C\) lies in \(C\).

​ Obviously every affine set is convex, since it contains the entire line between any two distinct points in it.

​ A point of the form \(\theta_1x_1+\cdots+\theta_kx_k\), where \(\theta_1+\cdots+\theta_k=1\) and \(\theta_i\ge0,i=1,...,k\), is a convex combination of point \(x_1,...,x_k\). A set is convex if and only if it contains every convex combination of its points.

​ The convex hull of a set \(C\), denoted \(\mathrm{conv}\ C\), is the set of all convex combinations of points in \(C\):

\[\mathrm{conv}\ C = \{\theta_1x_1+\cdots+\theta_kx_k\ |\ x_i\in C,\theta_i\ge 0,i=1,...,k,\theta_1+\cdots+\theta_k=1\} \]

​ \(\theta_i\ge0\) is the key difference between convex hull and affine hull.

​ More generally, suppose \(p:\R^n\rarr\R\) satisfies \(p(x)\ge0\) for all \(x\in C\) and \(\displaystyle{\int_Cp(x)\mathrm d}x =1\), where \(C\in \R^n\) is convex. Then

\[\int_Cp(x)x\ \mathrm dx\in C \]

if the integral exists.

1.4 Cones

​ A set \(C\) is called a cone, or nonnegative homogeneous, if for every \(x\in C\) and \(\theta\ge0\) we have \(\theta x\in C\). A set \(C\) is a convex cone if it is convex and a cone, which means that for any \(x_1, x_1\in C\) and \(\theta_1,\theta_2\ge 0\), we have

\[\theta_1x_1+\theta_2x_2\in C \]

​ conic combination: a point of the form \(\theta_1x_1+\cdots+\theta_kx_k\) with \(\theta_1,...,\theta_k\ge0\).

​ Every conic combination of every point in a convex cone is also in the same convex cone.

​ The conic hull of a set \(C\):

\[\{\theta_1x_1 +\cdots +\theta_kx_k\ |\ x_i\in C,\theta_i\ge0,i=1,...,k \} \]

1.5 Some important examples

• The empty set \(\empty\), any single point, and the whole space \(\R^n\) are affine(hence, convex) subsets of \(\R^n\)
• Any line is affine. If it passes zero, it is a subspace, hence also a convex cone.
• A line segment is convex, but not affine (unless it reduces to a point)
• A ray, which has the form \(\{x_0+\theta v\ |\ \theta\ge0\}\), where \(v\ne0\), is convex, but not affine. It is a convex cone if its base \(x_0\) is 0.
• Any subspace is affine, and a convex cone (hence convex).

2. Hyperplanes and halfspaces

​ A hyperplane is a set of the form

\[\{x\ |\ a^Tx=b\} \]

where \(a\in\R^n, a\ne 0, b\in \R\). Analytically, it is the solution set of a nontrivial linear equation among the components of \(x\). Geometrically, it is a hyperplane with normal vector \(a\), the constant \(b\) is the offset of the hyperplane from the origin.

​ It can also be expressed as

\[\{x\ |\ a^T(x-x_0)=0\} = x_0+a^\bot \]

where \(a^\bot\) denotes the orthogonal component of \(a\):

\[a^\bot = \{v\ |\ a^Tv=0 \} \]

​ A hyperplane divides \(\R^n\) into two halfspaces, which has the form of

\[\{x\ |\ a^Tx\le b \} \]

A halfspace is convex but not affine.

2.2 Euclidean balls and ellipsoids

​ A Euclidean ball in \(\R^n\) has the form

\[B(x_c, r) = \{x\ |\ \|x-x_c\|_2\le r\} = \{x\ |\ (x-x_c)^(x-x_c)\le r^2 \} \]

where \(r>0\).

​ A Euclidean ball is a convex set: if \(\|x_1-x_c\|_2\le r,\|x_2-x_c\|_2\le r\), and \(0\le\theta\le1\), then

\[\begin{align} \|\theta x_1+(1-\theta)x_2-x_c\| &= \|\theta(x-x_c)+(1-\theta)(x_2-x_c)\|_2\\ &\le\theta\|x-x_c\|_2+(1-\theta)\|x_2-x_c\|_2\\ &\le r \end{align} \]

​ A related family of convex sets is the ellipsoids, which have the form

\[\varepsilon = \left\{x\ |\ (x-x_c)^TP^{-1}(x-x_c)\le1 \right\} \]

where \(P\) is symmetric and positive definite. The matrix \(P\) determines how far the ellipsoid extends in every direction from the center \(x_c\); the length of the semi-axes of \(\varepsilon\) are give by \(\sqrt{\lambda_i}\) where \(\lambda_i\) are the eigenvalues of \(P\).

2.3 Norm balls and norm cones

​ The norm cone associated with the norm \(\|\cdot\|\) is the set

\[C = \left\{(x,t)\ |\ \|x\|\le t \right\}\subseteq\R^{n+1} \]

2.4 Polyhedra

​ A polyhedra is defined a solution set of a finite number of equalities and inequalities:

\[\mathcal P = \left\{x\ |\ a_j^Tx\le b_j,j=1,...,m,\ c^T_jx=d_j,j=1,...,p \right\} \]

A polyhedra is thus the intersection of a finite number of halfspaces and hyperplanes. Affine sets, rays, line segments, and halfspaces are all polyhedra.

​ The compact notation:

\[\mathcal P = \left\{x\ |\ Ax\preceq b, Cx=d \right\} \]

where

\[A=\begin{bmatrix}a_1^T\\\vdots\\a_m^T \end{bmatrix},\quad C= \begin{bmatrix}c_1^T\\\vdots\\c_p^T \end{bmatrix} \]

Simplexes

​ Simplexes are another important family of polyhedra. Suppose the \(k+1\) points \(v_0,...,v_k\in\R^n\) are affinely independent, which means that \(v_1-v_0,...,v_k-v_0\) are linearly independent. The simplex determined by them is given by

\[C = \mathrm{\textbf{conv}} \{v_1,...,v_k\} = \{\theta_0v_0+\cdots+\theta_kv_k\ |\ \theta\succeq 0, \textbf 1^T\theta=1 \} \]

The affine dimension of this simplex is \(k\), so it sometimes referred to as a \(k\)-dimensional simplex in \(\R^n\).

• A 1-dimensional simplex is a line segment.
• A 2-dimensional simplex is a triangle includes it interior.
• A 3-dimensional simplex is a tetrahedron.
• The unit complex is the n-dimensional simplex determined by a zero-vector and the unit vectors.

​ To describe the simplex as a polyhedron, we define \(y=(\theta_1,...,\theta_k)\) and

\[B=\left[v_1-v_0,\cdots,v_k-v_0\right] \in \R^{n\times n} \]

we can say that \(x\in C\) if and only if

\[x = v_0+By \]

Note that matrix \(B\) has rank \(k\), so there exists a nonsingular matrix \(A=(A_1,A_2)\in\R^{n\times n}\) such that

\[\begin{bmatrix}A_1\\A_2 \end{bmatrix}B=\begin{bmatrix}I\\0 \end{bmatrix} \]

Multiplying on the left with \(A\), we get

\[A_1x=A_1v_0+y,\quad A_2x=A_2v_0 \]

As \(y\) satisfies \(y\succeq0\) and \(\textbf 1^Tyt\le1\), in other words we have \(x\in C\) if and only if

\[A_2x=A_2v_0,\quad A_1x\succeq A_1v_0,\quad 1^TA_1x\le1+\textbf1^TA_1v_0 \]

which is a set of linear equalities and inequalities in \(x\), and so describes a polyhedron.

Convex hull description of polyhedra

​ The convex hull of polyhedron:

\[\mathrm{\textbf{conv}}\left\{v_1,...,v_k\right\} = \left\{\theta_1v_1+\cdots+\theta_kv_k\ |\ \theta\succeq 0, \textbf1^T\theta=1 \right\} \]

A generalization of this convex hull description:

\[\left\{\theta_1v_1+\cdots+\theta_kv_k\ |\ \theta_1+\cdots+\theta_m=1,\theta_i\ge0,i=1,...,k \right\} \]

where \(m\le k\). We only require the first \(m\) coefficients to sum to one.

2.5 The positive semidefinite cone

​ Notation \(S^n\) represents a symmetric \(n\times n\) matrix

\[S^n = \{X\in \R^{n\times n}\ |\ X = X^T \} \]

which is a vector space with dimension \(n(n+1)/2\). Also

\[\begin{align} &S^n_+ = \{X\in \R^{n\times n}\ |\ X\succeq 0 \}\\ &S^n_{++} = \{X\in \R^{n\times n}\ |\ X\succ 0 \} \end{align} \]

3. Operations that preserve convexity

3.1 Intersection

​ Convexity is preserved under intersection: if \(S_1\) and \(S_2\) are convex, then \(S_1\cap S_2\) is convex.

​ This property extends to the intersection of an infinite number of sets: if \(S_\alpha\) is convex for every \(\alpha\in\mathcal A\), then \(\bigcap_{\alpha\in\mathcal A}S_\alpha\) is convex. The positive semidefinite cone can be expressed as

\[\bigcap_{z\ne0}\left\{X\in S^n\ |\ z^TXz\ge 0\right\} \]

In fact, a closed convex set \(S\) is the intersection of all halfspaces that contain it:

\[S=\bigcap\ \left\{\mathcal H\ |\ \mathcal H\ \mathrm{halfspace}, S\subseteq\mathcal H\right\} \]

3.2 Affine functions

​ A function \(f: \R^n\rarr\R^m\) is affine if it is a sum of a linear function and a constant, i.e., if it has the form \(f(x)=Ax+b\), where \(A\in\R^{m\times n},b\in\R^m\).

​ Suppose \(S\subseteq\R^n\) is convex and \(f:\R^n\rarr\R^m\) is an affine function, then the image of \(S\) under \(f\)

\[f(S) =\left\{f(x)\ |\ x\in S \right\} \]

is convex. Similarly, if \(f:\R^k\rarr\R^n\) is an affine function, the inverse image of \(S\) under \(f\)

\[f^{-1}(S) = \{x\ |\ f(x)\in S\} \]

is convex.

3.3 Linear-fractional and perspective functions

The perspective function

​ We define the perspective function \(P: \R^{n+1}\rarr\R^n\), with domain \(\mathrm{\textbf{dom}}\ P=\R^n\times\R_{++}\). The perspective function scales or normalizes vectors so the last component is one, and then drops the last component.

​ Suppose that \(x=(\tilde x, x_{n+1}), y=(\tilde y, y_{n+1})\in\R^{n+1}\) with \(x_{n+1}>0,y_{n+1}>0\). Then for \(0<\theta<1\)

\[P(\theta x+(1-\theta)y) = \frac{\theta\tilde x+(1-\theta)\tilde y}{\theta x_{n+1}+(1-\theta)y_{n+1}}=\mu P(x) +(1-\mu)P(y) \]

where

\[\mu = \frac{\theta x_{n+1}}{\theta x_{n+1}+(1-\theta)y_{n+1}}\in[0,1] \]

The correspondence between \(\mu\) and \(\theta\) is monotonic.

​ The inverse image of a convex set under the perspective function is also convex: if \(C\subseteq\R^n\) is convex, then

\[P^{-1}(C) =\left\{(x,t)\ |\ x/t\in C, t>0 \right\} \]

is convex.

Linear-fractional functions

​ A linear-fractional function is formed by composing the perspective function with an affine function. Suppose \(g:\R^n\rarr\R^{m+1}\) is affine, i.e.,

\[g(x) =\begin{bmatrix}A\\c^T \end{bmatrix}x+\begin{bmatrix} b\\d\end{bmatrix} \]

where \(A\in\R^{m\times n}, b\in\R^m,c\in\R^n, d\in R\). The function \(f:\R^n\rarr\R^m\) given by \(f = P\circ g\), i.e.,

\[f(x)=(Ax+b)/(c^Tx+d),\quad \mathrm{\textbf {dom}}\ f=\{x\ |\ c^Tx+d>0\} \]

is called a linear-fractional (or projective) function.

​ Like the perspective function, linear-fractional functions preserve convexity.

4. Generalized inequalities

4.1 Proper cones and generalized inequalities

​ A cone \(K\subseteq\R^n\) is called a proper cone if it satisfies the following:

• \(K\) is convex
• \(K\) is closed
• \(K\) is solid, which means it has nonempty interior
• \(K\) is pointed, which means it contains no line (or equivalently, \(x\in K, -x\in K\Longrightarrow x=0\))

​ We associate with the proper cone \(K\) the partial ordering on \(\R^n\) defined by

\[x\preceq_Ky\Longleftrightarrow y-x\in K \]

Similarly, we define an associated strict partial ordering by

\[x\prec_Ky\Longleftrightarrow y-x\in\mathrm{\textbf{int}}\ K \]

\(\mathrm{\textbf{int}}\ K\) represents the interior of \(K\).

​ When \(K=\R_+\), the partial ordering is \(\le\) on \(\R\).

Properties of generalized inequalities

​ A generalized inequality \(\preceq_K\) satisfies many properties, such as

• preserved under addition
• transitive
• preserved under nonnegative scaling
• reflexive: \(x\preceq_K x\)
• antisymmetric: if \(x\preceq_Ky\) and \(y\preceq_Kx\), then \(x=y\)
• preserved under limits: if \(x_i\preceq_K y_i\), for \(i=1,...,x_i\rarr x\) and \(y_i\rarr y\) as \(i\rarr\infin\), then \(x\preceq_K y\)

4.2 Minimum and minimal elements

​ We say that \(x\in S\) is the minimum element of \(S\) if for every \(y\in S\), \(x\preceq_K y\). We define the maximum element in a similar way. If a set has minimum element, the set is unique.

​ We say that \(x\in S\) is the minimal element of \(S\) if \(y\in S, y\preceq_K x\), then \(y = x\). A set can have many different minimal (maximal) elements.

​ A point \(x\in S\) is the minimum element of \(S\) if and only if

\[S\subseteq x+K \]

​ A point \(x\in S\) is the minimal element of \(S\) if and only if

\[(x-K)\cap S=\{x\} \]

5. Separating and supporting hyperplanes

5.1 Separating hyperplane theorem

​ separating hyperplane theorem: Suppose \(C\) and \(D\) are nonempty disjoint convex sets, and \(C\cap D=\empty\), then there exists \(a\ne0\) and \(b\) such that for all \(x\in C\), \(ax\le b\), and for every \(x\in D\), \(ax\ge b\).

Proof of separating hyperplane theorem

​ Here we consider a special case, we assume that the Euclidean distance between \(C\) and \(D\), defined as

\[\mathrm{\textbf{dist}}(C,D)=\inf\{\|u-v\|_2\ |\ u\in C, v\in D\} \]

\(\inf S\) represents the greatest lower bound of set \(S\).

is positive, and there exists \(c\in C, d\in D\) that achieve the minimum distance.

​ Define

\[a=d-c,\quad b=\frac{\|d\|_2^2-\|c\|^2_2}2 \]

Now we need to prove that the affine function

\[f(x)=a^Tx-b=(d-c)^T(x-(1/2)(d+c)) \]

is nonpositive on \(C\), and nonnegative on \(D\), i.e., the hyperplane \(\{x\ |\ a^Tx=b\}\) separates \(C\) and \(D\). This hyperplane is perpendicular to the line segment between \(c\) and \(d\).

​ We first show that \(f\) is nonnegative on \(D\), suppose there is point \(u\in D\) for which

\[f(u) =(d-c)^T(u-d+(1/2)(d+c))=(d-c)^T(u-d)+\frac12\|d-c\|^2_2 \]

Obviously \((d-c)^T(u-d)<0\), now we observe that

\[\left.\frac{\mathrm d}{\mathrm dt}\|d+t(u-d)-c\|^2_2\right|_{t=0} = 2(d-c)^T(u-d)<0 \]

so for some small \(t>0\), with \(t\le1\), we have

\[\|d+t(u-d)-c\|_2 < \|d-c\|_2 \]

i.e., the point \(d+t(u-d)\) is a closer point to \(c\) than \(d\) is, which is contradictory to our assumption. So \(f\) is nonnegative on \(D\).

Strict separation

​ If the separating hyperplane we constructed above satisfies the stronger condition that \(a^Tx<b\) for all \(x\in C\) and \(a^Tx>b\) for all \(x\in D\). This is called strict separation of the sets \(C\) and \(D\).

Converse separating hyperplane theorems

​ The converse of the separating hyperplane theorem (i.e., existence of a separating hyperplane implies that \(C\) and \(D\) do not intersect) is not true.

​ But by adding conditions on \(C\) and \(D\), we can have the following result: any two convex sets, at lease one of which is open, are disjoint if and only if exists a separating hyperplane.

5.2 Supporting hyperplanes

​ Suppose \(C\subseteq\R^n\), and \(x_0\) is a point in its boundary \(\mathrm{\textbf{bd}}\ C\).

If \(a\ne0\) satisfies \(a^Tx\le a^Tx_0\) for all \(x\in C\), then the hyperplane \(\{x\ |\ a^Tx=a^Tx_0\}\) is called a supporting hyperplane to \(C\) at the point \(x_0\). This is equivalent to saying that the set \(C\) and the point \(x_0\) are separated by the hyperplane \(\{x\ |\ a^Tx=a^Tx_0\}\). And \(a\) is the normal vector of this hyperplane.

​ A basic result, called the supporting hyperplane theorem, states that for any point \(x_0\in\mathrm{\textbf{bd}}\ C\), there exists a supporting hyperplane to \(C\) at \(x_0\).

​ We distinguish two cases, if the interior of set \(C\) is not empty, the result follows immediately by applying the separating hyperplane theorem to the sets \(\{x_0\}\) and \(\mathrm{\textbf{int}}\ C\). If the interior of \(C\) is empty, then \(C\) must lies in a affine set of dimension less than \(n\). and any hyperplane containing that affine set contains \(C\) and \(x_0\), and is a supporting hyperplane.

6. Dual cones and generalized inequalities

6.1 Dual cones

​ Let \(K\) be a cone. The set

\[K^* = \{y\ |\ x^Ty\ge0\ \mathrm{for\ all\ }x\in K\} \]

is called a dual cone of \(K\). \(K^*\) is a cone, and is always convex, even when the original cone \(K\) is not.

​ Geometrically, \(y\in K^*\) if and only if \(-y\) is the normal of a hyperplane that supports \(K\) at the origin.

​ A cone whose dual cone is itself is called a self-dual cone.

​ Dual cones satisfy several properties, such as:

• closed and convex.
• \(K_1\subseteq K_2\) implies \(K_1^*\subseteq K_2^*\).
• If \(K\) has nonempty interior, then \(K^*\) is pointed.
• If the closure of \(K\) is pointed then \(K^*\) has nonempty interior.
• \(K^{**}\) is the closure of convex hull of \(K\). (Hence is \(K\) is convex and closed, \(K^{**}=K\).)

6.2 Dual generalized and inequalities

​ Now suppose that the convex cone \(K\) is proper, so it induces a generalized inequality \(\preceq_K\). The its dual cone \(K^*\) is also proper, and induces a generalized inequality \(\preceq_{K^*}\). We refer it as the dual of the generalized inequality \(\preceq_K\).

​ Some important proprtties relating a generalized inequality and its dual are:

• \(x\preceq_Ky\) if and only if \(\lambda^Tx\le\lambda^Ty\) for all \(\lambda\succeq_{K^*}0\).
• \(x\prec_Ky\) if and only if \(\lambda^Tx<\lambda^Ty\) for all \(\lambda\succeq_{K^*}0,\lambda\ne0\).

6.3 Minimum and minimal elements via dual inequalities

Dual characterization of minimum element

​ \(x\) is the minimum element of \(S\), with respect to the generalized inequality \(\preceq_K\), if and only if for all \(\lambda\succ_{K^*}0\), \(x\) is the unique minimizer of \(\lambda^Tz\) over \(z\in S\). Geometrically, this means that for any \(\lambda\succ_{K^*}0\), the hyperplane

\[\{z\ |\ \lambda^T(z-x)=0 \} \]

is a strict supporting hyperplane to \(S\) at \(x\).

Dual characterization of minimal element

​ If \(\lambda\succ_{K^*}0\) and \(x\) minimizes \(\lambda^Tz\) over \(z\in S\), then \(x\) is minimal.

​ Convexity plays an important role in the converse, provided the set \(S\) is convex, we can say that for any minimal element \(x\) there exists a nonzero \(\lambda\succeq_{K^*}0\) such that \(x\) minimizes \(\lambda^Tz\) over \(z\in S\).

​ This converse theorem cannot be strengthened to \(\lambda\succ_{K^*}0\).