Approximation Theory and Method part 1

Approximation Theory and Method part 1.

Recovery

Euclidean space

Definition (Euclidean Space) A Euclidean space is a finite-dimensional vector space over the reals $\mathbf{R}$, with an inner product $⟨\cdot ,\cdot ⟩$.

Euclidean space 是

• 有限维度的;
• 实的;
• 装备内积;

Euclidean space 可以理解为我们日常生活中的空间. 比如物体的位置, 移动的轨迹, 都可以描述.

Definition 2 (Inner Product) An inner product $⟨\cdot ,\cdot ⟩$ on a real vector space $\mathbf{X}$ is a symmetric, bilinear, positive-definite function

$$\begin{array}{rl}& ⟨\cdot ,\cdot ⟩:\mathbf{X}\times \mathbf{X}\to \mathbf{R}\\ & ({\mathit{x}}^{\ast },\mathit{x})\mapsto ⟨{\mathit{x}}^{\ast },\mathit{x}⟩.\end{array}$$

(Positive-definite means $⟨\mathit{x},\mathit{x}⟩>0$ unless $\mathit{x}=\mathbf{0}$.)

1. $\mathrm{\forall }\mathbf{u}\in V,(\mathbf{u},\mathbf{u})\ge 0$, and $(\mathbf{u},\mathbf{u})=0\iff \mathbf{u}=\mathbf{0}$;
2. $\mathrm{\forall }\mathbf{u},\mathbf{v}\in V$, holds $(\mathbf{u},\mathbf{v})=(\mathbf{v},\mathbf{u})$;
3. $\mathrm{\forall }\mathbf{u},\mathbf{v},\mathbf{w}\in V$, and $\mathrm{\forall }a,b\in \mathbb{R}$ holds $(a\mathbf{u}+b\mathbf{v},\mathbf{w})=a(\mathbf{u},\mathbf{w})+b(\mathbf{v},\mathbf{w})$

满足以下三个性质的叫内积:

• 对称性
• 双线性性
• 正定性

更加一般的定义:

https://en.wikipedia.org/wiki/Inner_product_space#cite_note-FOOTNOTESchaefer199944-8

• Conjugate symmetry
• Linearity
• Positive definiteness

Definition 3 (Orthogonal) Two vectors ${\mathit{x}}^{\ast }$ and $\mathit{x}$ are orthogonal if their inner product is zero,

$$⟨{\mathit{x}}^{\ast },\mathit{x}⟩=0.$$

Geometrically, orthogonal means perpendicular.

The Euclidean norm in ${\mathbb{R}}^2$ is given by

$$\Vert \mathbf{u}\Vert =\sqrt{(\mathbf{x},\mathbf{x})}=\sqrt{{\left(x_1\right)}^2+{\left(x_2\right)}^2}.$$

Euclidean norm 就是日常我们会用到的勾股定理.

Vector space

1.18 Definition addition, scalar multiplication （Linear Algebra Done Right）

An addition on a set $V$ is a function that assigns an element $u+v\in V$ to each pair of elements $u,v\in V$.

A scalar multiplication on a set $V$ is a function that assigns an element $\lambda v\in V$ to each $\lambda \in \mathbf{F}$ and each $v\in V$.
Now we are ready to give the formal definition of a vector space.

1.19 Definition vector space （Linear Algebra Done Right）
A vector space is a set $V$ along with an addition on $V$ and a scalar multiplication on $V$ such that the following properties hold:
commutativity

$$u+v=v+u\text{ for all }u,v\in V$$

associativity

$$(u+v)+w=u+(v+w)\text{ and }(ab)v=a(bv)\text{ for all }u,v,w\in V$$

and all $a,b\in \mathbf{F}$;
additive identity
there exists an element $0\in V$ such that $v+0=v$ for all $v\in V$;
additive inverse
for every $v\in V$, there exists $w\in V$ such that $v+w=0$;
multiplicative identity

$$1v=v\text{ for all }v\in V$$

distributive properties
$a(u+v)=au+av$ and $(a+b)v=av+bv$ for all $a,b\in \mathbf{F}$ and all $u,v\in V$.

• 装备了:
  • 加法
  • 数乘
• 满足:
  • 加法
    • 交换律
    • 结合律
    • 加法单位元
    • 加法逆元
  • 数乘
    • 单位元
  • 数乘和加法一起
    • 分配律

Vector Norms

Definition A norm is a function $\Vert \cdot \Vert :{\mathbb{C}}^m\to \mathbb{R}$ that assigns a real-valued length to each vector. In order to conform to a reasonable notion of length, a norm must satisfy the following three conditions. For all vectors $x$ and $y$ and for all scalars $\alpha \in \mathbb{C}$,
(1) (Positive definiteness/Point-separating) $\Vert x\Vert \ge 0$, and $\Vert x\Vert =0$ only if $x=0$,
(2) (Subadditivity/Triangle inequality) $\Vert x+y\Vert \le \Vert x\Vert +\Vert y\Vert$,
(3) (Absolute homogeneity) $\Vert \alpha x\Vert =|\alpha |\Vert x\Vert$.
In words, these conditions require that (1) the norm of a nonzero vector is positive, (2) the norm of a vector sum does not exceed the sum of the norms of its parts-the triangle inequality, and (3) scaling a vector scales its norm by the same amount.****

Inner product 可以引出 Euclidean norm, 而一般的 norm 是 Euclidean norm 的括张.

• 正定性
• 次可加性/三角形不等式
• "数量乘法"/绝对齐次

Examples

$$\begin{array}{rl}\Vert x{\Vert }_1& =\sum _{i=1}^m\left|x_i\right|,\\ \Vert x{\Vert }_2& ={\left(\sum _{i=1}^m{\left|x_i\right|}^2\right)}^{1/2}=\sqrt{x^{\ast }x},\\ \Vert x{\Vert }_{\mathrm{\infty }}& =\underset{1\le i\le m}{max}\left|x_i\right|,\\ \Vert x{\Vert }_p& ={\left(\sum _{i=1}^m{\left|x_i\right|}^p\right)}^{1/p}(1\le p<\mathrm{\infty }).\end{array}$$

Remark Inner product is a kind of norm, but the reverse does not hold.

Metric space

• Definition A metric space $(X,d)$ consists of a non-empty set $X$ and a function $d:X\times X\to [0,\mathrm{\infty })$ such that:
  (i) (Positivity) For all $x,y\in X,d(x,y)\ge 0$ with equality if and only if $x=y$.
  (ii) (Symmetry) For all $x,y\in X,d(x,y)=d(y,x)$.
  (iii) (Triangle Inequality) For all $x,y,z\in X$
  $$d(x,y)\le d(x,z)+d(z,y)$$
  A function d satisfying conditions (i)-(iii), is called a metric on $X$.

Remark Distance in metric space is a kind of norm, but the reverse does not hold.

Metric space 是 vector space 的括张:

• 定义在任意集合上;
• distance 无法通过 norm 生成.
  • norm 是 distance 的"子标准".

Example The metrics in this example may seem rather strange. Although they are not very useful in applications, they are handy to know about as they are totally different from the metrics we are used to from ${\mathbb{R}}^n$ and may help sharpen our intuition of how a metric can be. Let $X$ be any non-empty set, and define:

$$d(x,y)=\{\begin{array}{ll}0& \text{ if }x=y\\ 1& \text{ if }x\ne y\end{array}$$

It is not hard to check that $d$ is a metric on $X$, usually referred to as the discrete metric.

Example If we let $d(x,y)=|x-y|,(\mathbb{R},d)$ is a metric space. The first two conditions are obviously satisfied, and the third follows from the ordinary triangle inequality for real numbers:

$$d(x,y)=|x-y|=|(x-z)+(z-y)|\le |x-z|+|z-y|=d(x,z)+d(z,y)$$

Definition (Convergence of a Sequence) Let $(X,d)$ be a metric space. A sequencee $\left\{x_n\right\}$ in $X$ converges to a point $a\in X$ if there for every $ϵ>0$ exists an $N\in \mathbb{N}$ such that $d(x_n,a)<ϵ$ for all $n\ge N$. We write $\underset{n\to \mathrm{\infty }}{lim}{x}_n=a$ or $x_n\to a$.

Note that this definition exactly mimics the definition of convergence in $\mathbb{R}$ or ${\mathbb{R}}^n$. Here is an alternative formulation.

Lemma A sequence $\left\{x_n\right\}$ in a metric space $(X,d)$ converges to $a$ if and only if $\underset{n\to \mathrm{\infty }}{lim}d(x_n,a)=0$.

Proof: The distances $\left\{d(x_n,a)\right\}$ form a sequence of nonnegative numbers. This sequence converges to 0 if and only if there for every $ϵ>0$ exists an $N\in \mathbb{N}$ such that $d(x_n,a)<ϵ$ when $n\ge N$. But this is exactly what the definition above says.

Proposition A sequence in a metric point can not converge to more than one point.

Proof (Triangular inequality) Omitted.

如果收敛, 只能收敛到唯一的点.

Definition (Continuous function) Assume that $(X,d_X),(Y,d_Y)$ are two metric spaces. $A$ function $f:X\to Y$ is continuous at a point $a\in X$ if for every $ϵ>0$ there is a $\delta >0$ such that $d_Y(f(x),f(a))<ϵ$ whenever $d_X(x,a)<\delta$.

Definition (Balls) Now we define some auxiliary notations. If $a$ is an element of a metric space $X$, and $r$ is a positive number, the (open) ball centered at a with radius $r$ is the set

$$\mathrm{B}(a;r)=\{x\in X\mid d(x,a)<r\}$$

The ball without center is

$${\mathrm{B}}_0(a;r)=\{x\in X\mid 0<d(x,a)<r\}$$

And the closed ball is

$$\overline{\mathrm{B}(a;r)}=\{x\in X\mid d(x,a)\le r\}$$

the closed ball without center

$$\overline{{\mathrm{B}}_0(a;r)}=\{x\in X\mid 0<d(x,a)<r\}$$

Definition (interior point, exterior point, boundary point) If $A$ is a subset of $X$ and $x$ is a point in $X$, there are three possibilities:
(i) There is a ball $\mathrm{B}(x;r)$ around $x$ which is contained in $A$. In this case $x$ is called an interior point of $A$.
(ii) There is a ball $\mathrm{B}(x;r)$ around $x$ which is contained in the complement $A^c$. In this case $x$ is called an exterior point of $A$.
(iii) All balls $\mathrm{B}(x;r)$ around $x$ contains points in $A$ as well as points in the complement $A^c$. In this case $x$ is a boundary point of $A$.

可以既不是内点又不是外点:

一个典型的例子是考虑单位圆的边界上的某个点。在二维平面上，单位圆由所有与原点的距离等于1的点组成。如果我们选择单位圆上的某个点P，那么它既不是内点也不是外点。

对于点P而言，它不是内点，因为它的邻域中包含了圆外的点。无论我们选择多小的半径，该邻域都会包含圆外的点。

另一方面，点P也不是外点，因为它的邻域中包含了圆内的点。无论我们选择多小的半径，该邻域都会包含圆内的点。

因此，单位圆上的边界点P既不是内点也不是外点。边界点处于集合内部和外部的交界处，既有内点的特性又有外点的特性。

Definition (limit point) If $A$ is a subset of $X$ and $x$ is a point in $X$, $x$ is a limit point if and only if for all $r>0$, $B(x;r)\cap X\ne \varnothing$.

• 内点：如果一个点是集合的内点，则它必定是该集合的限制点。因为内点定义为存在一个邻域，该邻域完全包含在集合中，所以对于任何 $r>0$，该邻域都与集合有非空交集，从而满足限制点的定义。
• 外点：如果一个点是集合的外点，则它不是该集合的限制点。因为外点定义为存在一个邻域，该邻域完全位于集合的补集中，所以对于任何 $r>0$，该邻域与集合的交集为空，不满足限制点的定义。
• 边界点：边界点既可以是限制点也可以不是限制点。一个边界点处于集合内部和外部的交界处。如果边界点同时满足限制点的定义，那么它也是限制点。

Definition (open set) A subset $A$ of a metric space $X$ is open if and only if it only consists of interior points, i.e. for all $a\in A$, there is a ball $\mathrm{B}(a;r)$ around a which is contained in $A$.

Definition (closed set) A subset $A$ of a metric space $X$ is closed if and only if it consists all limit points.

同样, 开集和闭集也不是反义词:

考虑度量空间 $\mathbb{R}$ 上的半开半闭区间 $[0,1)$，它既不是开集（因为不包含1的邻域），也不是闭集（因为不包含其边界点1）。

• 既不是也不是: 随便举例子.

类似地，全体实数集 $\mathbb{R}$ 既是开集又是闭集，因为它既包含了所有内点（开集的特性），又包含了所有限制点（闭集的特性）。

• the only sets that are both open and closed in ${\mathbb{R}}^n$ in the standard topology are the empty set and ${\mathbb{R}}^n$ itself.

Proposition (complement of open is closed) A subset $A$ of a metric space $X$ is open if and only if its complement $A^c$ is closed.

Proof If $A$ is open, it does not contain any of the (common) boundary points. Hence they all belong to $A^c$, and $A^c$ must be closed.

Conversely, if $A^c$ is closed, it contains all boundary points, and hence $A$ can not have any. This means that $A$ is open.

The following observation may seem obvious, but needs to be proved:

Lemma All open balls $\mathrm{B}(a;r)$ are open sets, while all closed balls $\bar{\mathrm{B}}(a;r)$ are closed sets.

Proof: We prove the statement about open balls and leave the other as an exercise. Assume that $x\in \mathrm{B}(a;r)$; we must show that there is a ball $\mathrm{B}(x;ϵ)$ around $x$ which is contained in $\mathrm{B}(a;r)$. If we choose $ϵ=r-d(x,a)$, we see that if $y\in \mathrm{B}(x;ϵ)$ then by the Triangle Inequality

$$d(y,a)\le d(y,x)+d(x,a)<ϵ+d(x,a)=(r-d(x,a))+d(x,a)=r$$

Thus $d(y,a)<r$, and hence $\mathrm{B}(x;ϵ)\subset \mathrm{B}(a;r)$

Proposition (limit of a sequence will not run out that set) Assume that $F$ is a subset of a metric space $X$. The following are equivalent:
(i) $F$ is closed.
(ii) If $\left\{x_n\right\}$ is a convergent sequence of elements in $F$, then the limit $a=\underset{n\to \mathrm{\infty }}{lim}{x}_n$ always belongs to $F$.

• 闭集一定包含了所有序列的极限趋向的点.

Proof Assume that $F$ is closed and that $a$ does not belong to $F$. We must show that a sequence from $F$ cannot converge to $a$. Since $F$ is closed and contains all its boundary points, $a$ has to be an exterior point, and hence there is a ball $\mathrm{B}(a;ϵ)$ around $a$ which only contains points from the complement of $F$. But then a sequence from $F$ can never get inside $\mathrm{B}(a,ϵ)$, and hence cannot converge to $a$.

Assume now that that $F$ is not closed. We shall construct a sequence from $F$ that converges to a point outside $F$. Since $F$ is not closed, there is a boundary point $a$ that does not belong to $F$. For each $n\in \mathbb{N}$, we can find a point $x_n$ from $F$ in $\mathrm{B}(a;\frac{1}{n})$. Then $\left\{x_n\right\}$ is a sequence from $F$ that converges to a point a which is not in $F$.

Remark An alternative statement is:

Proposition (limit of a sequence will not run out that set too far) Assume that $F$ is a subset of a metric space $X$.
If $\left\{x_n\right\}$ is a convergent sequence of elements in $F$, then the limit $a=\underset{n\to \mathrm{\infty }}{lim}{x}_n$ is a limit point of $F$.

Definition (Cauchy sequence) A sequence $\left\{x_n\right\}$ in a metric space $(X,d)$ is a Cauchy sequence if for each $ϵ>0$ there is an $N\in \mathbb{N}$ such that $d(x_n,x_m)<ϵ$ whenever $n,m\ge N$.

对于给定的度量空间，如果对于任意给定的正数，存在一个正整数 $N$，使得序列中任意两个索引大于等于 $N$ 的元素之间的距离小于该正数，那么该序列就是柯西序列。

换句话说，对于柯西序列中的元素，随着索引的增大，它们之间的距离逐渐缩小并趋近于零。这意味着序列中的元素在足够远的位置处变得非常接近，表明序列逐渐收敛到一个极限值。

Proposition Every convergent sequence is a Cauchy sequence.

收敛的序列是柯西序列, 但是柯西序列不一定收敛:

https://math.stackexchange.com/questions/2731681/convergent-and-cauchy-sequences-in-metric-spaces

In ${\mathbb{R}}^n$, every Cauchy sequence converges. This is a property called completeness; a metric space $X$ is complete if every Cauchy sequence converges. Thus, in a complete metric space, which ${\mathbb{R}}^n$ is, a sequence is Cauchy if and only if it converges.

For your second question, just take a non-complete metric space, say, $\mathbb{Q}\subset \mathbb{R}$, and consider a sequence of rational numbers that are converging to $\sqrt{2}$ in $\mathbb{R}$. Since $\sqrt{2}$ is not a rational number, this sequence is Cauchy, but it does not converge in $\mathbb{Q}$.

• 找一个收敛到无理数的有理数序列:
  • 是柯西序列, 因为确实在"收敛";
  • 但是收敛不到那个值, 因为集合不完备.

Definition (completeness) A metric space is called complete if all Cauchy sequences converge.

• 收敛序列一定是柯西序列, 这个是谁都有的性质;
• 具备 completeness 的 metric space 中所有的柯西序列收敛.

Definition $A$ subset $S$ of a metric space $(X,d)$ is bounded if there exist $x_0\in X$ and $K\in \mathbb{R}$ such that $d(x,x_0)⩽K$ for all $x\in S$.

If $S$ satisfies the definition for some $x_0\in X$ and $K\in \mathbb{R}$, then it also satisfies the definition with $x_0$ replaced by any other point $x_1\in X$ and $K$ replaced by $K+d(x_0,x_1)$. For if $d(x,x_0)⩽K$ then

$$d(x,x_1)⩽d(x,x_0)+d(x_0,x_1)⩽K+d(x_0,x_1).$$

If $S$ satisfies 5.23 then $d(x,y)⩽d(x,x_0)+d(x_0,y)⩽2K$ for all x. $y\in S$. The following definition therefore makes sense.

Definition (totally bounded) A subset $A$ of a metric space $X$ is called totally bounded if for each $ϵ>0$ there is a finite number $\mathrm{B}(a_1,ϵ),\mathrm{B}(a_2,ϵ),\dots ,\mathrm{B}(a_n,ϵ)$ of balls of radius $ϵ$ that cover $A$ (i.e. $A\subset \mathrm{B}(a_1,ϵ)\cup \mathrm{B}(a_2,ϵ)\cup \dots \cup \mathrm{B}(a_n,ϵ))$.

https://math.stackexchange.com/questions/1251548/set-that-is-bounded-but-not-totally-bounded-reading-textbook

Take $U=\{e_n\mid n\in \mathbb{N}\}\subset {\ell }^{\mathrm{\infty }}(\mathbb{R})$
It is obviously bounded since $\mathrm{\forall }x\in U\Vert x\Vert =1$, but $\mathrm{\forall }x,y\in {\ell }^{\mathrm{\infty }}$ we have $d(x,y)=1$, so obviously for $ϵ=1$ there is no finite number of open balls with radius $ϵ$ that cover $U$ - cause each ball would contain at most one member of $U$.

考虑一个无穷多个元素的结合, 不一样的元素距离为 1, 一样的元素距离为 0. 这样:

• 是有界;
• 但不是全有界, 因为对于半径小于 1 的球来说需要无穷个.

Definition 17.2 A metric space $X$ is complete if every Cauchy sequence in $X$ converges (to a point of $X$ ).
Example 17.3 (a) $\mathbb{R}$ is complete by Theorem 4.18.
(b) $\mathbb{Q}$ is not complete, for any sequence in $\mathbb{Q}$ which converges in $\mathbb{R}$ to an irrational number such as $\sqrt{2}$ is a Cauchy sequence in $\mathbb{Q}$ which does not converge to any point in $\mathbb{Q}$.
(c) $(0,1)\subseteq \mathbb{R}$ is not complete, for the sequence $(1/n)$ is a Cauchy sequence in $(0,1)$ which docs not converge to any point in $(0,1)$.

More examples of complete metric spaces, arising from the previous chapter, will be given shortly.

Example 17.3(a) and (c) show that completeness is not a topological property, since $(0,1)$ and $\mathbb{R}$ are homeomorphic. However, it is invariant under uniform equivalence, in the sense of the next proposition.因此，集合 $A=\{1,2,3\}$ 在离散度量下是有界的但不是全有界的。这个例子展示了有界性和全有界性之间的区别。

因此，全有界性是有界性的一个更强要求，仅有界并不意味着全有界。

Theorem A subset $A$ of a complete metric space $X$ is compact if and only if it is closed and totally bounded.

Remark Totally bounded means that $A$ can be covered by finite many balls whose $r$ can be arbitrarily small.

Totally bounded is stronger than bounded, recall that closed and bounded does not yield compactness.

• 在 complete metric space 中, 全有界和闭可以得到紧,
• 但是有界和闭就不行.

Proposition Assume that $(X,d)$ is a complete metric space. If $A$ is a subset of $X,(A,d_A)$ is complete if and only if $A$ is closed.

Definition (covering) Let $(X,d)$ be a metric space. A covering of $X$ is a collection of sets whose union is $X$.

Definition (open covering) An open covering of $X$ is a collection of open sets whose union is $X$.

Definition (compactness) The metric space $X$ is said to be compact if every open covering has a finite subcovering.

Remark This abstracts the Heine-Borel property; indeed, the Heine-Borel theorem states that closed bounded subsets of the real line are compact.
Heine-Borel theorem Every covering of a closed interval $[a,b]-$ or more generally of a closed bounded set $X\subset \mathbb{R}$ - by a collection of open sets has a finite subcovering.

Definition (Sequentially compact) A metric space $X$ is said to be sequentially compact if every sequence ${\left(x_n\right)}_{n=1}^{\mathrm{\infty }}$ of points in $X$ has a convergent subsequence.

This abstracts the Bolzano-Weierstrass property; indeed, the Bolzano-Weierstrass theorem states that closed bounded subsets of the real line are sequentially compact.

Bolzano-Weierstrass theorem Every bounded sequence of real numbers has a convergent subsequence.
This can be rephrased as:
Bolzano-Weierstrass theorem (rephrased) Let $X$ be any closed bounded subset of the real line. Then any sequence $\left(x_n\right)$ of points in $X$ has a subsequence converging to a point of $X$.

Theorem (Compactness of metric spaces) For a metric space $X$, the following are equivalent:
(a) $X$ is compact, i.e. every open covering of $X$ has a finite subcovering.
(b) Every collection of closed sets in $X$ with the finite intersection property has a nonempty intersection.
(c) If $F_1\supseteq F_2\supseteq F_3\supseteq \dots$ is a decreasing sequence of nonempty closed sets in $X$, then $\bigcap _{n=1}^{\mathrm{\infty }}{F}_n$ is nonempty.
(d) $X$ is sequentially compact, i.e. every sequence in $X$ has a convergent subsequence.
(e) $X$ is totally bounded and complete.

最重要的定理:

• 任意 (例如无限个) 开覆盖必定有有限个子覆盖能盖住 $X$.

Proof https://www.ucl.ac.uk/~ucahad0/3103_handout_2.pdf

Proposition

(a) A closed subset of a compact space is compact.
(b) A compact subset of any metric space is closed.
(c) A finite union of compact sets is compact.

Proposition (Compactness of subsets in $\mathbb{R}$ ) $A$ subset $A\subseteq \mathbb{R}$ is compact if and only if it is closed and bounded.
The corresponding result for ${\mathbb{R}}^n$ is an easy consequence:

Proposition (Compactness of subsets in ${\mathbb{R}}^n$ ) $A$ subset $A\subseteq {\mathbb{R}}^n$ is compact if and only if it is closed and bounded.

• 欧式空间是有限维度的;
• 欧氏空间中闭且有界意味着紧.

Corollary (existence of optimality) Let $X$ be a compact metric space, and let $f:X\to \mathbb{R}$ be continuous. Then $f[X]$ is bounded, and there exist points $a,b\in X$ such that $f(a)=\underset{x\in X}{inf}f(x)$ and $f(b)=$ $\underset{x\in X}{sup}f(x)$

Remark In a finite dimensional normed space, i.e. ${\mathbb{R}}^n$ or ${\mathbb{C}}^n$, a set is compact if and only if it is closed and bounded. But this is false when it comes to infinite dimensional spaces.

• 紧在优化相关理论中很重要, 他对最优的存在性做出了保证.

Example ${\ell }^{\mathrm{\infty }}$. The closed unit ball in ${\ell }^{\mathrm{\infty }}$ is not compact - indeed, it is not even separable (see Proposition 1.18 last week along with Proposition 2.3). ${}^5$

Example ${\mathbb{R}}^{\mathrm{\infty }}$ (which is not a Euclidean space) is not compact.

Proof To show that ${\mathbb{R}}^{\mathrm{\infty }}$ is not compact, we can construct an open cover of ${\mathbb{R}}^{\mathrm{\infty }}$ that does not have a finite subcover.

Consider the following open cover of ${\mathbb{R}}^{\mathrm{\infty }}$: $$\mathcal{U}=U_n:n\in \mathbb{N},$$ where $U_n$ is the open ball centered at the origin with radius $n$: $$U_n=(x_1,x_2,\dots )\in {\mathbb{R}}^{\mathrm{\infty }}:\sqrt{x_1^2+x_2^2+\cdots }<n.$$ To see that $\mathcal{U}$ is indeed an open cover of ${\mathbb{R}}^{\mathrm{\infty }}$, let $(x_1,x_2,\dots )\in {\mathbb{R}}^{\mathrm{\infty }}$ be arbitrary. Then, there exists $N\in \mathbb{N}$ such that $\sqrt{x_1^2+x_2^2+\cdots }<N$. Thus, $(x_1,x_2,\dots )\in U_N$, so $\mathcal{U}$ covers ${\mathbb{R}}^{\mathrm{\infty }}$.

To show that $\mathcal{U}$ does not have a finite subcover, suppose to the contrary that $\mathcal{V}=U_{n_1},U_{n_2},\dots ,U_{n_k}$ is a finite subcover of $\mathcal{U}$. Let $m=max{n}_1,n_2,\dots ,n_k$. Then, for any $(x_1,x_2,\dots )\in {\mathbb{R}}^{\mathrm{\infty }}$ with $\sqrt{x_1^2+x_2^2+\cdots }\ge m+1$, we have $(x_1,x_2,\dots )\notin U_n$ for any $n\in n_1,n_2,\dots ,n_k$. Thus, $\mathcal{V}$ cannot cover ${\mathbb{R}}^{\mathrm{\infty }}$.

Therefore, we have shown that ${\mathbb{R}}^{\mathrm{\infty }}$ is not compact.

Example The subset of ${\mathbb{R}}^{\mathrm{\infty }}$ is also not compact.

Remark To prove a set $X$ is not compact, we only need to find an open cover which does not have a finite subcover which covers $X$. Conversely, however, if we want to show that the set $X$ is compact, we have to show that all of its open covers have finite subcover.

The approximation problem and existence of best approximations

three main ingredients of an approximation calculation, which are as follows:

• A function, or some data, or more generally a member of a set, that is to be approximated. We call it $f$;
• A set, $\mathcal{A}$ say, of approximations, which in the case of the given examples is the set of all straight lines.
• A means of selecting an approximation from $\mathcal{A}$.

Definition (better approximation) $a_0\in \mathcal{A}$ is a better approximation than $a_1\in \mathcal{A}$ if the inequality

$$d(a_0,f)<d(a_1,f)$$

is satisfied.

Definition (best approximation) We define $a^{\ast }\in \mathcal{A}$ to be a best approximation if the condition

$$d(a^{\ast },f)⩽d(a,f)$$

holds for all $a\in \mathcal{A}$.

Theorem 1.1 (metric space: compactness $\Rightarrow$ optimality)
If $\mathcal{A}$ is a compact set in a metric space $\mathcal{B}$, then, for every $f$ in $\mathcal{B}$, there exists an element $a^{\ast }\in \mathcal{A}$, such that $d(a^{\ast },f)⩽d(a,f)$ holds for all $a\in \mathcal{A}$.

用来逼近的集合是紧的, 意味着对于任意一个目标, 都存在一个 (还没有唯一性) 最优逼近.

最优的存在是由紧保证的.

Theorem 1.2 (finite dimensional linear space: $\Rightarrow$ optimality)
If $\mathcal{A}$ is a finite-dimensional linear space in a normed linear space $\mathcal{B}$, then, for every $f\in \mathcal{B}$, there exists an element of $\mathcal{A}$ that is a best approximation from $\mathcal{A}$ to $f$.

Proof. Let the subset ${\mathcal{A}}_0$ contain the elements of $\mathcal{A}$ that satisfy the condition (create a closed and bounded condition)

$$\Vert a\Vert ⩽2\Vert f\Vert$$

It is compact because it is a closed and bounded subset of a finitedimensional space. It is not empty: for example it contains the zero element. Therefore, by Theorem 1.1, there is a best approximation from ${\mathcal{A}}_0$ to $f$ which we call $a_0^{\ast }$. By definition the inequality

$$\Vert a-f\Vert ⩾\Vert a_0^{\ast }-f\Vert ,a\in {\mathcal{A}}_0$$

holds. Alternatively, if the element $a$ is in $\mathcal{A}$ but is not in ${\mathcal{A}}_0$ then, because condition (1.13) is not obtained we have the bound

$$\begin{array}{rl}\Vert a-f\Vert & ⩾\Vert a\Vert -\Vert f\Vert \\ & >\Vert f\Vert \\ & ⩾\Vert a_0^{\ast }-f\Vert ,\end{array}$$

where the last line makes further use of the fact that the zero element is in ${\mathcal{A}}_0$. Hence expression $(1.14)$ is satisfied for all $a$ in $\mathcal{A}$, which proves that $a_0^{\ast }$ is a best approximation.

Some criteria: Norms

Pros and cons of various $L_P$- norms:

• 2-norm

  • continuous, calculus toolbox is usable

  • easy to calculate (actually a really helpful property)

  • meaningful, explainable in statistics

  • however, very sensitive when outliers exist

• 1-norm

  • good at deal with outliers
  • hard to calculate for non-smooth
    • recall subgradient

• $\mathrm{\infty }$-norm

  • considered as a overall control, since:

Definition $\mathcal{C}[a,b]$ is all real-valued functions that are defined on the interval $[a,b]$ of the real line.

关于闭区间上的连续函数, 他们的 norm 定义如下:

For finite $p$ the $L_p$ norm in $\mathcal{C}[a,b]$ is defined to have the value

$$\Vert f{\Vert }_p={[{\int }_a^b|f(x){|}^p\mathrm{d}x]}^{1/p},1⩽p<\mathrm{\infty }$$

and in ${\mathcal{R}}^m$ it has the value

$$\Vert f{\Vert }_p={\left[\sum _{i=1}^m{\left|y_i\right|}^p\right]}^{1/p},1⩽p<\mathrm{\infty }$$

where $\{y_i;i=1,2,\dots ,m\}$ are the components of $f$. The $\mathrm{\infty }$-norms are the expressions

$$\Vert f{\Vert }_{\mathrm{\infty }}=\underset{a⩽x⩽b}{max}|f(x)|$$

and

$$\Vert f{\Vert }_{\mathrm{\infty }}=\underset{1⩽i⩽m}{max}\left|y_i\right|$$

respectively.

Theorem 1.3
For all $e$ in $\mathcal{C}[a,b]$ the inequalities

$$\Vert e{\Vert }_1⩽(b-a{)}^{\frac{1}{2}}\Vert e{\Vert }_2⩽(b-a)\Vert e{\Vert }_{\mathrm{\infty }}$$

hold.

Remark if we can control $\mathrm{\infty }$-norm in a certain range, then the others like 1-norm and 2-norm are controlled consequently. But the reverse is not true, if we control some $p$-norms to a relative small value, the $\mathrm{\infty }$-norm may not even change.

The uniqueness of best approximations

Theorem 2.1
Let $\mathcal{B}$ be a normed linear space. Then, for any $f\in \mathcal{B}$ and for any $r>0$, the ball

$$\mathcal{N}(f,r)=\{x:\Vert x-f\Vert ⩽r,x\in \mathcal{B}\}$$

is convex.

Proof ...

Theorem 2.2
Let $\mathcal{A}$ be a convex set in a normed linear space $\mathcal{B}$, and let $f$ be any point of $\mathcal{B}$ such that there exists a best approximation from $\mathcal{A}$ to $f$. Then the set of best approximations is convex.

Conditions for uniqueness of the best approximation

Theorem 2.3
Let $\mathcal{A}$ be a compact and strictly convex set in a normed linear space $\mathcal{B}$. Then, for all $f\in \mathcal{B}$, there is just one best approximation from $\mathcal{A}$ to $f$.

Definition (convex norm) The norm is defined to be strictly convex if and only if the unit ball centered on the origin, namely $\mathcal{N}(0,1)$, is strictly convex.

Theorem 2.4
Let $\mathcal{A}$ be a convex set in a normed linear space $\mathcal{B}$, whose norm is strictly convex. Then, for all $f\in \mathcal{B}$, there is at most one best approximation from $\mathcal{A}$ to $f$.

两种唯一的情况, 两边只要有一个是"严格"的就能导致唯一性.

• 严格凸的逼近集合和任意 norm;
• convex norm 和一个普通的凸集合;

Theorem 2.5
Let $A$ be a compact set in a metric space $B$, such that for every $f$ in $B$ there is only one best approximation in $A,X(f)$ say. Then the operator $X$, defined by the best approximation condition, is continuous.

• 逼近算子是连续的.

Proof (hard)