理论推导中常用到的一些概念
An unbiased risk estimator vs. The same minimizer
- 矫正后的 loss 可以导致一个无偏的（consistent）估计，
\[\mathbb E_{p(x,\tilde{y})}[\ell_{correct}^1(h(x),\tilde{y})] = \mathbb E_{p(x,y)}[\ell(h(x),y)]，\forall\,h
\]
- 两者有相同的 minimizer:
\[\mathop{\arg\!\min}_h\mathbb E_{p(x,\tilde{y})}[\ell_{correct}^2(h(x),\tilde{y})] = \mathop{\arg\!\min}_h\mathbb E_{p(x,y)}[\ell(h(x),y)]
\]
上述条件 (2) 比 （1）要弱一些：
- (1) 可以推出（2）
- 在 (1) 不成立的情形下，（2）有可能成立。
- (2) 一般是在保留相同的 bayesian optimal classifier
Reference:
- NIPS-13. Learning with noisy label
- CVPR-17. Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach
- ICML-20. Does Label Smoothing Mitigate Label Noise?
- ICML-20. Learning with Multiple Complementary Labels
Statistically consistent, classifier-consistency, risk-consistent
统计相容性，更多的考虑，在 \(n\to\infty\) 的情况下，两者之间的差别
Reference:
- NIPS-19 Are anchor points really indispensable in label-noise learning?（提到 risk-consistent, classifier consistent）
- ICML-20 Does Label Smoothing Mitigate Label Noise? (提到 classification consistency）
- ICML-13 On the statistical Consistency of Algorithms for Binary Classification under Class Imbalance
- ICML-20. Progressive Identification of True Labels for Partial-Label Learning (classifier-consistency)
- ICML-20 Learning with multiple complementary labels (classifier-consistency, risk-consistent)
- NIPS-20 Provably consistent partial-label learning (risk consistent, classifier-consistent)
Excess risk bound vs. Generalization bound vs. learnability
(1). Excess risk 主要考虑的是当前 (ERM 算法所导出)分类器 与 最优的分类器的泛化误差的 gap
(2). Generalization bound 考虑的是经验误差与泛化误差的 uniform 的 gap，对假设空间中的所有假设同时成立，因此需要用 Rademacher complexity or VC dim 来刻画假设空间的复杂度。
(3). 有了 generalization bound, 就非常容易导出 excess risk bound, 几乎就是两倍的关系。（参见 Foundations of ML (2nd) Proposition 4.1 ）
(4). 可学习性考虑的是 ERM 算法输出的分类器的泛化误差 与 最优的分类器的泛化误差之间的 gap，其实就是 Excess risk。
参考文献：
- ICML20. Class-Weighted Classification: Trade-offs and Robust Approaches.
- ICML20. Learning with Bounded Instance- and Label-dependent Label Noise.
Plug-in classifiers
Reference
- NIPS09
- ICML20.
- ICML20. Class-Weighted Classification: Trade-offs and Robust Approaches
- 之前审稿的 rejection paper
Loss unbounded below 导致 overfit
不同于 0-1 error， 凸 loss 通常是无界的，会导致给与 outlier 过大的权重
Reference:
- NIPS-09
- ICML-20. Learning with Multiple Complementary Labels
- NIPS-19. Robust Bi-Tempered Logistic Loss Based on Bregman Divergences
0-1 loss non-convex, non-smooth
Bayes classifier 其实是在优化 0-1 loss， 也就是在优化错误的概率。
Reference:
- NeuroComputing-15. Making Risk Minimization Tolerant to Label Noise
