contrastive loss

昨天Ke Kiaming大神的新文章 MoCo问世,reID中的contrastive loss逐渐往上游影响。自己对这一块一直没有一个总结梳理，趁着学习这篇文章的机会整理一下，挖个坑慢慢填

---

Distance metric learning aims to learn an embedding representation of the data that preserves the distance between similar data points close and dissimilar data points far on the embedding space¹

1. Improved Deep Metric Learning with Multi-class N-pair Loss Objective [NIP2016] [pdf] [code-pytorch]

• 定义： 

• • 样本数据$x \in \mathcal{X}$，标签为$y \in \{1,2,...,L\}$，$x_+,x_-$分别表述输入样本的正负样本对（同类/不同类）；$f(.;\theta): \mathcal{X} \rightarrow \mathbb{R}^K$表示feature embedding，$f(x)$是feature embedding vector;$m$是margin
  • Contrastive loss takes pairs of examples as input and trains a network to predict whether two inputs are from the same class or not: $$\mathcal{L}^{m}_{cont}(x_i,x_j;f)=\mathbb{1}\{y_i=y_j\}||f_i-f_j||_2^2+\mathbb{1}\{y_i \ne y_j\}max(0,m-||f_i-f_j||_2^2)^2$$

  • Triplet loss shares a similar spirit to contrastive loss, but is composed of triplets, each consisting of a query, a positive example (to the query), and a negative example:$$\mathcal{L}^{m}_{cont}(x,x_+,x_-;f)=max(0, ||f-f_+||_2^2+||f-f_-||_2^2)^2+m$$

• 方法

  • 针对之前的相关方法每次只有一个负样本训练存在的收敛慢、局部最优（可通过hard negative mining解决）的问题，提出了multi-class N-pair loss和negative class mining
  • multi-class N-pair loss
    • 每个batch选$N$个class，每个class选一对样本，即$\{(x_1, x_1^+),...,(x_N, x_N^+)\}$，建立N个tuplets: $\{S_i\}_{i=1}^N$,其中$S_i={x_i, x_1^+, x_2^+, ..., x_N^+}$构成了一对正样本和N-1对负样本
    • 损失函数$$\mathcal{L}_{N-pair-mc}(\{(x_i,x_i^+)\}_{i=1}^N;f)= \frac{1}{N} \sum_{i=1}^N log(1+\sum_{j \ne i} exp(f_i^T f_j^+ - f_i^T f_i^+))$$
    • 而$$log(1+\sum_{i=1}^{L-1} exp(f^T f_i - f^T f^+))=-log \frac{exp(f^T f^+)}{exp(f^T f^+)+ \sum_{i=1}^{L-1}exp(f^T f_i )}$$ 和softmax非常像！
  • negative class mining
    1. Evaluate Embedding Vectors: choose randomly a large number of output classes C; for each class, randomly pass a few (one or two) examples to extract their embedding vectors.
    2. Select Negative Classes: select one class randomly from C classes from step 1. Next, greedily add a new class that violates triplet constraint the most w.r.t. the selected classes till we reach N classes. When a tie appears, we randomly pick one of tied classes. 这一步不是很懂，大概是每次从剩下类中找最hard的类的sample，直到N？

    3. Finalize N-pair: draw two examples from each selected class from step 2.

2. Unsupervised Feature Learning via Non-Parametric Instance Discrimination [pdf] [code-pytorch]

 

3. Momentum Contrast for Unsupervised Visual Representation Learning