Retrofitting Analysis

Retrofitting Analysis

To figure out the process of retrofitting[1] objective updating, we do the following math.

Forward Derivation

\[\psi(Q) = \sum_{i=1}^{n}\left[ \alpha_i||q_i-\hat{q_i}||^2 + \sum\beta||q_i-q_j||^2 \right] \\ \frac{\partial \psi(Q)}{\partial q_i} = \alpha_i(q_i-\hat{q_i}) + \sum\beta(q_i-q_j) = 0 \\ (\alpha_i+\sum\beta_{ij})q_i -\alpha_i\hat{q_i} -\sum\beta_{ij}q_j = 0 \\ q_i = \frac{\sum\beta_{ij}q_j+\alpha_i\hat{q_i}}{\sum\beta_{ij}+\alpha_i} \]

Backward Derivation

This is how I understood this updating equation.

In the paper[1], it has mentioned "We take the first derivative of \(\psi\) with respect to one qi vector, and by equating it to zero", hence we get follow idea:

\[\frac{\partial\psi(Q)}{\partial q_i} = 0 \]

And,

\[q_i = \frac{\sum\beta_{ij}q_j+\alpha_i\hat{q_i}}{\sum\beta_{ij}+\alpha_i} \\ \alpha_iq_i - \alpha_i\hat{q_j} + \sum\beta_{ij}q_i - \sum\beta q_j = 0 \\ \alpha_i(q_i-\hat{q_j})+ \sum\beta_{ij}(q_i-q_j) = 0 \]

Apparently,

\[\frac{\partial\psi(Q)}{\partial q_i} = \alpha_i(q_i-\hat{q_j})+ \sum\beta_{ij}(q_i-q_j) = 0 \]

Reference

Faruqui M, Dodge J, Jauhar S K, et al. Retrofitting Word Vectors to Semantic Lexicons[J]. ACL, 2015.