DualGNN: Dual Graph Neural Network for Multimedia Recommendation

目录

• 概
• 符号说明
• DualGCN
• 代码

Wang Q., Wei Y., Yin J., Wu J., Song X. and Nie L. DualGNN: Dual graph neural network for multimedia recommendation. IEEE Transactions on Multimedia, 2023.

概

多模态 + user co-occureence graph -> recommendation.

文章中提到的 modality-missing 问题挺好的, 但是我并没有看到有什么特别的设计是解决这个问题的.

符号说明

• \(\mathcal{U}\), users, \(|\mathcal{U}| = N\);
• \(\mathcal{I}\), micor-videos, \(|\mathcal{I}| = M\);
• \(\mathcal{G} = (\mathcal{V}, \mathcal{E})\), interaction graph, \(\mathcal{V} = \mathcal{U} \cup \mathcal{I}\);
• \(m \in \mathcal{M} = \{v, a, t\}\), modality, \(v\): visual, \(a\): acoustic, \(t\): textual;
• \(\bm{u}_m^{(0)}\), 随机初始化的 user embedding for modality \(m\);
• \(\bm{i}_m^{(0)}\), modality features.

DualGCN

• 首先, 通过 user-item interaction graph, 对 user/item 的信息进行初步的转换, 这部分对于每个模态是独立的:

  \[\bm{u}_m^{(l+1)} = \sum_{i \in \mathcal{N}_u} \frac{ 1 }{ \sqrt{|\mathcal{N}_u|} \sqrt{|\mathcal{N}_i|} }\bm{i}_m^{(l)}, \\ \bm{i}_m^{(l+1)} = \sum_{i \in \mathcal{N}_u} \frac{ 1 }{ \sqrt{|\mathcal{N}_u|} \sqrt{|\mathcal{N}_i|} }\bm{i}_m^{(l)}, \]

  然后, 对各层求和:

  \[\bm{u}_m = \sum_{l=0}^L \bm{u}_m^{(l)}, \quad \bm{i}_m = \sum_{l=0}^L \bm{i}_m^{(l)}. \]

• 然后是, 多模态信息的一个融合, 用于得到用户的一个表征, 作者给了三种方案:

  1. Attentively concatenation construction:
     \[\bm{u}_{mul} = \bm{W}_m \bm{h}_u + \bm{b}_m, \\ \bm{h}_u = \alpha_{u, v} \bm{u}_v \| \alpha_{u, a} \bm{u}_a \| \alpha_{u, t} \bm{u}_t. \]

  2. Attentively sum construction:
     \[\bm{u}_{mul} = \alpha_{u,v} \bm{u}_v + \alpha_{u, a} \bm{u}_a + \alpha_{u, t} \bm{u}_t. \]

  3. Attentively maximum construction:
     \[\bm{u}_{mul} = \max( \alpha_{u, v} \bm{u}_v, \alpha_{u,a} \bm{u}_a, \alpha_{u, t} \bm{u}_t ). \]

• 接着 \(\bm{u}_{mul}\) 还要通过 user-user graph 来进行进一步的转换. 构造 co-occurrence graph, 对于每个 user, 作者取它的 top-K 个 frequent users 作为邻居, 便的权重赋为 \(1\), 其它为 0. 接下来通过下列二者之一的方式进行转换:

  1. Mean aggregation:
     \[\bm{u}_{mul}^{(l'+1)} = \bm{u}_{mul}^{(l')} + \sum_{u' \in \mathcal{N}_{u, c}} \frac{1}{|\mathcal{N}_{u, c}|} {\bm{u}_{mul}'}^{(l')}, \]
     其中 \(\mathcal{N}_{u,c}\) 表示 user \(u\) 的 top-\(K\) 邻居.
  2. Softmax weighted aggregation:
     \[\bm{u}_{mul}^{(l'+1)} = \bm{u}_{mul}^{(l')} + \sum_{u' \in \mathcal{N}_{u, c}} \frac{ \exp(C_{u, u'}) }{ \sum_{u' \in \mathcal{N}_{u, c}} \exp(C_{u, u'}) } {\bm{u}_{mul}'}^{(l')}, \]
     其中 \(C_{u, u'}\) 表示 users \(u, u'\) 的 co-occurrence times.

• 最后得到最终 user/item 的表示:

  \[\bm{u}^* = \bm{u}_{mul}^{(L')}, \quad \bm{i}^* = \bm{i}_v + \bm{i}_a + \bm{i}_t. \]

  然后 score 的预测通过内积得到

  \[y_{u,i} = {\bm{u}^*}^T \bm{i}^*. \]

• 训练通过 BPR 损失.

代码

[official]