论文中矩阵的各种trick
最小二乘中优化顺序交换
\[XQc\to XCq
\]
其中\(X\in R^{n,p},\,Q\in R^{p,k},\, c\in R^{k,1},\, C\in R^{p, kp},\, q\in R^{kp, 1}\)，\(n\)表示样本数，\(p\)表示特征维度，\(c\)表示pattern的个数。
推导过程如下：
\[\begin{align*}(XQc)_{r, 1}&=\sum_l(XQ)_{r,l}\,c_{l,1}\\&=\sum_l \sum_m X_{r,m}Q_{m,l}\,c_{l,1}\\&=\sum_m\sum_l X_{r, m} \,c_{l,1}Q_{m,l}\end{align*}
\]
令\(m\)先固定，如\(m=1\)，\(l=\{1, \cdots ,k\}\)上式为
\[X_{r,1}c_{(1,\cdots,k),1}Q_{1, (1,\cdots,k)}
\]
可以看到\(c\)对\(Q\)每一行都做了矩阵乘法，而\(m\)是变动的，不难可以想象出\(I\otimes c.T\)，\(Q\)要拉成已为向量。代码如下：
import numpy as np
np.random.seed(10)
X = np.random.randn(5, 10)
Q = np.random.randn(10, 3)
c = np.random.randn(3, 1)
# original
result = X.dot(Q.dot(c))
# transformed
I = np.eye(10)
q = Q.reshape(-1, 1)
#
C = np.outer(I, c.T).reshape(10, -1)
# C = np.multiply.outer(I, c.T).reshape(10, -1)
# C = np.einsum('ab, cd->abcd', I, c.T).reshape(10, -1)
new_result = X.dot(C.dot(q))
print("变换是否等价"， np.allclose(result, new_result, rtol=1e-4))
MMD
\[\Vert \frac{1}{n_s}\sum\limits_{i=1}^{n_s}A^Tx_i-\frac{1}{n_t}\sum\limits_{i=n_s}^{n_s+n_t}A^Tx_i\Vert_2^2
\]
例子来源于[1]，具体的过程再赘述。
\[\begin{align*}\Vert \frac{1}{n_s}\sum\limits_{i=1}^{n_s}A^Tx_i-\frac{1}{n_t}\sum\limits_{i=n_s}^{n_s+n_t}A^Tx_i\Vert_2^2&=\Vert \frac{1}{n_s}A^T X_s {\mathbf 1}-\frac{1}{n_t}A^T X_t{\mathbf 1}\Vert_2^2\\&=tr\left(A^T\left(\begin{bmatrix}X_s & X_t\end{bmatrix}\begin{bmatrix}\frac{1}{n_s}^2 & -\frac{1}{n_sn_t}\\-\frac{1}{n_sn_t}& \frac{1}{n_t}^2\end{bmatrix}\begin{bmatrix}X_s \\ X_t\end{bmatrix}\right)A\right)\\&=tr(A^TXMX^TA)\end{align*}
\]
Scatter Matrix
\[AHA^T
\]
Scatter matrix可以度量矩阵的方差[2]，\(H=I-\frac{1}{n}{\mathbf 1}{\mathbf 1}^T\)方差的定义是
\[\begin{align*}var&=\sum_{i=1}^n (x_i-m)(x_i-m)^T\\&=\sum_{i=1}^nx_ix_i^T-n\bar{x}\bar{x}^T\end{align*}
\]
其中\(\bar{x}=\frac{1}{n}\sum x_i\)。令\(X=\begin{bmatrix} x_1& \cdots&x_n\end{bmatrix}\)，则上式可以表达为
\[\begin{align*}var &= XIX^T-X(\frac{1}{n}{\mathbf 1}{\mathbf 1}^T)X^T\end{align*}
\]
