线性判别分析(fisher)

线性判别分析

线性判别分析中有降维，把数据都投影到同一条线上，然后在直线上取一个阈值，将直线分成两条射线，每一条代表一个分类。会损失一些数据信息，但如果这些信息是一些干扰信息，丢失也未尝不是好事。

线性判别分析之后的结果是一个向量，其他的不行吗？

主要指导思想(目标)：类内小，类间大。

公式推导

我们得到的是向量，为了方便计算损失，不妨设$||ww||=1$，每一个数据$X{X}_i$看作一个向量。那么$X{X}_{i}ww$是每个数据在$ww$方向上的投影。与$ww$的其中一个平面是划分平面。

两个不同类别分别命名为$C_1$和$C_2$，用$\mu \mu$,$\mu {\mu }_{C_1}$, $\mu {\mu }_{C_2}$分别代表全部数据，$C_1$数据,$C_2$数据的均值，用$\mathrm{\Sigma }\mathrm{\Sigma }$,$\mathrm{\Sigma }{\mathrm{\Sigma }}_{C_1}$, $\mathrm{\Sigma }{\mathrm{\Sigma }}_{C_2}$分别代表全部数据，$C_1$数据,$C_2$数据的协方差矩阵。
$\tilde{\mu }$和${\tilde{\sigma }}^2$表示投影的均值和方差。

$$\begin{array}{ccl}\mu \mu & =& \frac{1}{N}\sum _1^{N}X{X}_i\\ \mu {\mu }_{C_1}& =& \frac{1}{N_{C_1}}\sum _1^{N_{C_1}}X{X}_{C_{1}i}\\ \mu {\mu }_{C_2}& =& \frac{1}{N_{C_2}}\sum _1^{N_{C_2}}X{X}_{C_{1}i}\\ \mathrm{\Sigma }\mathrm{\Sigma }& =& \frac{1}{N}\sum _1^N(X{X}_i-\mu \mu )(X{X}_i-\mu \mu {)}^T\\ \mathrm{\Sigma }{\mathrm{\Sigma }}_{C_1}& =& \frac{1}{N_{C_1}}\sum _1^{N_{C_1}}(X{X}_{C_{1}i}-\mu {\mu }_{C_1})(X{X}_{C_{1}i}-\mu {\mu }_{C_1}{)}^T\\ \mathrm{\Sigma }{\mathrm{\Sigma }}_{C_2}& =& \frac{1}{N_{C_2}}\sum _1^{N_{C_2}}(X{X}_{C_{2}i}-\mu {\mu }_{C_2})(X{X}_{C_{2}i}-\mu {\mu }_{C_2}{)}^T\end{array}$$

$$\begin{array}{ccl}\tilde{\mu }& =& \frac{1}{N}\sum _1^{N}X{X}_i\theta \theta \\ {\tilde{\mu }}_{C_1}& =& \frac{1}{N_{C_1}}\sum _1^{N_{C_1}}X{X}_{C_{1}i}\theta \theta \\ {\tilde{\mu }}_{C_2}& =& \frac{1}{N_{C_2}}\sum _1^{N_{C_2}}X{X}_{C_{2}i}\theta \theta \\ {\tilde{\sigma }}^2& =& \frac{1}{N}\sum _1^N(X{X}_i\theta \theta -\tilde{\mu }{)}^2\\ {\tilde{\sigma }}_{C_1}^2& =& \frac{1}{N_{C_1}}\sum _1^{N_{C_1}}(X{X}_{C_{1}i}\theta \theta -{\tilde{\mu }}_{C_1}{)}^2\\ {\tilde{\sigma }}_{C_2}^2& =& \frac{1}{N_{C_2}}\sum _1^{N_{C_2}}(X{X}_{C_{2}i}\theta \theta -{\tilde{\mu }}_{C_2}{)}^2\end{array}$$

类间：$({\tilde{\mu }}_{C_1}-{\tilde{\mu }}_{C_2}{)}^2$

类内：${\tilde{\sigma }}_{C_1}^2+{\tilde{\sigma }}_{C_2}^2$

目标函数:$J(\theta \theta )=\frac{({\tilde{\mu }}_{C_1}-{\tilde{\mu }}_{C_2}{)}^2}{{\tilde{\sigma }}_{C_1}^2+{\tilde{\sigma }}_{C_2}^2}$

$$\begin{array}{ccl}J(\theta \theta )& =& \frac{({\tilde{\mu }}_{C_1}-{\tilde{\mu }}_{C_2}{)}^2}{{\tilde{\sigma }}_{C_1}^2+{\tilde{\sigma }}_{C_2}^2}\\ \mathrm{分}\mathrm{子}& =& ({\tilde{\mu }}_{C_1}-{\tilde{\mu }}_{C_2}{)}^2\\ & =& (\frac{1}{N_{C_1}}\sum _1^{N_{C_1}}X{X}_{C_{1}i}\theta \theta -\frac{1}{N_{C_2}}\sum _1^{N_{C_2}}X{X}_{C_{2}i}\theta \theta {)}^2\\ & =& ((\mu {\mu }_{C_1}-\mu {\mu }_{C_2})\theta \theta {)}^2\\ & =& \theta {\theta }^T(\mu {\mu }_{C_1}-\mu {\mu }_{C_2}{)}^T(\mu {\mu }_{C_1}-\mu {\mu }_{C_2})\theta \theta \\ {\tilde{\sigma }}_{C_1}^2& =& \frac{1}{N_{C_1}}\sum _1^{N_{C_1}}(X{X}_{C_{1}i}\theta \theta -{\tilde{\mu }}_{C_1}{)}^2\\ & =& \frac{1}{N_{C_1}}\sum _1^{N_{C_1}}(X{X}_{C_{1}i}\theta \theta -\frac{1}{N_{C_1}}\sum _1^{N_{C_1}}X{X}_{C_{1}i}\theta \theta {)}^2\\ & =& \frac{1}{N_{C_1}}\sum _1^{N_{C_1}}((X{X}_{C_{1}i}-\frac{1}{N_{C_1}}\sum _1^{N_{C_1}}X{X}_{C_{1}i})\theta \theta {)}^2\\ & =& \frac{1}{N_{C_1}}\sum _1^{N_{C_1}}((X{X}_{C_{1}i}-\mu {\mu }_{C_1})\theta \theta {)}^2\\ & =& \frac{1}{N_{C_1}}\sum _1^{N_{C_1}}\theta {\theta }^T(X{X}_{C_{1}i}-\mu {\mu }_{C_1}{)}^T(X{X}_{C_{1}i}-\mu {\mu }_{C_1})\theta \theta \\ & =& \theta {\theta }^T(\frac{1}{N_{C_1}}\sum _1^{N_{C_1}}(X{X}_{C_{1}i}-\mu {\mu }_{C_1}{)}^T(X{X}_{C_{1}i}-\mu {\mu }_{C_1}))\theta \theta \\ & =& \theta {\theta }^T\mathrm{\Sigma }{\mathrm{\Sigma }}_{C_1}\theta \theta \\ {\tilde{\sigma }}_{C_2}^2& =& \theta {\theta }^T\mathrm{\Sigma }{\mathrm{\Sigma }}_{C_2}\theta \theta \\ \mathrm{分}\mathrm{母}& =& \theta {\theta }^T\mathrm{\Sigma }{\mathrm{\Sigma }}_{C_1}\theta \theta +\theta {\theta }^T\mathrm{\Sigma }{\mathrm{\Sigma }}_{C_2}\theta \theta \\ & =& \theta {\theta }^T(\mathrm{\Sigma }{\mathrm{\Sigma }}_{C_1}+\mathrm{\Sigma }{\mathrm{\Sigma }}_{C_2})\theta \theta \end{array}$$

$\therefore$

$\begin{array}{ccl}J(\theta \theta )& =& \frac{\theta {\theta }^T(\mu {\mu }_{C_1}-\mu {\mu }_{C_2}{)}^T(\mu {\mu }_{C_1}-\mu {\mu }_{C_2})\theta \theta }{\theta {\theta }^T(\mathrm{\Sigma }{\mathrm{\Sigma }}_{C_1}+\mathrm{\Sigma }{\mathrm{\Sigma }}_{C_2})\theta \theta }\end{array}$

设$S_b=(\mu {\mu }_{C_1}-\mu {\mu }_{C_2}{)}^T(\mu {\mu }_{C_1}-\mu {\mu }_{C_2})$,$S_w=\mathrm{\Sigma }{\mathrm{\Sigma }}_{C_1}+\mathrm{\Sigma }{\mathrm{\Sigma }}_{C_2}$

$S_b$就是类内方差

$S_w$就是类间方差

此时$J(\theta \theta )=\frac{\theta {\theta }^T{S}_b\theta \theta }{\theta {\theta }^T{S}_w\theta \theta }$

求导

$$\begin{array}{rcl}\frac{\partial J(\theta \theta )}{\partial \theta \theta }& =& \frac{\partial \frac{\theta {\theta }^{T}S{S}_b\theta \theta }{\theta {\theta }^{T}S{S}_w\theta \theta }}{\partial \theta \theta }\\ & =& \frac{\partial (\theta {\theta }^{T}S{S}_b\theta \theta (\theta {\theta }^{T}S{S}_w\theta \theta {)}^{-1})}{\partial \theta \theta }\\ & =& \frac{\partial (\theta {\theta }^{T}S{S}_b\theta \theta )}{\partial \theta \theta }(\theta {\theta }^{T}S{S}_w\theta \theta {)}^{-1}+\theta {\theta }^{T}S{S}_b\theta \theta \frac{\partial ((\theta {\theta }^{T}S{S}_w\theta \theta {)}^{-1})}{\partial \theta \theta }\\ & =& 2\theta {\theta }^{T}S{S}_b(\theta {\theta }^{T}S{S}_w\theta \theta {)}^{-1}+\theta {\theta }^{T}S{S}_b\theta \theta (-\frac{1}{(\theta {\theta }^{T}S{S}_w\theta \theta {)}^2})(2\theta {\theta }^{T}S{S}_w)\end{array}$$

令导数等于零

$$\begin{array}{rcl}00& =& 2S{S}_b\theta \theta (\theta {\theta }^{T}S{S}_w\theta \theta {)}^{-1}+\theta {\theta }^{T}S{S}_b\theta \theta (-\frac{1}{(\theta {\theta }^{T}S{S}_w\theta \theta {)}^2})(2S{S}_w\theta \theta )\\ 2S{S}_b\theta \theta (\theta {\theta }^{T}S{S}_w\theta \theta {)}^{-1}& =& \theta {\theta }^{T}S{S}_b\theta \theta (\frac{1}{(\theta {\theta }^{T}S{S}_w\theta \theta {)}^2})(2S{S}_w\theta \theta )\\ S{S}_b\theta \theta (\theta {\theta }^{T}S{S}_w\theta \theta )& =& (\theta {\theta }^{T}S{S}_b\theta \theta )S{S}_w\theta \theta \\ (\theta {\theta }^{T}S{S}_b\theta \theta )S{S}_w\theta \theta & =& S{S}_b\theta \theta (\theta {\theta }^{T}S{S}_w\theta \theta )\\ \theta \theta & =& S{S}_w^{-1}\frac{\theta {\theta }^{T}S{S}_w\theta \theta }{\theta {\theta }^{T}S{S}_b\theta \theta }S{S}_b\theta \theta \\ \theta \theta & =& S{S}_w^{-1}\frac{\theta {\theta }^{T}S{S}_w\theta \theta }{\theta {\theta }^{T}S{S}_b\theta \theta }(\mu {\mu }_{C_1}-\mu {\mu }_{C_2}{)}^T(\mu {\mu }_{C_1}-\mu {\mu }_{C_2})\theta \theta \end{array}$$

$\because$
$\frac{\theta {\theta }^{T}S{S}_w\theta \theta }{\theta {\theta }^{T}S{S}_b\theta \theta }$,$(\mu {\mu }_{C_1}-\mu {\mu }_{C_2})\theta \theta$是一个数，不影响$\theta \theta$的方向

$\therefore$
$\theta \theta \propto S{S}_w^{-1}(\mu {\mu }_{C_1}-\mu {\mu }_{C_2}{)}^T$

$\mathcal{if}S{S}_w\propto II$

$\theta \theta \propto (\mu {\mu }_{C_1}-\mu {\mu }_{C_2}{)}^T$

求任意一个点的投影

$pro{j}_{\theta \theta }(x)=x^T\theta \theta$

求阈值

$\begin{array}{rcl}threshold& =& \frac{N_{C_1}{\tilde{\mu }}_{C_1}+N_{C_2}{\tilde{\mu }}_{C_1}}{N_{C_1}+N_{C_2}}\\ & =& \frac{N_{C_1}\mu {\mu }_{C_1}\theta \theta +N_{C_2}\mu {\mu }_{C_1}\theta \theta }{N_{C_1}+N_{C_2}}\end{array}$

依赖

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

人工数据集

n = 100
X = np.random.multivariate_normal((1, 1), [[0.64, 0], [0, 0.64]], size = int(n/2))
X = np.insert(X, 50, np.random.multivariate_normal((3, 3), [[0.64, 0], [0,0.64]], size = int(n/2)),0)
#X = np.insert(X, 0, 1, 1)
m = X.shape[1]
y = np.array([1]*50+[-1]*50).reshape(-1,1)
plt.scatter(X[:50, -2], X[:50, -1])
plt.scatter(X[50:, -2], X[50:, -1], c = "#ff4400")

<matplotlib.collections.PathCollection at 0x7f2b50e680d0>

X1 = X[(y==1).reshape(-1)]
X0 = X[(y==-1).reshape(-1)]
n1 = np.array([[X1.shape[0]]])
n0 = np.array([[X0.shape[0]]])
mu1 = X1.mean(axis = 0).reshape(-1,1)
mu0 = X0.mean(axis = 0).reshape(-1,1)
Sigma1 = np.cov(X1.T)
Sigma0 = np.cov(X0.T)
theta = (Sigma1 + Sigma0) @ (mu1 - mu0)
threshold = (n1*mu1 + n0*mu0).T@theta/(n1 + n0)
def getForecast(x):
    return x.T @ theta
threshold

array([[-10.45793931]])

预测

print(f'{ 1 if getForecast(np.array([[1],[1]])) > threshold else 0}')

1

分界展示

plt.scatter(X[:50, -2], X[:50, -1])
plt.scatter(X[50:, -2], X[50:, -1], c = "#ff4400")
for i in np.arange(-1,5,0.02):
    for j in np.arange(-1,5,0.02):
        if abs(getForecast(np.array([[i],[j]])) - threshold) <0.01:
            plt.scatter(i,j,c="#000000")