基于朴素贝叶斯模型的文本分类
(原创文章,转载请注明出处!)
一、朴素贝叶斯模型
模型一:
将一个文本文档使用一个词的向量来表示。
通常文档中出现的词的个数是有限的,假设要将文档分成两类(类别0、1),分类的所有文档可能出现100个词(词典中词的个数,在实际应用中,选择训练文档中出现次数最多的n个词,n从10000到50000),那么一个特定的文档就可以用一个100维的向量来表示。每一维上要么是0,要么是1。
朴素贝叶斯模型的思想是分别对两类文档建模,那么文档类别0和类别1都会通过训练学习到自己的模型参数p(x|y=0)、p(x|y=1)、p(y=1),那么当来了个新文档x,,分别用类别0和类别1的模型来计算p(y=0|x,)=p(x|y=0)p(y=0),p(y=1|x,)=p(x|y=1)p(y=1),然后比较哪个概率大就决定新的文档属于哪一类。(或者是比较log(p(x|y=0)) + log(p(y=0)),log(p(x|y=1)) + log(p(y=1)) )
要计算一个文档的类别,首先要计算每个词与文档类别的关系。
运用朴素贝叶斯模型的来进行计算时,还有一个关键的假设:各个词与类别是条件独立的。写成公式如下:
p(w1w2w3......w99w100|y) = p(w1|y)p(w2|y,w1)p(w3|y,w1,w2)......p(w100|y,w1,w2,...,w99)
=p(w1|y)p(w2|y)p(w3|y)......p(w100|y)
那么,Φj|y=1=p(wj=1|y=1),词j与类别1的关系
Φj|y=0=p(wj=1|y=0),词j与类别0的关系
建立以Φj|y=1,Φj|y=0,Φy=1=p(y=1)为参数的似然函数,通过极大似然法可以求出:
Φj|y=1=∑i=1...m[xji=1 and yi=1] / ∑i=1...m[yi=1] ,用训练样本中类别为1且出现了词j的文档数 除以 所有类别为1的文档数
Φj|y=0=∑i=1...m[xji=1 and yi=0] / ∑i=1...m[yi=0] ,用训练样本中类别为0且出现了词j的文档数 除以 所有类别为0的文档数
Φy=1=训练样本中类别为1的文数 / 训练样本中所有文档数
当来了一个新样本x,,x,=[w1, w2, w3, ...... w100, ]
计算p(y=1|x,) = [p(x,|y=1)p(y=1)] / p(x,) # 将p(x)忽略掉
= [Φj=1|y=1w1, Φj=2|y=1w2, Φj=3|y=1w3,...... Φj=100|y=1w100,] #将该向量中不为0的元素求乘积
* Φy=1
p(y=0|x,)= [Φj=1|y=0w1, Φj=2|y=0w2, Φj=3|y=0w3,...... Φj=100|y=0w100,] #将该向量中不为0的元素求乘积
* (1 - Φy=1)
比较两个值的大小,就能确定新样本的类别。
模型二:
在以上的模型中,只关注一个词是否在文档中出现,出现了就将词向量中对应的元素置为1,否则就是0。
但通常某个词可能在一个文档的不同位置多次出现,如何能更好的描述这一客观事实?
假设词典中词的总数还是100个,词的序号k={1,2,3, ... , 100},共有m个文档,第个i文档包括有ni个词,文档xi的第j个词为xji,xji可以是总共100个词中的任意一个。
词典中第K个词与类别1的关系:Φk|y=1=∑i=1...m∑j=1...ni[xji=k and yi=1] / ∑i=1...m[(yi=1)*ni],
其中[xji=k and yi=1]表示第i个文档时类型1,且在该文档中的第j个词是词K。
词典中第K个词与类别0的关系:Φk|y=0=∑i=1...m∑j=1...ni[xji=k and yi=0] / ∑i=1...m[(yi=0)*ni],
其中[xji=k and yi=0]表示第i个文档时类型0,且在该文档中的第j个词是词K。
Φy=1同上一模型中一样。
通过计算得到 类别1的词典向量:[Φ1|y=1 Φ2|y=1 Φ3|y=1 ...... Φ100|y=1 ], 类别0的词典向量:[Φ1|y=0 Φ2|y=0 Φ3|y=0 ...... Φ100|y=0 ]
当来了一个新样本x,,有n个词,x,=[w1, w2, w3, ...... wn, ]。
计算p(y=1|x,) ={遍历x,把其中每个词对应的Φk|y=1求乘机} * Φy=1 ,p(y=0|x,) ={遍历x,把其中每个词对应的Φk|y=0求乘机} * Φy=1 。
比较两者中,较大者,就是新样本对应的类别。
还有一点:
如果词典中的一个词在所有训练样本都没有出现,但在新文档中出现了,通过训练,计算的该词的Φk|y=1=0,Φk|y=0=0 (模型一中也类似)。
那么计算新文档的p(y=1|x,), p(y=0|x,) 就都会是0.
如何克服这个问题?可以将计算Φk|y=1和Φk|y=0的公式做一点修改:
Φk|y=1={∑i=1...m∑j=1...ni[xji=k and yi=1] + 1} / {∑i=1...m[(yi=1)*ni] + 100 }
Φk|y=0={∑i=1...m∑j=1...ni[xji=k and yi=0] + 1} / {∑i=1...m[(yi=0)*ni] + 100}
二、实现
使用R编程语言,基于模型二的实现代码如下:
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
## = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = ## initialize ## = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = library(Matrix) numTrainDocs <- 700 # number of documents, 400, 100, 50 numTokens <- 2500 # number of words totally trDoc <- read.table(file="train-features.txt") trLabel <- read.table(file="train-labels.txt") x <- sparseMatrix(i=trDoc$V1, j=trDoc$V2, x=trDoc$V3, dims=c(numTrainDocs,numTokens)) x <- as.matrix(x) y <- matrix(data=trLabel$V1,ncol=1) ## = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = ## training ## = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = ## 1. calculate the phi of y, i.e. p(y=1) phiOfy1 <- colSums(y) / dim(y)[1] phiOfy0 <- 1 - phiOfy1 ## 2. calculate the phi of k to given y=1 or 0, i.e. p(Xj=k|y=1) p(Xj=k|y=0) ## # # # # The number of parameters is numTokens, which is same to the vocabulary length. # i.e. Each word in vocabulary is a feature. Each feature need a parameter. phiArray1 <- numeric(numTokens) # y=1, parameter vector phiArray0 <- numeric(numTokens) # y=0, parameter vector sum1 <- sum(x[y==1,]) sum0 <- sum(x[y==0,]) for (i in 1:numTokens) { # calculte each parameter, one by one phiArray1[i] <- ( sum(x[y==1,i]) + 1 ) / ( sum1 + numTokens ) phiArray0[i] <- ( sum(x[y==0,i]) + 1 ) / ( sum0 + numTokens ) } ## = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = ## test ## = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = xTestDoc <- read.table(file="test-features.txt") yTest <- read.table(file="test-labels.txt") xTest <- sparseMatrix(i=xTestDoc$V1, j=xTestDoc$V2, x=xTestDoc$V3) xTest <- as.matrix(xTest) numTestDoc <- dim(xTest)[1] numTestTokens <- dim(xTest)[2] testDocProbility1 <- xTest %*% as.matrix(log(phiArray1)) + log(phiOfy1) testDocProbility0 <- xTest %*% as.matrix(log(phiArray0)) + log(phiOfy0) testLabel <- (testDocProbility1 > testDocProbility0) testLabel[testLabel == TRUE] <- 1 classErrorNum <- sum(abs(yTest - as.vector(testLabel))) classErrorRatio <- classErrorNum / numTestDoc
三、测试结果
实验的数据中,词共有2500个(词典中词个数),测试四次,训练文档个数分别是700,400,100,50,测试集有260个文档.
训练文档数 | 错分类文档数 | 错分率 |
700 | 5 | 1.92% |
400 | 6 | 2.30% |
100 | 6 | 2.30% |
50 | 7 | 2.69% |
随着训练文档数增加,错分率下降。总体来说,朴素贝叶斯进行文档分类的错分率还是相对较低的。