sklearn机器学习-特征提取1
scikit-learn机器学习的特征提取部分较多nlp内容，故学到一半学不下去，看完nltk再来补上
scikit-learn机器学习的特征提取这一章感觉讲的不是特别好，所以会结合着来看
首先是Dictvectorizer
from sklearn.feature_extraction import DictVectorizer
onehot_encoder = DictVectorizer()
X = [{'city':'New York'},{'city':'San Francisco'},{'city':'Chapel Hill'}]
print(onehot_encoder.fit_transform(X).toarray())
[[0. 1. 0.]
[0. 0. 1.]
[1. 0. 0.]]
这里的toarray方法，在很多one-hot方法中都有，需要注意一下
然后这里必须传入字典形式
measurements = [{'city':'Beijing','country':'CN','temperature':33.},{'city':'London','country':'UK','temperature':12.},{'city':'San Fransisco','country':'USA','temperature':18.}]
#从sklearn.feature_extraction导入DictVectorizer
from sklearn.feature_extraction import DictVectorizer
vec = DictVectorizer()
#输出转化后的特征矩阵
print(vec.fit_transform(measurements).toarray())
#输出各个维度的特征含义
print(vec.get_feature_names_out())
[[ 1. 0. 0. 1. 0. 0. 33.]
[ 0. 1. 0. 0. 1. 0. 12.]
[ 0. 0. 1. 0. 0. 1. 18.]]
['city=Beijing' 'city=London' 'city=San Fransisco' 'country=CN'
'country=UK' 'country=USA' 'temperature']
StandardScalar:
$X = \frac{x-\mu}{\sigma}$
MinMaxScalar:
$X = \frac{x-x_{min}(axis=0)}{x_{max}(axis=0)-x_{min}(axis=0)}$
scale:
$X = \frac{x-x_{mean}}{\sigma}$
或者还可以使用RoubustScalar
from sklearn import preprocessing
import numpy as np
X = np.array([[0,0,5,13,9,1],[0,0,13,15,10,15],[0,3,15,2,0,11]])
print(preprocessing.scale(X))
[[ 0. -0.70710678 -1.38873015 0.52489066 0.59299945 -1.35873244]
[ 0. -0.70710678 0.46291005 0.87481777 0.81537425 1.01904933]
[ 0. 1.41421356 0.9258201 -1.39970842 -1.4083737 0.33968311]]
或者也可以直接写
from sklearn.preprocessing import scale
import numpy as np
X = np.array([[0,0,5,13,9,1],[0,0,13,15,10,15],[0,3,15,2,0,11]])
sc = scale(X)
print(sc)
词袋模型
希望我能用一段中文描述，多记一下这个CountVectorizer
CountVectorizer是属于常见的特征数值计算类，是一个文本特征提取方法，对于每一个训练文本，只考虑每种词汇在该训练文本中出现的频率，也即ConutVectorizer会将文本中的词语转换为词频矩阵
corpus = ['UNC played Duke in basketball','duke lost the basketball game']
from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer()
print(vectorizer.fit_transform(corpus).todense())
print(vectorizer.vocabulary_)
注意，这里有Duke，还有duke
[[1 1 0 1 0 1 0 1]
[1 1 1 0 1 0 1 0]]
{'unc': 7, 'played': 5, 'duke': 1, 'in': 3, 'basketball': 0, 'lost': 4, 'the': 6, 'game': 2}
corpus.append("I ate a sandwich")
print(vectorizer.fit_transform(corpus).todense())
print(vectorizer.vocabulary_)
[[0 1 1 0 1 0 1 0 0 1]
[0 1 1 1 0 1 0 0 1 0]
[1 0 0 0 0 0 0 1 0 0]]
{'unc': 9, 'played': 6, 'duke': 2, 'in': 4, 'basketball': 1, 'lost': 5, 'the': 8, 'game': 3, 'ate': 0, 'sandwich': 7}
上述文档中，第一行应该是和第二行比较接近，在sklearn中，用eculidean_distances来计算向量间的距离
from sklearn.metrics.pairwise import euclidean_distances
from sklearn.feature_extraction.text import CountVectorizer
corpus = ['UNC played Duke in basketball','Duke lost the basketball game','I ate a sandwich']# 文集
vectorizer =CountVectorizer()#
counts = vectorizer.fit_transform(corpus).todense() #得到文集corpus的特征向量，并将其转为密集矩阵
print(counts)
for x,y in [[0,1],[0,2],[1,2]]:
dist = euclidean_distances(counts[x],counts[y])
print('文档{}与文档{}的距离{}'.format(x,y,dist))
[[0 1 1 0 1 0 1 0 0 1]
[0 1 1 1 0 1 0 0 1 0]
[1 0 0 0 0 0 0 1 0 0]]
文档0与文档1的距离[[2.44948974]]
文档0与文档2的距离[[2.64575131]]
文档1与文档2的距离[[2.64575131]]
维度太高的话，复杂度会比较大，一般需要用到降维的方法
方法一在上述中用到了，就是所有大写都变成了小写
方法二，停用词过滤，即加上stop_words
vectorizer = CountVectorizer(stop_words='english')
print(vectorizer.fit_transform(corpus).todense())
print(vectorizer.vocabulary_)
[[0 1 1 0 0 1 0 1]
[0 1 1 1 1 0 0 0]
[1 0 0 0 0 0 1 0]]
{'unc': 7, 'played': 5, 'duke': 2, 'basketball': 1, 'lost': 4, 'game': 3, 'ate': 0, 'sandwich': 6}
方法三，还是CountVectorizer，这次是词干提取和词形还原
词干提取：去后缀
词形还原：将单词的复杂形态转变成最基础的形态
词干提取主要方法：Porter、Snowball、Lancaster
词形还原主要方法：使用NLTK库，再加上WordNet方法
在实际的代码中，注意一下一下内容的输出
corpus = ['he ate the sandwiches ','Every sandwich was eaten by him']
vectorizer = CountVectorizer(binary=True,stop_words='english')
print(vectorizer.fit_transform(corpus).todense())
print(vectorizer.vocabulary_)
[[1 0 0 1]
[0 1 1 0]]
{'ate': 0, 'sandwiches': 3, 'sandwich': 2, 'eaten': 1}
corpus = ['he ate the sandwiches every day','Every sandwich was eaten by him']
vectorizer = CountVectorizer(binary=True,stop_words='english')
print(vectorizer.fit_transform(corpus).todense())
print(vectorizer.vocabulary_)
[[1 1 0 0 1]
[0 0 1 1 0]]
{'ate': 0, 'sandwiches': 4, 'day': 1, 'sandwich': 3, 'eaten': 2}
corpus = ['jack ate the sandwiches every day','Every sandwich was eaten by him']
vectorizer = CountVectorizer(binary=True,stop_words='english')
print(vectorizer.fit_transform(corpus).todense())
print(vectorizer.vocabulary_)
[[1 1 0 1 0 1]
[0 0 1 0 1 0]]
{'jack': 3, 'ate': 0, 'sandwiches': 5, 'day': 1, 'sandwich': 4, 'eaten': 2}
可以看出，上述的词干提取，并不是简答剔除重复的，而是将无用的信息页剔除掉了，这一点需要注意。同时也反映了CountVectorizer的好用
接下来使用nltk
corpus = ['I am gathering ingredients for the sandwich.','There were many wizards at the gathering.']
from nltk.stem.wordnet import WordNetLemmatizer
lemmatizer = WordNetLemmatizer()
print(lemmatizer.lemmatize('gathering','v'))
print(lemmatizer.lemmatize('gathering','n'))
gather
gathering
