2、数据规范化
数据规范化
均值-方差规范化、极差规范化
由于变量或指标的单位不同，造成有些指标数据值非常大，而有些非常小，在模型运算过程中大的数据会把小的数据覆盖掉，造成模型失真。因此，需要对这些数据做规范化处理，或者说去量纲化。
均值-方差规范化：是指变量或者指标数据减去其均值再除以标准差得到的数据。新数据均值为0，方差为1。其公式如下：
𝑥∗=𝑥−𝑚𝑒𝑎𝑛(𝑥)𝑠𝑡𝑑(𝑥)x^∗=(x-mean(x))/(std(x))
极差规范化： 是指变量或是指标数据减去其最小值，再除以最大值与最小值之差，得到新的数据。新数据取值范围再[0,1]。其计算公式为：
𝑥∗=𝑥−min(𝑥)max𝑥−min(𝑥)x^∗=(x-min(x))/(max(x)-min(x))
1、读取数据
#读取数据
import numpy as np
data=np.load('data.npy')
data=data[:,1:]
data
array([[ 17. , 66.17647059, 32. , 1614.96618125,
13.15625 ],
[ 8. , 68.6875 , 36. , 143.56458056,
3.80555556],
[ 16. , 65.84375 , 43. , 1344.13137674,
12.69767442],
...,
[ 10. , 67.95 , 24. , 115.87417083,
2.79166667],
[ 21. , 66.5 , 41. , 538.71289268,
20.31707317],
[ 11. , 78.27272727, 9. , 62.98323333,
9.44444444]])
#个人爱好，jupyter内看着舒服
import pandas as pd
data = pd.DataFrame(data)
data
| 0 | 1 | 2 | 3 | 4 | |
|---|---|---|---|---|---|
| 0 | 17.0 | 66.176471 | 32.0 | 1614.966181 | 13.156250 |
| 1 | 8.0 | 68.687500 | 36.0 | 143.564581 | 3.805556 |
| 2 | 16.0 | 65.843750 | 43.0 | 1344.131377 | 12.697674 |
| 3 | 2.0 | 75.000000 | 2.0 | 0.365700 | 1.000000 |
| 4 | 27.0 | 65.740741 | 60.0 | 991.953787 | 11.100000 |
| ... | ... | ... | ... | ... | ... |
| 830 | 35.0 | 66.057143 | 44.0 | 127.945364 | 12.250000 |
| 831 | 14.0 | 69.714286 | 7.0 | 32.219643 | 15.571429 |
| 832 | 10.0 | 67.950000 | 24.0 | 115.874171 | 2.791667 |
| 833 | 21.0 | 66.500000 | 41.0 | 538.712893 | 20.317073 |
| 834 | 11.0 | 78.272727 | 9.0 | 62.983233 | 9.444444 |
835 rows × 5 columns
2、导入预处理库
#导入预处理库
from sklearn.impute import SimpleImputer
from sklearn import preprocessing
3、预处理空值
#均值填充空值
#利用Imputer 创建填充对象imp_mean
imp_mean = SimpleImputer(missing_values=np.nan, strategy='mean',verbose=0)
#fit_transform一步完成调取结果
imp_mean = imp_mean.fit_transform(data)
imp_mean
array([[ 17. , 66.17647059, 32. , 1614.96618125,
13.15625 ],
[ 8. , 68.6875 , 36. , 143.56458056,
3.80555556],
[ 16. , 65.84375 , 43. , 1344.13137674,
12.69767442],
...,
[ 10. , 67.95 , 24. , 115.87417083,
2.79166667],
[ 21. , 66.5 , 41. , 538.71289268,
20.31707317],
[ 11. , 78.27272727, 9. , 62.98323333,
9.44444444]])
#看着舒服
pd.DataFrame(imp_mean)
| 0 | 1 | 2 | 3 | 4 | |
|---|---|---|---|---|---|
| 0 | 17.0 | 66.176471 | 32.0 | 1614.966181 | 13.156250 |
| 1 | 8.0 | 68.687500 | 36.0 | 143.564581 | 3.805556 |
| 2 | 16.0 | 65.843750 | 43.0 | 1344.131377 | 12.697674 |
| 3 | 2.0 | 75.000000 | 2.0 | 0.365700 | 1.000000 |
| 4 | 27.0 | 65.740741 | 60.0 | 991.953787 | 11.100000 |
| ... | ... | ... | ... | ... | ... |
| 830 | 35.0 | 66.057143 | 44.0 | 127.945364 | 12.250000 |
| 831 | 14.0 | 69.714286 | 7.0 | 32.219643 | 15.571429 |
| 832 | 10.0 | 67.950000 | 24.0 | 115.874171 | 2.791667 |
| 833 | 21.0 | 66.500000 | 41.0 | 538.712893 | 20.317073 |
| 834 | 11.0 | 78.272727 | 9.0 | 62.983233 | 9.444444 |
835 rows × 5 columns
4、均值-方差规范化(Z-Score规范化)
X1 = imp_mean
scaled_x = preprocessing.scale(X1)
pd.DataFrame(scaled_x)
| 0 | 1 | 2 | 3 | 4 | |
|---|---|---|---|---|---|
| 0 | 0.200258 | -0.827606 | 0.055546 | 2.843538 | 0.769541 |
| 1 | -0.689187 | -0.092243 | 0.206625 | -0.185541 | -0.651561 |
| 2 | 0.101431 | -0.925045 | 0.471013 | 2.285988 | 0.699848 |
| 3 | -1.282151 | 1.756395 | -1.077545 | -0.480335 | -1.077944 |
| 4 | 1.188531 | -0.955211 | 1.113098 | 1.560983 | 0.457036 |
| ... | ... | ... | ... | ... | ... |
| 830 | 1.979150 | -0.862552 | 0.508783 | -0.217695 | 0.631811 |
| 831 | -0.096223 | 0.208455 | -0.888696 | -0.414760 | 1.136596 |
| 832 | -0.491533 | -0.308222 | -0.246611 | -0.242546 | -0.805650 |
| 833 | 0.595568 | -0.732860 | 0.395474 | 0.627925 | 1.857831 |
| 834 | -0.392705 | 2.714824 | -0.813157 | -0.351429 | 0.205428 |
835 rows × 5 columns
5、极差规范化(Min-max 规范化)
X2 = imp_mean
# 将数据进行 [0,1] 规范化
min_max_scaler = preprocessing.MinMaxScaler()
minmax_x = min_max_scaler.fit_transform(X2)
pd.DataFrame(minmax_x)
| 0 | 1 | 2 | 3 | 4 | |
|---|---|---|---|---|---|
| 0 | 0.380952 | 0.044063 | 0.251969 | 0.339418 | 0.138139 |
| 1 | 0.166667 | 0.171583 | 0.283465 | 0.030155 | 0.031881 |
| 2 | 0.357143 | 0.027166 | 0.338583 | 0.282493 | 0.132928 |
| 3 | 0.023810 | 0.492158 | 0.015748 | 0.000057 | 0.000000 |
| 4 | 0.619048 | 0.021935 | 0.472441 | 0.208472 | 0.114773 |
| ... | ... | ... | ... | ... | ... |
| 830 | 0.809524 | 0.038003 | 0.346457 | 0.026872 | 0.127841 |
| 831 | 0.309524 | 0.223728 | 0.055118 | 0.006753 | 0.165584 |
| 832 | 0.214286 | 0.134130 | 0.188976 | 0.024335 | 0.020360 |
| 833 | 0.476190 | 0.060493 | 0.322835 | 0.113208 | 0.219512 |
| 834 | 0.238095 | 0.658361 | 0.070866 | 0.013218 | 0.095960 |
835 rows × 5 columns
