机器学习建模中--先“特征选择”还是先“划分数据集”?
先说结论:应该先“划分数据集”,再进行“特征选择”。这样可以避免数据泄露。
测试集就应该当做“看不见的数据”,只能在最后用一次,按照这个原则处理。
代码实例:
# -*- coding: utf-8 -*-
import numpy as np
from sklearn.feature_selection import SelectKBest
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
#===================错误做法:特征选择在前,划分数据集在后============================
# #---错误做法的结果很好----
# # random data:
# X = np.random.randn(500, 10000)
# y = np.random.choice(2, size=500)
# selector = SelectKBest(k=25)
# # first select features
# X_selected = selector.fit_transform(X,y)
# # then split
# X_selected_train, X_selected_test, y_train, y_test = train_test_split(X_selected, y, test_size=0.25, random_state=42)
# # fit a simple logistic regression
# lr = LogisticRegression()
# lr.fit(X_selected_train,y_train)
# # predict on the test set and get the test accuracy:
# y_pred = lr.predict(X_selected_test)
# acc = accuracy_score(y_test, y_pred)
# print(acc)
# #几次结果为0.712,0.688,0.792,0.776,0.648
# #---检验一下这种错误做法;泛化性能很差----
# X_new = np.random.randn(500, 10000)
# y_new = np.random.choice(2, size=500)
# # select the same features in the new data
# X_new_selected = selector.transform(X_new)
# # predict and get the accuracy:
# y_new_pred = lr.predict(X_new_selected)
# acc_new = accuracy_score(y_new, y_new_pred)
# print(acc_new)
# # 几次结果为:0.498,0.504, 0.492, 0.538
#=============正确做法:先划分数据集,再特征选择===============================
# random data:
X = np.random.randn(500, 10000)
y = np.random.choice(2, size=500)
# split first
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
# then select features using the training set only
selector = SelectKBest(k=25)
X_train_selected = selector.fit_transform(X_train,y_train)
# fit again a simple logistic regression
lr = LogisticRegression()
lr.fit(X_train_selected,y_train)
# select the same features on the test set, predict, and get the test accuracy:
X_test_selected = selector.transform(X_test)
y_pred = lr.predict(X_test_selected)
acc = accuracy_score(y_test, y_pred)
print(acc)
# 几次的结果为:0.48,0.472,0.52