模型融合---Stacking调参总结
1. 回归
训练了两个回归器，GBDT和Xgboost，用这两个回归器做stacking
使用之前已经调好参的训练器
gbdt_nxf = GradientBoostingRegressor(learning_rate=0.06,n_estimators=250,
min_samples_split=700,min_samples_leaf=70,max_depth=6,
max_features='sqrt',subsample=0.8,random_state=75)
xgb_nxf = XGBRegressor(learning_rate=0.06,max_depth=6,n_estimators=200,random_state=75)
事先建好stacking要用到的矩阵
from sklearn.model_selection import KFold,StratifiedKFold kf = StratifiedKFold(n_splits=5,random_state=75,shuffle=True) from sklearn.metrics import r2_score train_proba = np.zeros((len(gbdt_train_data),2)) train_proba = pd.DataFrame(train_proba) train_proba.columns = ['gbdt_nxf','xgb_nxf'] test_proba = np.zeros((len(gbdt_test_data),2)) test_proba = pd.DataFrame(test_proba) test_proba.columns = ['gbdt_nxf','xgb_nxf']
reg_names = ['gbdt_nxf','xgb_nxf']
for i,reg in enumerate([gbdt_nxf,xgb_nxf]):
pred_list = []
col = reg_names[i]
for train_index,val_index in kf.split(gbdt_train_data,gbdt_train_label):
x_train = gbdt_train_data.loc[train_index,:].values
y_train = gbdt_train_label[train_index]
x_val = gbdt_train_data.loc[val_index,:].values
y_val = gbdt_train_label[val_index]
reg.fit(x_train,y_train)
y_vali = reg.predict(x_val)
train_proba.loc[val_index,col] = y_vali
print('%s cv r2 %s'%(col,r2_score(y_val,y_vali)))
y_testi = reg.predict(gbdt_test_data.values)
pred_list.append(y_testi)
test_proba.loc[:,col] = np.mean(np.array(pred_list),axis=0)
r2值最高为0.79753,效果还不是特别的好
然后用五折交叉验证，每折都预测整个测试集，得到五个预测的结果，求平均，就是新的预测集；而训练集就是五折中任意四折预测该折的训练集得到的标签的集合
因为有两个训练器，GBDT和Xgboost，所以我们得到了两列的train_proba
最后对新的训练集和测试集做回归，得到我们的结果
#使用逻辑回归做stacking
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
scalar = StandardScaler()
# train_proba = train_proba.values
# test_proba = test_proba.values
scalar.fit(train_proba)
train_proba = scalar.transform(train_proba)
test_proba = scalar.transform(test_proba)
lr = LogisticRegression(tol=0.0001,C=0.5,random_state=24,max_iter=10)
kf = StratifiedKFold(n_splits=5,random_state=75,shuffle=True)
r2_list = []
pred_list = []
for train_index,val_index in kf.split(train_proba,gbdt_train_label):#训练集的标签还是一开始真实的训练集的标签
x_train = train_proba[train_index]
y_train = gbdt_train_label[train_index]
x_val = train_proba[val_index]
y_val = gbdt_train_label[val_index]
lr.fit(x_train,y_train)
y_vali = lr.predict(x_val)
print('lr stacking cv r2 %s'%(r2_score(y_val,y_vali)))
r2_list.append(r2_score(y_val,y_vali))
y_testi = lr.predict(test_proba)
pred_list.append(y_testi)
print(lr.coef_,lr.n_iter_)#过拟合很严重
2. 分类
经过对每个单模型进行调参之后，我们可以把这些模型进行 stacking 集成。
如上图所示，我们将数据集分成均匀的5部分进行交叉训练，使用其中的4部分训练，之后将训练好的模型对剩下的1部分进行预测，同时预测测试集；经过5次cv之后，我们可以得到训练集每个样本的预测值，同时得到测试集的5个预测值，我们将测试集的5个测试集进行平均。有多少个基模型，我们会得到几组不同的预测值；最后使用一个模型对上一步得到预测结果再进行训练预测，得到stacking结果。stacking模型一般使用线性模型。
stacking 有点像神经网络，基模型就像底层的神经网络对输入数据进行特征的提取，如下图所示：
首先我们先定义一个DataFrame 格式数据结构荣来存储中间预测结果：
train_proba = np.zeros((len(train), 6)) train_proba = pd.DataFrame(train_proba) train_proba.columns = ['rf','ada','etc','gbc','sk_xgb','sk_lgb'] test_proba = np.zeros((len(test), 6)) test_proba = pd.DataFrame(test_proba) test_proba.columns = ['rf','ada','etc','gbc','sk_xgb','sk_lgb']
定义基模型，交叉训练预测
rf = RandomForestClassifier(n_estimators=700, max_depth=13, min_samples_split=30,\
min_weight_fraction_leaf=0.0, random_state=24, verbose=0)
ada = AdaBoostClassifier(n_estimators=450, learning_rate=0.1, random_state=24)
gbc = GradientBoostingClassifier(learning_rate=0.08,n_estimators=150,max_depth=9,
min_samples_leaf=70,min_samples_split=900,
max_features='sqrt',subsample=0.8,random_state=10)
etc = ExtraTreesClassifier(n_estimators=290, max_depth=12, min_samples_split=30,random_state=24)
sk_xgb = XGBClassifier(learning_rate=0.05,n_estimators=400,
min_child_weight=20,max_depth=3,subsample=0.8, colsample_bytree=0.8,
reg_lambda=1., random_state=10)
sk_lgb = LGBMClassifier(num_leaves=31,max_depth=3,learing_rate=0.03,n_estimators=600,
subsample=0.8, colsample_bytree=0.9, objective='binary',
min_child_weight=0.001, subsample_freq=1, min_child_samples=10,
reg_alpha=0.0, reg_lambda=0.0, random_state=10, n_jobs=-1,
silent=True, importance_type='split')
kf = StratifiedKFold(n_splits=5,random_state=233,shuffle=True)
clf_name = ['rf','ada','etc','gbc','sk_xgb','sk_lgb']
for i,clf in enumerate([rf,ada,etc,gbc,sk_xgb,sk_lgb]):
pred_list = []
col = clf_name[i]
for train_index, val_index in kf.split(train,label):
X_train = train.loc[train_index,:].values
y_train = label[train_index]
X_val = train.loc[val_index,:].values
y_val = label[val_index]
clf.fit(X_train, y_train)
y_vali = clf.predict_proba(X_val)[:,1]
train_proba.loc[val_index,col] = y_vali
print("%s cv auc %s" % (col, roc_auc_score(y_val, y_vali)))
y_testi = clf.predict_proba(test.values)[:,1]
pred_list.append(y_testi)
test_proba.loc[:,col] = np.mean(np.array(pred_list),axis=0)
使用逻辑回归做最后的stacking
scaler = StandardScaler()
train_proba = train_proba.values
test_proba = test_proba.values
scaler.fit(train_proba)
train_proba = scaler.transform(train_proba)
test_proba = scaler.transform(test_proba)
lr = LogisticRegression(tol=0.0001, C=0.5, random_state=24, max_iter=10)
kf = StratifiedKFold(n_splits=5,random_state=244,shuffle=True)
auc_list = []
pred_list = []
for train_index, val_index in kf.split(train_proba,label):
X_train = train_proba[train_index]
y_train = label[train_index]
X_val = train_proba[val_index]
y_val = label[val_index]
lr.fit(X_train, y_train)
y_vali = lr.predict_proba(X_val)[:,1]
print("lr stacking cv auc %s" % (roc_auc_score(y_val, y_vali)))
auc_list.append(roc_auc_score(y_val, y_vali))
y_testi = lr.predict_proba(test_proba)[:,1]
pred_list.append(y_testi)
print(lr.coef_, lr.n_iter_)
最终各个基模型和stacking模型的 auc 得分如下图所示：
分别为 0.8415，0.8506，0.8511，0.8551，0.8572，0.8580，0.8584。
