python大战机器学习——模型评估、选择与验证

1、损失函数和风险函数

(1)损失函数：常见的有 0-1损失函数  绝对损失函数  平方损失函数  对数损失函数

(2)风险函数：损失函数的期望      经验风险：模型在数据集T上的平均损失

　 根据大数定律，当N趋向于∞时，经验风险趋向于风险函数

2、模型评估方法

(1)训练误差与测试误差

　　训练误差：关于训练集的平均损失

　　测试误差：定义模型关于测试集的平均损失。其反映了学习方法对未知测试数据集的预测能力

(2)泛化误差：学到的模型对未知数据的预测能力。其越小，该模型越有效。泛化误差定义为所学习模型的期望风险

(3)过拟合：对已知数据预测得很好，对未知数据预测得很差的现象。原因是将训练样本本身的一些特点当做了所有潜在样本都具有的一般性质，这会造成泛化能力的下降。常用的防止过拟合的办法为正则化。正则化是基于结构化风险最小化策略的实现。

3、模型评估

(1)留出法：直接将数据划分为三个互斥的部分，然后在训练集上训练模型，在验证集上选择模型，最后用测试集上的误差作为泛化误差的估计。

(2)交叉验证法(S折交叉验证法)：数据随机划分为S个互不相交且大小相同的子集，利用S-1个子集数据训练模型，利用余下的一个子集测试模型。对S种组合依次重复进行，获取测试误差的均值。

(3)留一法：留出一个样例作为测试集。其缺点就是当数据集比较大时计算量太大

(4)自助法：先从T中随机取出一个样本放入采样集T_S中，再把该样本放回T中。经过N次随机采样操作，得到包含N个样本的采样集T_S。将TS用作训练集，T-T_S用过测试集。

4、性能度量

(1)测试准确率和测试错误率

(2)混淆矩阵

　　查准率：P=TP/(TP+FP)  ，即所有预测为正类的结果中，真正的正类的比例

　　查全率：R=TP/(TP+FN)，即正真的正类中，被分类器找出来的比例

　　不同的问题中，判别标准不同。对于推荐系统，更侧重于查准率（即推荐的结果中，用户真正感兴趣的比例）；对于医学诊断系统，更侧重于查全率（即疾病被发现的比例）

　　2/F1=1/P+1/R

5、ROC曲线

　　真正例率：TPR=TP/(TP+FN)

　　假正例率：FPR=FP/(TN+FP)，刻画的是分类器错认为正类的负实例占所有负实例的比例

　　以真正例率为纵轴、假正例率为横轴作图，就得到ROC曲线。在ROC图中，对角线对应于随机猜想模型。点(0,1)对应于理想模型。通常ROC曲线越靠近点(0,1)越好。

6、偏差方差分解

代码如下：

from sklearn.metrics import zero_one_loss,log_loss
from sklearn.model_selection import train_test_split,KFold,StratifiedKFold,LeaveOneOut,cross_val_score
from sklearn.datasets import load_digits,load_iris
from sklearn.svm import LinearSVC,SVC
from sklearn.metrics import accuracy_score,precision_score,recall_score,f1_score,classification_report
from sklearn.metrics import confusion_matrix,precision_recall_curve,roc_curve
from sklearn.metrics import mean_absolute_error,mean_squared_error,classification_report
from sklearn.multiclass import OneVsRestClassifier
from sklearn.model_selection import validation_curve,learning_curve,GridSearchCV,RandomizedSearchCV
import matplotlib.pyplot as plt
from sklearn.preprocessing import label_binarize
from sklearn.linear_model import LogisticRegression
import numpy as np
#zero_one_loss
# y_true=[1,1,1,1,1,0,0,0,0,0]
# y_pred=[0,0,0,1,1,1,1,1,0,0]
# print("zero_one_loss<fraction>:",zero_one_loss(y_true,y_pred,normalize=True))
# print("zero_one_loss<num>:",zero_one_loss(y_true,y_pred,normalize=False))

#log_loss
# y_true=[1,1,1,0,0,0]
# y_pred=[[0.1,0.9],
#         [0.2,0.8],
#         [0.3,0.7],
#         [0.7,0.3],
#         [0.8,0.2],
#         [0.9,0.1]
#         ]
# print("log_loss<average>:",log_loss(y_true,y_pred,normalize=True))
# print("log_loss<total>:",log_loss(y_true,y_pred,normalize=False))

#train_test_split
# X=[
#     [1,2,3,4],
#     [11,12,13,14],
#     [21,22,23,24],
#     [31,32,33,34],
#     [41,42,43,44],
#     [51,52,53,54],
#     [61,62,63,64],
#     [71,72,73,74]
# ]
# Y=[1,1,0,0,1,1,0,0]
# X_train,X_test,Y_train,Y_test=train_test_split(X,Y,test_size=0.4,random_state=0)
# print("X_train=",X_train)
# print("X_test=",X_test)
# print("Y_train=",Y_train)
# print("Y_test=",Y_test)
# X_train,X_test,Y_train,Y_test=train_test_split(X,Y,test_size=0.4,random_state=0,stratify=Y)
# print("X_train=",X_train)
# print("X_test=",X_test)
# print("Y_train=",Y_train)
# print("Y_test=",Y_test)

#KFold
# X=np.array([
#     [1,2,3,4],
#     [11,12,13,14],
#     [21,22,23,24],
#     [31,32,33,34],
#     [41,42,43,44],
#     [51,52,53,54],
#     [61,62,63,64],
#     [71,72,73,74],
#     [81,82,83,84]
# ])
# Y=np.array([1,1,0,0,1,1,0,0,1])
#
# folder=KFold(n_splits=3,random_state=0,shuffle=False)
# for train_index,test_index in folder.split(X,Y):
#     print("Train Index:",train_index)
#     print("Test Index:",test_index)
#     print("X_train:",X[train_index])
#     print("X_test:",X[test_index])
#     print("")
#
# shuffle_folder=KFold(n_splits=3,random_state=0,shuffle=True)
# for train_index,test_index in shuffle_folder.split(X,Y):
#     print("Train Index:",train_index)
#     print("Test Index:",test_index)
#     print("X_train:",X[train_index])
#     print("X_test:",X[test_index])
#     print("")

#StratifiedKFold
# stratified_folder=StratifiedKFold(n_splits=4,random_state=0,shuffle=False)
#as the operation is similar to the above,pass

#LeaveOneOut,too easy,pass
# loo=LeaveOneOut(len(Y))

#cross_val_score
# digits=load_digits()
# X=digits.data
# Y=digits.target
#
# result=cross_val_score(LinearSVC(),X,Y,cv=10)
# print("Cross Val Score is:",result)

#accuracy_score,pass
# accuracy_score(y_true,y_pred,normalize=True/False)

#precision_score,pass
# precision_socre(y_true,y_pred)

#recall_score,pass
# recall_score(y_true,y_pred)

#f1_score,pass
# f1_score(y_true,y_pred)

#fbeta_score,pass
# fbeta_score(y_true,y_pred,beta=num_beta)

#classification_report
# y_true=[1,1,1,1,1,0,0,0,0,0]
# y_pred=[0,0,1,1,0,0,0,0,0,0]
# print("Classification Report:\n",classification_report(y_true,y_pred,target_names=["class_0","class_1"]))

#confusion_matrix,pass
# confusion_matrix(y_true,y_pred,labels=[0,1])

#precision_recall_curve
# iris=load_iris()
# X=iris.data
# Y=iris.target
# #print(X,'\n',Y)
# Y=label_binarize(Y,classes=[0,1,2])
# n_classes=Y.shape[1]
# # print(n_classes,'\n',Y)
# np.random.seed(0)
# n_samples,n_features=X.shape
# # print(n_samples,'\n',n_features)
# X=np.c_[X,np.random.randn(n_samples,200*n_features)]
# # n_samples,n_features=X.shape
# # print(n_samples,'\n',n_features)
# x_train,x_test,y_train,y_test=train_test_split(X,Y,test_size=0.5,random_state=0)
# clf=OneVsRestClassifier(SVC(kernel='linear',probability=True,random_state=0))
# clf.fit(x_train,y_train)
# y_score=clf.fit(x_train,y_train).decision_function(x_test)
# # print(y_score)
# fig=plt.figure()
# ax=fig.add_subplot(1,1,1)
# precision=dict()
# recall=dict()
# for i in range(n_classes):
#     precision[i],recall[i],_=precision_recall_curve(y_test[:,i],y_score[:,i])
#     ax.plot(recall[i],precision[i],label="target=%s"%i)
# ax.set_xlabel("Recall Score")
# ax.set_ylabel("Precision Score")
# ax.set_title("P-R")
# ax.legend(loc="best")
# ax.set_xlim(0,1.1)
# ax.set_ylim(0,1.1)
# ax.grid()
# plt.show()

#roc_curve,roc_auc_score,pass
# roc_curve(y_test,y_score)

#mean_absolute_error,pass
# mean_absolute_error(y_true,y_pred)

#mean_squared_error,pass
# mean_squared_error(y_true,y_pred)

#validation_curve,pass
# validation_curve(LinearSVC(),X,Y,param_name="C",param_range=np.logspace(-2,2),cv=10,scoring="accuracy")

#learning_curve,pass
# train_size=np.linspace(0.1,1.0,endpoint=True,dtype='float')
# learning_curve(LinearSVC(),X,Y,cv=10,scoring="accuracy",train_sizes=train_size)

#GridSearcgCV
# digits=load_digits()
# x_train,x_test,y_train,y_test=train_test_split(digits.data,digits.target,test_size=0.25,random_state=0,stratify=digits.target)
# tuned_parameters=[{'penalty':['l1','l2'],'C':[0.01,0.05,0.1,0.5,1,5,10,50,100],'solver':['liblinear'],'multi_class':['ovr']},
#                   {'penalty':['l2'],'C':[0.01,0.05,0.1,0.5,1,5,10,50,100],'solver':['lbfgs'],'multi_class':['ovr','multinomial']},
#                   ]
# clf=GridSearchCV(LogisticRegression(tol=1e-6),tuned_parameters,cv=10)
# clf.fit(x_train,y_train)
# print("Best parameters set found:",clf.best_params_)
# print("Grid scores:")
# for params,mean_score,scores in clf.grid_scores_:
#     print("\t%0.3f(+/-%0.03f) for %s"%(mean_score,scores.std()*2,params))
# 
# print("Optimized Score:",clf.score(x_test,y_test))
# print("Detailed classification report:")
# y_true,y_pred=y_test,clf.predict(x_test)
# print(classification_report(y_true,y_pred))

#RandomizedSearchCV
# RandomizedSearchCV(LogisticRegression(penalty='l2',solver='lbfgs',tol=1e-6,tuned_parameters,cv=10,scoring='accuracy',n_iter=100))