Kaggle项目实战一：Titanic: Machine Learning from Disaster
项目地址
https://www.kaggle.com/c/titanic
项目介绍：
除了乘客的编号以外，还包括下表中10个字段，构成了数据的所有特征
|
Variable |
Definition |
Key |
|
survival |
是否存活 |
0 = No, 1 = Yes |
|
pclass |
票的等级 |
1 = 1st, 2 = 2nd, 3 = 3rd |
|
sex |
性别 |
|
|
Age |
年龄 |
|
|
sibsp |
同乘配偶或兄弟姐妹 |
|
|
parch |
同乘孩子或父母 |
|
|
ticket |
票号 |
|
|
fare |
乘客票价 |
|
|
cabin |
客舱号码 |
|
|
embarked |
登船港口 |
C = Cherbourg, Q = Queenstown, S = Southampton |
一、导入数据
train_df = pd.read_csv("..\train.csv") test_df = pd.read_csv("..\test.csv")
查看数据整体缺失情况
结果如下：存在null值得字段有Age、Fare和Cabin，其中Cabin缺失最为严重，缺失率77.1%
train_df.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 891 entries, 0 to 890 Data columns (total 12 columns): PassengerId 891 non-null int64 Survived 891 non-null int64 Pclass 891 non-null int64 Name 891 non-null object Sex 891 non-null object Age 714 non-null float64 SibSp 891 non-null int64 Parch 891 non-null int64 Ticket 891 non-null object Fare 891 non-null float64 Cabin 204 non-null object Embarked 889 non-null object
连续型变量分布情况
train_df.describe() PassengerId Survived Pclass Age SibSp \ count 891.000000 891.000000 891.000000 714.000000 891.000000 mean 446.000000 0.383838 2.308642 29.699118 0.523008 std 257.353842 0.486592 0.836071 14.526497 1.102743 min 1.000000 0.000000 1.000000 0.420000 0.000000 25% 223.500000 0.000000 2.000000 20.125000 0.000000 50% 446.000000 0.000000 3.000000 28.000000 0.000000 75% 668.500000 1.000000 3.000000 38.000000 1.000000 max 891.000000 1.000000 3.000000 80.000000 8.000000 Parch Fare count 891.000000 891.000000 mean 0.381594 32.204208 std 0.806057 49.693429 min 0.000000 0.000000 25% 0.000000 7.910400 50% 0.000000 14.454200 75% 0.000000 31.000000 max 6.000000 512.329200
离散变量情况（ 包括客舱号码，登船港口，票的等级，性别）
train_df.describe(include=['O']) Name Sex Ticket Cabin Embarked count 891 891 891 204 889 unique 891 2 681 147 3 top Greenberg, Mr. Samuel male 347082 B96 B98 S freq 1 577 7 4 644
- Total samples are 891 or 40% of the actual number of passengers on board the Titanic (2,224).
- Survived is a categorical feature with 0 or 1 values.
- Around 38% samples survived representative of the actual survival rate at 32%.
- Most passengers (> 75%) did not travel with parents or children.
- Nearly 30% of the passengers had siblings and/or spouse aboard.
- Fares varied significantly with few passengers (<1%) paying as high as $512.
- Few elderly passengers (<1%) within age range 65-80.
讨论特征增加和删除：
delete：用户id，用户名称可能需要删掉
create：Age range feature，fare range feature
discuss：年龄小的，性别为女的获救的几率应该比较大
train_df[['Pclass', 'Survived']].groupby(['Pclass'], as_index=False).mean().sort_values(by='Survived', ascending=False) Pclass Survived 0 1 0.629630 1 2 0.472826 2 3 0.242363
train_df[['Sex', 'Survived']].groupby(['Sex'], as_index=False).mean().sort_values(by='Survived', ascending=False) Sex Survived 0 female 0.742038 1 male 0.188908
二、可视化
Survival by Age, Class and Gender
g = sns.FacetGrid(train_df, col='Survived') g.map(plt.hist, 'Age', bins=20)
Survival by Age, Class and Gender
grid = sns.FacetGrid(train_df, col = "Pclass", row = "Sex", hue = "Survived", palette = 'seismic') grid = grid.map(plt.scatter, "PassengerId", "Age") grid.add_legend()
三、处理数据
3.1 去掉没得用的特征
删除数据中对预测没有实际效果的特征，提高模型速度，减少分析流程。
需要删除的特征有：客舱号码Cabin、票号Tickets
train_df = train_df.drop(['Ticket','Cabin'],axis=1) test_df = test_df.drop(['Ticket','Cabin'],axis=1)
3.2 建立新的特征
对人名进行分析发现，带有master的一般都活下来了，于是对人名进行拆分，提取，和.之间的数据
combine = [train_df, test_df]
for dataset in combine:
dataset['Title'] = dataset.Name.str.extract('([A-Za-z]+)\.',expand=False)
