[Machine Learning with Python] My First Data Preprocessing Pipeline with Titanic Dataset

The Dataset was acquired from https://www.kaggle.com/c/titanic

For data preprocessing, I firstly defined three transformers:

• DataFrameSelector: Select features to handle.
• CombinedAttributesAdder: Add a categorical feature Age_cat which divided all passengers into three catagories according to their ages.
• ImputeMostFrequent: Since the SimpleImputer( ) method was only suitable for numerical variables, I wrote an transformer to impute string missing values with the mode value. Here I was inspired by https://stackoverflow.com/questions/25239958/impute-categorical-missing-values-in-scikit-learn.

Then I wrote pipelines separately for different features

• For numerical features, I applied DataFrameSelector, SimpleImputer and StandardScaler
• For categorical features, I applied DataFrameSelector, ImputeMostFrequent and OneHotEncoder
• For the new created feature Age_cat, since itself was a category but was derived from a numerical feature, I wrote an individual pipeline to impute the missing values and encode the categories.

Finally, we can build a full pipeline through FeatureUnion. Here is the code:

# Read data
import pandas as pd
import numpy as np
import os
titanic_train = pd.read_csv('Dataset/Titanic/train.csv')
titanic_test = pd.read_csv('Dataset/Titanic/test.csv')
submission = pd.read_csv('Dataset/Titanic/gender_submission.csv')

# Divide attributes and labels
titanic_labels = titanic_train['Survived'].copy()
titanic = titanic_train.drop(['Survived'],axis=1)

# Feature Selection
from sklearn.base import BaseEstimator, TransformerMixin

class DataFrameSelector(BaseEstimator, TransformerMixin):  
    def __init__(self,attribute_name):
        self.attribute_name = attribute_name
    def fit(self, X):
        return self
    def transform (self, X, y=None):
        if 'Pclass' in self.attribute_name:
            X['Pclass'] = X['Pclass'].astype(str)
        return X[self.attribute_name]

# Feature Creation
class CombinedAttributesAdder(BaseEstimator, TransformerMixin):
    def fit(self, X, y=None):
        return self  # nothing else to do
    def transform(self, X, y=None):
        Age_cat = pd.cut(X['Age'],[0,18,60,100],labels=['child', 'adult', 'old'])
        Age_cat=np.array(Age_cat)        
        return pd.DataFrame(Age_cat,columns=['Age_Cat'])

# Impute Categorical variables
class ImputeMostFrequent(BaseEstimator, TransformerMixin):      
    def fit(self, X, y=None):
        self.fill = pd.Series([X[c].value_counts().index[0] for c in X],index=X.columns)
        return self
    def transform(self, X, y=None):
        return X.fillna(self.fill)

#Pipeline
from sklearn.impute import SimpleImputer # Scikit-Learn 0.20+
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import OneHotEncoder
from sklearn.pipeline import FeatureUnion

num_pipeline = Pipeline([
    ('selector',DataFrameSelector(['Age','SibSp','Parch','Fare'])),
    ('imputer', SimpleImputer(strategy="median")),
    ('std_scaler', StandardScaler()),
])

cat_pipeline = Pipeline([
    ('selector',DataFrameSelector(['Pclass','Sex','Embarked'])),
    ('imputer',ImputeMostFrequent()),
    ('encoder', OneHotEncoder()),
])

new_pipeline = Pipeline([
    ('selector',DataFrameSelector(['Age'])),
    #('imputer', SimpleImputer(strategy="median")),
    ('attr_adder',CombinedAttributesAdder()),
    ('imputer',ImputeMostFrequent()),
    ('encoder', OneHotEncoder()),
])

full_pipeline = FeatureUnion([
    ("num", num_pipeline),
    ("cat", cat_pipeline),
    ("new", new_pipeline),
])

titanic_prepared = full_pipeline.fit_transform(titanic)

Another thing I want to mention is that the output of a pipeline should be a 2D array rather a 1D array. So if you wanna choose only one feature, don't forget to transform the 1D array by reshape() method. Otherwise, you will receive an error like

ValueError: Expected 2D array, got 1D array instead

Specifically, apply reshape(-1,1) for column and reshape(1,-1). More about the issue can be found at https://stackoverflow.com/questions/51150153/valueerror-expected-2d-array-got-1d-array-instead.