Feature Engineering

 

常规操作：

选择列

print(data[["prov_id","gender_0"]])

 

合并列：

data = data.join

 

JOIN：

pd.merge

 

UNION

pd.concat

 

删除列

data = data.drop(['prov_id','gender_id'],axis=1)

 

异常值处理：

1. remove skew

Why: 

   Many model built on the hypothsis that the input data are distributed as a 'Normal Distribution'(Gaussian Distribution). So if the input data is more like Normal Distribution, the results are better.

Methods:

•     remove skewnewss: log function.

 

2. standardization

Why:

    Different data have different scale, to avoid give to high weight to those data with large scale.

Methods:

•     min-max = (data - min) / (max - min)
•     z-score = (data - mean) / (sd), sd standard deviation

 

3. manual remove

Why:

    sometimes we know that some columns are meanless, so we just remove it manually.

Method:

•     columns like "ID", "timestamp"

 

4. remove columns with too many nulls

Why:

    if a feature has too many nulls, it's not reliable.

Method:

•     count the percentage of nulls.

 

5. drop outlier

Why:

     outliers are the special cases for a set of data. they don't represent the common experience. so they will not contribute to a model, on the contrary, they will be harmful for our models.

Methods:

•     remove data that >= an extreme value, or <= an extreme value.

 

6. convert enum to one-hot

good: ignore order

bad: too dimension for big value space

 

in pandas:

pd.get_dummies

 

data.join(data2)