hdf5文件、tqdm模块、nunique、read_csv、sort_values、astype、fillna
pandas.DataFrame.to_hdf(self, path_or_buf, key, **kwargs)：
Hierarchical Data Format (HDF) ，to add another DataFrame or Series to an existing HDF file， please use append mode and a different a key.
df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]}, index=['a', 'b', 'c'])
df.to_hdf('data.h5', key='df', mode='w', format='table')
# format : {‘fixed’, ‘table’}, default ‘fixed’
# ‘fixed’: Fixed format. Fast writing/reading. Not-appendable, nor searchable
# ‘table’: Table format. Write as a PyTables Table structure which may perform worse but allow more flexible operations like searching / selecting subsets of the data
s = pd.Series([1, 2, 3, 4])
s.to_hdf('data.h5', key='s')
pd.read_hdf('data.h5', 'df')
pd.read_hdf('data.h5', 's')
tqdm模块显示进度条：
tqdm(self, iterable=None, desc=None, total=None, leave=True, file=None, ncols=None, mininterval=0.1, maxinterval=10.0, miniters=None, ascii=None, disable=False, unit='it', unit_scale=False, dynamic_ncols=False, smoothing=0.3, bar_format=None, initial=0, position=None, postfix=None, unit_divisor=1000, write_bytes=None, gui=False, **kwargs)
iterable : iterable, optional
total : int, optional. The number of expected iterations. If unspecified, len(iterable) is used if possible.
for x in tqdm(train_df['request_timestamp'].values,total=len(train_df)): localtime=time.localtime(x) wday.append(localtime[6]) hour.append(localtime[3])
https://lorexxar.cn/2016/07/21/python-tqdm/
https://tqdm.github.io/docs/tqdm/
pandas.DataFrame.nunique：https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.nunique.html
DataFrame.nunique(self, axis=0, dropna=True)
Count distinct observations over requested axis. Return Series with number of distinct observations. Can ignore NaN values.
>>> df = pd.DataFrame({'A': [1, 2, 3], 'B': [1, 1, 1]})
>>> df
A B
0 1 1
1 2 1
2 3 1
>>> df.nunique()
A 3
B 1
dtype: int64
>>> df.nunique(axis=1)
0 1
1 2
2 2
dtype: int64
pandas.read_csv(...)常见参数：
sep : str, default ‘,’
header : int, list of int, default ‘infer’. Row number(s) to use as the column names, and the start of the data. Default behavior is to infer the column names: if no names are passed the behavior is identical to header=0 and column names are inferred from the first line of the file, if column names are passed explicitly then the behavior is identical to header=None.
names : array-like, optional. List of column names to use. Duplicates in this list are not allowed.
df=pd.read_csv('data/testA/totalExposureLog.out', sep='\t',names=['id','request_timestamp','position','uid','aid','imp_ad_size','bid','pctr','quality_ecpm','totalEcpm'])
DataFrame.sort_values(self, by, axis=0, ascending=True, inplace=False, kind='quicksort', na_position='last') # axis这个参数的默认值为0，匹配的是index，跨行进行排序，当axis=1时，匹配的是columns，跨列进行排序 # by这个参数要求传入一个字符或者是一个字符列表，用来指定按照axis的中的哪个元素来进行排序 # ascending这个参数的默认值是True，按照升序排序，当传入False时，按照降序进行排列 # kind这个参数表示按照什么样算法来进行排序，默认值是quicksort（快速排序），也可以传入mergesort（归并排序）或者是heapsort（堆排序） df.sort_values(by='col1') df.sort_values(by=['col1', 'col2'])
DataFrame.astype(self, dtype, copy=True, errors='raise', **kwargs) # dtype : data type, or dict of column name # Use a numpy.dtype or Python type to cast entire pandas object to the same type. Alternatively, use {col: dtype, …}, where col is a column label and dtype is a numpy.dtype or Python type to cast one or more of the DataFrame’s columns to column-specific types. d = {'col1': [1, 2], 'col2': [3, 4]} df = pd.DataFrame(data=d) df.dtypes df.astype('int32').dtypes df.astype({'col1': 'int32'}).dtypes
DataFrame.fillna(self, value=None, method=None, axis=None, inplace=False, limit=None, downcast=None, **kwargs) # fillna()会填充nan数据，返回填充后的结果。如果希望在原DataFrame中修改，则把inplace设置为True
