Python数据分析库pandas ------ DataFrame
DataFrame的定义
1 data = {
2 'color': ['blue', 'green', 'yellow', 'red', 'white'],
3 'object': ['ball', 'pen', 'pecil', 'paper', 'mug'],
4 'price': [1.2, 1, 2.3, 5, 6]
5 }
6 frame0 = pd.DataFrame(data)
7 print(frame0)
8 frame1 = pd.DataFrame(data, columns=['object', 'price'])
9 print(frame1)
10 frame2 = pd.DataFrame(data, index=['张三','李斯','王五','陈久','小明'])
11 print(frame2)
12 Out[1]:
13 color object price
14 0 blue ball 1.2
15 1 green pen 1.0
16 2 yellow pecil 2.3
17 3 red paper 5.0
18 4 white mug 6.0
19 object price
20 0 ball 1.2
21 1 pen 1.0
22 2 pecil 2.3
23 3 paper 5.0
24 4 mug 6.0
25 color object price
26 张三 blue ball 1.2
27 李斯 green pen 1.0
28 王五 yellow pecil 2.3
29 陈久 red paper 5.0
30 小明 white mug 6.0
使用index参数可以设置index信息
选取元素
1 print(frame1.columns)
2 print(frame2.index)
3 print(frame2['price'])
4 print(frame2.price)
5 Out[2]:
6 Index(['object', 'price'], dtype='object')
7 Index(['张三', '李斯', '王五', '陈久', '小明'], dtype='object')
8 张三 1.2
9 李斯 1.0
10 王五 2.3
11 陈久 5.0
12 小明 6.0
13 Name: price, dtype: float64
14 张三 1.2
15 李斯 1.0
16 王五 2.3
17 陈久 5.0
18 小明 6.0
19 Name: price, dtype: float64
一般我们常需要按列取值，那么DataFrame提供了 loc 和 iloc 供大家选择，但是两者之间是由区别的。
1 print(frame2)
2 print(frame2.loc['王五']) # loc可以使用字符串类型的index,而iloc只能是int型的
3 print(frame0.iloc[2])
4 Out[3]:
5 color object price
6 张三 blue ball 1.2
7 李斯 green pen 1.0
8 王五 yellow pecil 2.3
9 陈久 red paper 5.0
10 小明 white mug 6.0
11 color yellow
12 object pecil
13 price 2.3
14 Name: 王五, dtype: object
15 color yellow
16 object pecil
17 price 2.3
18 Name: 2, dtype: object
一般取值操作
1 print(frame2[2:3]) # 取行
2 print(frame0['object']) # 取列
3 print(frame0['object'][1:3]) # 取列的元素
4 print(frame0.iloc[0:4, 1:3]) # 取一块的元素 ********************************************************************
5 Out[4]:
6 color object price
7 王五 yellow pecil 2.3
8 0 ball
9 1 pen
10 2 pecil
11 3 paper
12 4 mug
13 Name: object, dtype: object
14 1 pen
15 2 pecil
16 Name: object, dtype: object
17 object price
18 0 ball 1.2
19 1 pen 1.0
20 2 pecil 2.3
21 3 paper 5.0
元素的赋值
1 data = { 2 'color': ['blue', 'green', 'yellow', 'red', 'white'], 3 'object': ['ball', 'pen', 'pecil', 'paper', 'mug'], 4 'price': [1.2, 1, 2.3, 5, 6] 5 } 6 frame2 = pd.DataFrame(data, index=['张三', '李斯', '王五', '陈久', '小明']) 7 print("----*----\n", frame2) 8 frame2.index.name = 'usr_id' # 给index名字赋值 9 frame2.columns.name = 'item' # 给columns名字赋值 10 frame2['new'] = 12 # 给不存在的列赋值，会自动生成一列 11 print("----*----\n", frame2) 12 frame2['new'] = [3.0,1.3,2.2,0.8,1.1] # 可以指定具体不同的内容 13 print("----*----\n", frame2) 14 # 注意添加一列Series数据时，必须要注意index要一致，不一致的地方会用NaN替换 15 ser = pd.Series(np.arange(5), index=['张三', '李斯', '王五', '陈久', '小明']) 16 frame2['old'] = ser 17 print("----*----\n", frame2) 18 frame2.at['王五','price']= 22 # 改变具体一个元素的值 19 print("----*----\n", frame2) 20 Out[5]： 21 ----*---- 22 color object price 23 张三 blue ball 1.2 24 李斯 green pen 1.0 25 王五 yellow pecil 2.3 26 陈久 red paper 5.0 27 小明 white mug 6.0 28 ----*---- 29 item color object price new 30 usr_id 31 张三 blue ball 1.2 12 32 李斯 green pen 1.0 12 33 王五 yellow pecil 2.3 12 34 陈久 red paper 5.0 12 35 小明 white mug 6.0 12 36 ----*---- 37 item color object price new 38 usr_id 39 张三 blue ball 1.2 3.0 40 李斯 green pen 1.0 1.3 41 王五 yellow pecil 2.3 2.2 42 陈久 red paper 5.0 0.8 43 小明 white mug 6.0 1.1 44 ----*---- 45 item color object price new old 46 usr_id 47 张三 blue ball 1.2 3.0 0 48 李斯 green pen 1.0 1.3 1 49 王五 yellow pecil 2.3 2.2 2 50 陈久 red paper 5.0 0.8 3 51 小明 white mug 6.0 1.1 4 52 ----*---- 53 item color object price new old 54 usr_id 55 张三 blue ball 1.2 3.0 0 56 李斯 green pen 1.0 1.3 1 57 王五 yellow pecil 22.0 2.2 2 58 陈久 red paper 5.0 0.8 3 59 小明 white mug 6.0 1.1 4
赋值补充
1 print(frame2.isin([1, 'paper'])) 2 print("----*----\n", frame2[frame2.isin([1, 'paper'])]) 3 del frame2['old'] # 删除old列 4 print(frame2) 5 d1 = { 6 'red':{2012:22,2013:33}, 7 'white':{2011:13,2012:22,2013:16}, 8 'blue':{2011:17,2012:27,2013:18} 9 } 10 frame3 = pd.DataFrame(d1) 11 print(frame3) 12 print(frame3.T) 13 Out[6]: 14 item color object price new old 15 usr_id 16 张三 False False False False False 17 李斯 False False True False True 18 王五 False False False False False 19 陈久 False True False False False 20 小明 False False False False False 21 ----*---- 22 item color object price new old 23 usr_id 24 张三 NaN NaN NaN NaN NaN 25 李斯 NaN NaN 1.0 NaN 1.0 26 王五 NaN NaN NaN NaN NaN 27 陈久 NaN paper NaN NaN NaN 28 小明 NaN NaN NaN NaN NaN 29 item color object price new 30 usr_id 31 张三 blue ball 1.2 3.0 32 李斯 green pen 1.0 1.3 33 王五 yellow pecil 22.0 2.2 34 陈久 red paper 5.0 0.8 35 小明 white mug 6.0 1.1 36 red white blue 37 2011 NaN 13 17 38 2012 22.0 22 27 39 2013 33.0 16 18 40 2011 2012 2013 41 red NaN 22.0 33.0 42 white 13.0 22.0 16.0 43 blue 17.0 27.0 18.0
Index对象
1 ins = pd.Series([5,0,3,8,4],index=['red','blue','yellow','white','green'])
2 print(ins.index)
3 print(ins.idxmin()) # 返回一个索引，该索引对应的value最小
4 print(ins.idxmax()) # 返回一个索引，该索引对应的value最大
5 # 重复标签的Index
6 serd = pd.Series(range(6),index=['white','white','blue','green','green','yellow'])
7 print("serd['white']:\n", serd['white'])
8 print("判断index是否存在重复项:", serd.index.is_unique) # 判断index是否存在重复项
9 # 更换索引
10 ser = pd.Series([1,2,3,4,5],index=['one','two','three','four','five'])
11 # ser.reindex(['four','five','six','one', 'two']) # 按这里给定的顺序设置index
12 ser.reindex(['张三', '王五', '陈久', '小明', '李斯'])
13 print("Series:ser :\n", ser)
14 Out[7]:
15 Index(['red', 'blue', 'yellow', 'white', 'green'], dtype='object')
16 blue
17 white
18 serd['white']:
19 white 0
20 white 1
21 dtype: int64
22 判断index是否存在重复项: False
23 Series:ser :
24 one 1
25 two 2
26 three 3
27 four 4
28 five 5
29 dtype: int64
注意上面的 Series 用 reindex 改变了index， 但是如果在生成Series 时用了np.array()，这样是改变不了index的。
自动编制索引
1 ser2 = pd.Series([1,5,6,3],index =[0,3,5,6])
2 print(ser2)
3 print(ser2.reindex(range(6),method='ffill')) #插值，以得到一个index完整的序列（前插）,index满足range(6)
4 print(ser2.reindex(range(6),method='bfill')) #插值，以得到一个index完整的序列（后插）
5 Out[8]:
6 0 1
7 3 5
8 5 6
9 6 3
10 dtype: int64
11 0 1
12 1 1
13 2 1
14 3 5
15 4 5
16 5 6
17 dtype: int64
18 0 1
19 1 5
20 2 5
21 3 5
22 4 6
23 5 6
24 dtype: int64
删除操作
1 ser3 = pd.Series(np.arange(4.),index=['red','blue','yellow','white'])
2 print(ser3.drop('yellow')) # ser3并没有变
3 frame = pd.DataFrame(np.arange(16).reshape((4,4)),index=['blue','yellow','red','white'],columns=['ball','pen','pencil','paper'])
4 print(frame)
5 print(frame.drop(['blue','yellow'])) #默认删除行
6 print(frame.drop(['pen','pencil'],axis=1)) #删除列
7 Out[9]:
8 red 0.0
9 blue 1.0
10 white 3.0
11 dtype: float64
12 ball pen pencil paper
13 blue 0 1 2 3
14 yellow 4 5 6 7
15 red 8 9 10 11
16 white 12 13 14 15
17 ball pen pencil paper
18 red 8 9 10 11
19 white 12 13 14 15
20 ball paper
21 blue 0 3
22 yellow 4 7
23 red 8 11
24 white 12 15
DataFrame之间的运算
1 frame1 = pd.DataFrame(np.arange(16).reshape((4,4)),index=['red','blue','yellow','white'],columns=['ball','pen','pencil','paper'])
2 print(frame1)
3 frame2 = pd.DataFrame(np.arange(12).reshape((4,3)),index=['blue','green','white','yellow'],columns=['mug','pen','ball'])
4 print(frame2)
5 print(frame1 + frame2) # 等价于：frame1.add(frame2)
6 frame3 = pd.DataFrame(np.arange(16).reshape((4,4)),index=['red','blue','yellow','white'],columns=['ball','pen','pencil','paper'])
7 ser1 = pd.Series(np.arange(4),index=['ball','pen','pencil','paper'])
8 print(frame3 - ser1)
9 ser1['mug'] = 9
10 print(frame3 - ser1)
11 Out[9]:
12 ball pen pencil paper
13 red 0 1 2 3
14 blue 4 5 6 7
15 yellow 8 9 10 11
16 white 12 13 14 15
17 mug pen ball
18 blue 0 1 2
19 green 3 4 5
20 white 6 7 8
21 yellow 9 10 11
22 ball mug paper pen pencil
23 blue 6.0 NaN NaN 6.0 NaN
24 green NaN NaN NaN NaN NaN
25 red NaN NaN NaN NaN NaN
26 white 20.0 NaN NaN 20.0 NaN
27 yellow 19.0 NaN NaN 19.0 NaN
28 ball pen pencil paper
29 red 0 0 0 0
30 blue 4 4 4 4
31 yellow 8 8 8 8
32 white 12 12 12 12
33 ball mug paper pen pencil
34 red 0 NaN 0 0 0
35 blue 4 NaN 4 4 4
36 yellow 8 NaN 8 8 8
37 white 12 NaN 12 12 12
通用函数
1 frame2 = pd.DataFrame(np.arange(12).reshape((4,3)),index=['blue','green','white','yellow'],columns=['mug','pen','ball'])
2 # 通用函数,Numpy中的通用函数这里也适用
3 print(np.sqrt(frame2))
4 Out[10]:
5 mug pen ball
6 blue 0.000000 1.000000 1.414214
7 green 1.732051 2.000000 2.236068
8 white 2.449490 2.645751 2.828427
9 yellow 3.000000 3.162278 3.316625
按行按列操作的函数
1 print(frame2)
2 # 按行按列操作的函数 .apply()
3 f = lambda x: x.max() - x.min()
4 print(frame2.apply(f))
5 print(frame2.apply(f, axis=1)) # 按行执行函数f
6 def f1(x):
7 return pd.Series([x.min(),x.max()],index=['min','max'])
8 print(frame2.apply(f1))
9 Out[11]:
10 mug pen ball
11 blue 0 1 2
12 green 3 4 5
13 white 6 7 8
14 yellow 9 10 11
15 mug 9
16 pen 9
17 ball 9
18 dtype: int64
19 blue 2
20 green 2
21 white 2
22 yellow 2
23 dtype: int64
24 mug pen ball
25 min 0 1 2
26 max 9 10 11
统计函数
1 print(frame2.sum()) # 按列统计求和
2 print(frame2.describe()) # 按列做统计描述
3 Out[12]:
4 mug pen ball
5 blue 0 1 2
6 green 3 4 5
7 white 6 7 8
8 yellow 9 10 11
9 mug 18
10 pen 22
11 ball 26
12 dtype: int64
13 mug pen ball
14 count 4.000000 4.000000 4.000000
15 mean 4.500000 5.500000 6.500000
16 std 3.872983 3.872983 3.872983
17 min 0.000000 1.000000 2.000000
18 25% 2.250000 3.250000 4.250000
19 50% 4.500000 5.500000 6.500000
20 75% 6.750000 7.750000 8.750000
21 max 9.000000 10.000000 11.000000
排序
1 frame2 = pd.DataFrame(np.arange(12).reshape((4,3)),index=['blue','white','yellow','green'],columns=['mug','pen','ball'])
2 # 根据索引排序
3 ser = pd.Series([5,0,3,8,4],index=['red','blue','yellow','white','green'])
4 print(ser.sort_index())
5 print(ser.sort_index(ascending=False))
6 print(frame2.sort_index())
7 print(frame2.sort_index(axis=1))
8 # 根据对象排序
9 frame2.at['yellow','pen'] = 5.9
10 print(frame2.sort_values(by='pen'))
11 # ser.rank() 对ser进行排序，index对应着数值的序号
12 print(ser.rank()) # rank(self, axis=0, method='average', numeric_only=None, na_option='keep', ascending=True, pct=False)
13 print(ser.rank(method = 'first'))
14 print(ser.rank(ascending=False)) # 降序排位
15 print(frame2.rank()) # 按列的元素排位
16 Out[13]:
17 blue 0
18 green 4
19 red 5
20 white 8
21 yellow 3
22 dtype: int64
23 yellow 3
24 white 8
25 red 5
26 green 4
27 blue 0
28 dtype: int64
29 mug pen ball
30 blue 0 1 2
31 green 9 10 11
32 white 3 4 5
33 yellow 6 7 8
34 ball mug pen
35 blue 2 0 1
36 white 5 3 4
37 yellow 8 6 7
38 green 11 9 10
39 mug pen ball
40 blue 0 1 2
41 white 3 4 5
42 yellow 6 5 8
43 green 9 10 11
44 red 4.0
45 blue 1.0
46 yellow 2.0
47 white 5.0
48 green 3.0
49 dtype: float64
50 red 4.0
51 blue 1.0
52 yellow 2.0
53 white 5.0
54 green 3.0
55 dtype: float64
56 red 2.0
57 blue 5.0
58 yellow 4.0
59 white 1.0
60 green 3.0
61 dtype: float64
62 mug pen ball
63 blue 1.0 1.0 1.0
64 white 2.0 2.0 2.0
65 yellow 3.0 3.0 3.0
66 green 4.0 4.0 4.0
相关系数与协方差
1 seq2 = pd.Series([3,4,3,4,5,4,3,2],['2006','2007','2008','2009','2010','2011','2012','2013'])
2 seq = pd.Series([1,2,3,4,4,3,2,1],['2006','2007','2008','2009','2010','2011','2012','2013'])
3 print(seq.corr(seq2)) # 计算相关系数
4 print(seq.cov(seq2)) # 计算协方差
5 frame2 = pd.DataFrame([[1,4,3,6],[4,5,6,1],[3,3,1,5],[4,1,6,4]],index=['red','blue','yellow','white'],columns = ['ball','pen','pencil','paper'])
6 print(frame2.corr()) # 列之间两两相关系数矩阵
7 print(frame2.cov())
8 # corrwith()方法可以计算DataFrame对象的列或行与Series对象或其他DataFrame对象元素"两两"之间的相关性
9 ser = pd.Series([5,0,3,8],index=['red','blue','yellow','white'])
10 print(frame2.corrwith(ser)) # corrwith(self, other, axis=0, drop=False)
11 frame = pd.DataFrame([[1, 3, 5, 6], [5, 8, 9, 1],[3,6,4,2],[4,8,7,3]],index=['red','blue','yellow','white'],columns = ['ball','pen','pencil','paper'])
12 print(frame2.corrwith(frame))
13 Out[14]:
14 0.7745966692414835
15 0.8571428571428571
16 ball pen pencil paper
17 ball 1.000000 -0.276026 0.577350 -0.763763
18 pen -0.276026 1.000000 -0.079682 -0.361403
19 pencil 0.577350 -0.079682 1.000000 -0.692935
20 paper -0.763763 -0.361403 -0.692935 1.000000
21 ball pen pencil paper
22 ball 2.000000 -0.666667 2.000000 -2.333333
23 pen -0.666667 2.916667 -0.333333 -1.333333
24 pencil 2.000000 -0.333333 6.000000 -3.666667
25 paper -2.333333 -1.333333 -3.666667 4.666667
26 ball -0.140028
27 pen -0.869657
28 pencil 0.080845
29 paper 0.595854
30 dtype: float64
31 ball 0.966092
32 pen -0.268455
33 pencil 0.920575
34 paper 0.785714
35 dtype: float64
NaN值的操作
1 frame3 = pd.DataFrame([[6,np.nan,6],[np.nan,np.nan,np.nan],[2,np.nan,5]],index = ['blue','green','red'],columns = ['ball','mug','pen'])
2 print(frame3)
3 print(frame3.notnull()) # 输出一个布尔矩阵，True表示非空
4 print(frame3.dropna()) # 行有NaN就删除
5 print(frame3.dropna(how ='all')) # 删除全是NaN的
6 print(frame3.fillna(6.6)) #指定缺失值填充
7 print(frame3.fillna({'ball':1,'mug':0,'pen':99}))
8 Out[15]:
9 ball mug pen
10 blue 6.0 NaN 6.0
11 green NaN NaN NaN
12 red 2.0 NaN 5.0
13 ball mug pen
14 blue True False True
15 green False False False
16 red True False True
17 Empty DataFrame
18 Columns: [ball, mug, pen]
19 Index: []
20 ball mug pen
21 blue 6.0 NaN 6.0
22 red 2.0 NaN 5.0
23 ball mug pen
24 blue 6.0 6.6 6.0
25 green 6.6 6.6 6.6
26 red 2.0 6.6 5.0
27 ball mug pen
28 blue 6.0 0.0 6.0
29 green 1.0 0.0 99.0
30 red 2.0 0.0 5.0
等级索引
1 mser = pd.Series(np.random.rand(8),index=[['white','white','white','blue','blue','red','red','red'],['up','down','right','up','down','up','down','left']])
2 print(mser, "\n-----*-----\n",mser.index)
3 print(mser['white'])
4 print(mser[:,'up'])
5 print(mser['white','up'])
6 frame = mser.unstack() #把等级索引Series转换成简单的DataFrame对象
7 print(frame)
8 test = frame.stack() # 变回去
9 print("----*----\n", test)
10 mframe = pd.DataFrame(np.random.randn(16).reshape(4,4),index =[['white','white','red','red'],['up','down','up','down']],columns=[['pen','pen','paper','paper'],[1,2,1,2]])
11 print("mframe:\n", mframe)
12 mframe.columns.names =['objects','id']
13 mframe.index.names = ['colors','status']
14 print("mframe:\n", mframe)
15 mframe.swaplevel('colors','status') #互换位置
16 print("mframe:\n", mframe)
17 print("----*----\n", mframe.sort_index(level='colors')) #根据层级排序, ascending=False
18 print("----*----\n", mframe.sum(level='colors')) #按照层级统计
19 print("----*----\n", mframe.sum(level='id',axis=1)) #按照层级统计
20 Out[15]:
21 white up 0.510320
22 down 0.564982
23 right 0.253983
24 blue up 0.308429
25 down 0.895921
26 red up 0.555668
27 down 0.312702
28 left 0.680157
29 dtype: float64
30 -----*-----
31 MultiIndex(levels=[['blue', 'red', 'white'], ['down', 'left', 'right', 'up']],
32 labels=[[2, 2, 2, 0, 0, 1, 1, 1], [3, 0, 2, 3, 0, 3, 0, 1]])
33 up 0.510320
34 down 0.564982
35 right 0.253983
36 dtype: float64
37 white 0.510320
38 blue 0.308429
39 red 0.555668
40 dtype: float64
41 0.5103202702540969
42 down left right up
43 blue 0.895921 NaN NaN 0.308429
44 red 0.312702 0.680157 NaN 0.555668
45 white 0.564982 NaN 0.253983 0.510320
46 ----*----
47 blue down 0.895921
48 up 0.308429
49 red down 0.312702
50 left 0.680157
51 up 0.555668
52 white down 0.564982
53 right 0.253983
54 up 0.510320
55 dtype: float64
56 mframe:
57 pen paper
58 1 2 1 2
59 white up 0.145684 -1.665620 1.511783 -1.128178
60 down 0.364897 0.334767 0.488259 1.555273
61 red up 2.005307 0.071610 -0.778413 1.109162
62 down 1.376714 -0.478544 0.209413 -1.361551
63 mframe:
64 objects pen paper
65 id 1 2 1 2
66 colors status
67 white up 0.145684 -1.665620 1.511783 -1.128178
68 down 0.364897 0.334767 0.488259 1.555273
69 red up 2.005307 0.071610 -0.778413 1.109162
70 down 1.376714 -0.478544 0.209413 -1.361551
71 mframe:
72 objects pen paper
73 id 1 2 1 2
74 colors status
75 white up 0.145684 -1.665620 1.511783 -1.128178
76 down 0.364897 0.334767 0.488259 1.555273
77 red up 2.005307 0.071610 -0.778413 1.109162
78 down 1.376714 -0.478544 0.209413 -1.361551
79 ----*----
80 objects pen paper
81 id 1 2 1 2
82 colors status
83 red down 1.376714 -0.478544 0.209413 -1.361551
84 up 2.005307 0.071610 -0.778413 1.109162
85 white down 0.364897 0.334767 0.488259 1.555273
86 up 0.145684 -1.665620 1.511783 -1.128178
87 ----*----
88 objects pen paper
89 id 1 2 1 2
90 colors
91 white 0.510581 -1.330853 2.000042 0.427095
92 red 3.382021 -0.406933 -0.569000 -0.252389
93 ----*----
94 id 1 2
95 colors status
96 white up 1.657467 -2.793798
97 down 0.853157 1.890040
98 red up 1.226894 1.180773
99 down 1.586127 -1.840095
清澈的爱，只为中国
