02. Pandas 1|数据结构Series、Dataframe
1."一维数组"Series
Pandas数据结构Series：基本概念及创建
s.index 、 s.values
# Series 数据结构 # Series 是带有标签的一维数组，可以保存任何数据类型（整数，字符串，浮点数，Python对象等）,轴标签统称为索引 import numpy as np
import pandas as pd
>>> s = pd.Series(np.random.rand(5)) >>> print(s,type(s)) 0 0.610318 1 0.235660 2 0.606445 3 0.070794 4 0.217530 dtype: float64 <class 'pandas.core.series.Series'> >>> print(s.index,type(s.index)) RangeIndex(start=0, stop=5, step=1) <class 'pandas.core.indexes.range.RangeIndex'> >>> print(s.values, type(s.values)) [0.61031815 0.23566007 0.60644485 0.0707941 0.21753049] <class 'numpy.ndarray'> >>> # .index查看series索引，类型为rangeindex # .values查看series值，类型是ndarray # 核心：series相比于ndarray，是一个自带索引index的数组 → 一维数组 + 对应索引 # 所以当只看series的值的时候，就是一个ndarray # series和ndarray较相似，索引切片功能差别不大 # series和dict相比，series更像一个有顺序的字典（dict本身不存在顺序），其索引原理与字典相似（一个用key，一个用index）
1.1 Series 创建方法
由字典创建，字典的key就是index，values就是values
#Series 创建方法一：由字典创建，字典的key就是index，values就是values >>> dic = {'a':1,'b':2,'c':3,'4':4,'5':5} >>> s = pd.Series(dic) >>> print(s) a 1 b 2 c 3 4 4 5 5 dtype: int64 # 注意：key肯定是字符串，假如values类型不止一个会怎么样？ → dic = {'a':1 ,'b':'hello' , 'c':3, '4':4, '5':5} >>> dic = {'a':1 ,'b':'hello' , 'c':3, '4':4, '5':5} >>> s = pd.Series(dic) >>> print(s) a 1 b hello c 3 4 4 5 5 dtype: object >>>
# Series 创建方法二：由数组创建(一维数组) >>> arr = np.random.randn(5) >>> s = pd.Series(arr) # 默认index是从0开始，步长为1的数字 >>> print(arr) [1.08349965 0.52441811 0.76972371 0.35454797 0.39607907] >>> print(s) 0 1.083500 1 0.524418 2 0.769724 3 0.354548 4 0.396079 dtype: float64 >>> >>> s = pd.Series(arr,index = ['a','b','c','d','e'],dtype = np.object) >>> print(s) a 1.083500 b 0.524418 c 0.769724 d 0.354548 e 0.396079 dtype: object # index参数：设置index，长度保持一致 # dtype参数：设置数值类型
# Series 创建方法三：由标量创建 >>> s = pd.Series(10,index = range(4)) >>> print(s) 0 10 1 10 2 10 3 10 dtype: int64 # 如果data是标量值，则必须提供索引。该值会重复，来匹配索引的长度
# Series 名称属性：name >>> s1 = pd.Series(np.random.randn(5)) >>> print(s1) 0 -0.441627 1 -0.082186 2 0.379461 3 0.163183 4 0.851316 dtype: float64 >>> s2 = pd.Series(np.random.randn(5),name='test') >>> print(s2) 0 -0.951756 1 0.039272 2 0.618596 3 -0.027975 4 0.409068 Name: test, dtype: float64 >>> print(s1.name,s2.name,type(s2.name)) None test <class 'str'> # name为Series的一个参数，创建一个数组的 名称 # .name方法：输出数组的名称，输出格式为str，如果没用定义输出名称，输出为None >>> s3 = s2.rename('hahaha') >>> print(s3) 0 -0.951756 1 0.039272 2 0.618596 3 -0.027975 4 0.409068 Name: hahaha, dtype: float64 >>> print(s3.name,s2.name) hahaha test >>> # .rename()重命名一个数组的名称，并且新指向一个数组，原数组不变
1.2 Series：索引
位置下标索引： s[0] 、s[-1]不错在报错哦 、s[1:4]左闭右开；
标签索引： s.['b'] 、s[ ['a', 'b', 'c' ] ] 、s['a':'c']末端包含哦 ；
布尔索引： s.isnull () s.notnull() s[s>50] s[ s.notnull() ]
# 位置下标，类似序列 >>> s = pd.Series(np.random.rand(5)) >>> print(s) 0 0.233801 1 0.828125 2 0.184925 3 0.297279 4 0.346561 dtype: float64 >>> print(s[0],type(s[0]),s[0].dtype) 0.23380091830372507 <class 'numpy.float64'> float64 >>> print(float(s[0]),type(float(s[0]))) 0.23380091830372507 <class 'float'> #print(s[-1]) # 位置下标从0开始 # 输出结果为numpy.float格式， # 可以通过float()函数转换为python float格式 # numpy.float与float占用字节不同 # s[-1]结果如何？ 会报错
# 标签索引 >>> s = pd.Series(np.random.rand(5),index=['a','b','c','d','e']) >>> print(s) a 0.685577 b 0.998041 c 0.451358 d 0.832554 e 0.090653 dtype: float64 >>> print(s['a'],type(s['a']),s['a'].dtype) 0.6855772922411842 <class 'numpy.float64'> float64 # 方法类似下标索引，用[]表示，内写上index，注意index是字符串 >>> sci = s[['a','b','e']] >>> print(sci,type(sci)) a 0.685577 b 0.998041 e 0.090653 dtype: float64 <class 'pandas.core.series.Series'> >>> # 如果需要选择多个标签的值，用[[]]来表示（相当于[]中包含一个列表） # 多标签索引结果是新的数组
# 切片索引 >>> s1 = pd.Series(np.random.rand(5)) >>> s2 = pd.Series(np.random.rand(5),index=['a','b','c','d','e']) >>> print(s1,'\n',s2) 0 0.917653 1 0.763179 2 0.837807 3 0.344435 4 0.360922 dtype: float64 a 0.126537 b 0.699155 c 0.289233 d 0.831209 e 0.273572 dtype: float64 >>> >>> print(s1[1:4],s1[4]) #左闭右开 1 0.763179 2 0.837807 3 0.344435 dtype: float64 0.36092197040034457 >>> print(s2['a':'c'],s2['c']) #用index做切片末端是包含的 a 0.126537 b 0.699155 c 0.289233 dtype: float64 0.28923306798234194 >>> print(s2[0:3],s2[3]) a 0.126537 b 0.699155 c 0.289233 dtype: float64 0.8312088483742163
# 注意：用index做切片是末端包含
>>> print(s2[:-1]) a 0.126537 b 0.699155 c 0.289233 d 0.831209 ##不包含末端的e dtype: float64 >>> print(s2[::2]) a 0.126537 c 0.289233 e 0.273572 dtype: float64
# 下标索引做切片，和list写法一样
# 布尔型索引 >>> s = pd.Series(np.random.rand(3)*100) >>> s[4] = None >>> print(s) 0 19.9515 1 59.9133 2 97.9854 4 None dtype: object >>> bs1 = s > 50 >>> bs2 = s.isnull() >>> bs3 = s.notnull() >>> print(bs1, type(bs1),bs1.dtype) 0 False 1 True 2 True 4 False dtype: bool <class 'pandas.core.series.Series'> bool >>> print(bs2, type(bs2),bs2.dtype) 0 False 1 False 2 False 4 True dtype: bool <class 'pandas.core.series.Series'> bool >>> print(bs3, type(bs3),bs3.dtype) 0 True 1 True 2 True 4 False dtype: bool <class 'pandas.core.series.Series'> bool >>> # 数组做判断之后，返回的是一个由布尔值组成的新的数组 # .isnull() / .notnull() 判断是否为空值 (None代表空值，NaN代表有问题的数值，两个都会识别为空值) >>> print(s[s > 50]) 1 59.9133 2 97.9854 dtype: object >>> print(s[bs3]) 0 19.9515 1 59.9133 2 97.9854 dtype: object >>> # 布尔型索引方法：用[判断条件]表示，其中判断条件可以是 一个语句，或者是 一个布尔型数组！
1.3 Series：基本技巧
数据查看（.head() .tail() ） / 重新索引就是对index做重新排序（reindex(列表)） / 对齐 （ s1+s2 ）/ 添加（s1.append(s2)）、 修改s['a']=10 、 删除值s.drop('a')
# 数据查看 >>> s = pd.Series(np.random.rand(50)) >>> print(s.head(10)) 0 0.282475 1 0.012153 2 0.642487 3 0.906513 4 0.195709 5 0.828506 6 0.194632 7 0.197138 8 0.503566 9 0.897846 dtype: float64 >>> print(s.tail()) 45 0.963916 46 0.642688 47 0.865840 48 0.835746 49 0.905786 dtype: float64 # .head()查看头部数据 # .tail()查看尾部数据 # 默认查看5条
# 重新索引reindex # .reindex将会根据索引重新排序，如果当前索引不存在，则引入缺失值 >>> s = pd.Series(np.random.rand(3),index=['a','b','c']) >>> print(s) a 0.239126 b 0.862137 c 0.501479 dtype: float64 >>> s1 = s.reindex(['c','b','a','d']) >>> print(s1) c 0.501479 b 0.862137 a 0.239126 d NaN dtype: float64 # .reindex()中也是写列表 # 这里'd'索引不存在，所以值为NaN >>> s2 = s.reindex(['c','b','a','d'],fill_value=0) # fill_value参数：填充缺失值的值
>>> print(s2)
c 0.501479
b 0.862137
a 0.239126
d 0.000000 dtype: float64
# Series对齐 >>> s1 = pd.Series(np.random.rand(3),index=['Jack','Marry','Kris']) >>> s2 = pd.Series(np.random.rand(3),index=['Wang','Jack','Marry']) >>> print(s1) Jack 0.583406 Marry 0.603579 Kris 0.812511 dtype: float64 >>> print(s2) Wang 0.582852 Jack 0.975184 Marry 0.990203 dtype: float64 >>> print(s1+s2) Jack 1.558589 Kris NaN Marry 1.593783 Wang NaN dtype: float64 # Series 和 ndarray 之间的主要区别是，Series 上的操作会根据标签自动对齐 # index顺序不会影响数值计算，以标签来计算 # 空值和任何值计算结果扔为空值
# 删除：.drop >>> s = pd.Series(np.random.rand(5),index=list('ngjur')) >>> print(s) n 0.239752 g 0.643085 j 0.313229 u 0.231923 r 0.836070 dtype: float64 >>> s1 = s.drop('n') >>> print(s1) g 0.643085 j 0.313229 u 0.231923 r 0.836070 dtype: float64 >>> s2 = s.drop(['g','j']) >>> print(s2) n 0.239752 u 0.231923 r 0.836070 dtype: float64 >>> print(s) n 0.239752 g 0.643085 j 0.313229 u 0.231923 r 0.836070 dtype: float64 # drop 删除元素之后返回副本(inplace=False)
# 添加 >>> s1 = pd.Series(np.random.rand(5)) >>> s2 = pd.Series(np.random.rand(5),index=list('ngjur')) >>> print(s1,'\n',s2) 0 0.417249 1 0.226655 2 0.798018 3 0.984398 4 0.304693 dtype: float64 n 0.354443 g 0.609306 j 0.103994 u 0.392755 r 0.302959 dtype: float64 >>> s1[5] = 100 >>> s2['a'] = 100 >>> print(s1,'\n',s2) 0 0.417249 1 0.226655 2 0.798018 3 0.984398 4 0.304693 5 100.000000 dtype: float64 n 0.354443 g 0.609306 j 0.103994 u 0.392755 r 0.302959 a 100.000000 dtype: float64 # 直接通过下标索引/标签index添加值 >>> s3 = s1.append(s2) >>> print(s3,'\n',s1) 0 0.417249 1 0.226655 2 0.798018 3 0.984398 4 0.304693 5 100.000000 n 0.354443 g 0.609306 j 0.103994 u 0.392755 r 0.302959 a 100.000000 dtype: float64 0 0.417249 1 0.226655 2 0.798018 3 0.984398 4 0.304693 5 100.000000 dtype: float64 # 通过.append方法，直接添加一个数组 # .append方法生成一个新的数组，不改变之前的数组
# 修改 >>> s = pd.Series(np.random.rand(3),index=['a','b','c']) >>> print(s) a 0.246992 b 0.349735 c 0.395859 dtype: float64 >>> s['a'] = 100 >>> s[['b','c']] = 200 >>> print(s) a 100.0 b 200.0 c 200.0 dtype: float64 >>> # 通过索引直接修改，类似序列
2. Pandas数据结构Dataframe
2.1 基本概念及创建
"二维数组"Dataframe：是一个表格型的数据结构，包含一组有序的列，其列的值类型可以是数值、字符串、布尔值等。
Dataframe中的数据以一个或多个二维块存放，不是列表、字典或一维数组结构。
# Dataframe 数据结构 # Dataframe是一个表格型的数据结构，“带有标签的二维数组”。 # Dataframe带有index（行标签）和columns（列标签） >>> data = {'name':['Jack','Tom','Marry'], ... 'age':[18,19,20], ... 'gender':['m','m','w']} >>> frame = pd.DataFrame(data) >>> print(frame) name age gender 0 Jack 18 m 1 Tom 19 m 2 Marry 20 w >>> print(type(frame)) <class 'pandas.core.frame.DataFrame'> >>> print(frame.index,'\n该数据类型为:',type(frame.index)) RangeIndex(start=0, stop=3, step=1) 该数据类型为: <class 'pandas.core.indexes.range.RangeIndex'> >>> print(frame.columns,'\n该数据类型为:',type(frame.columns)) Index(['name', 'age', 'gender'], dtype='object') 该数据类型为: <class 'pandas.core.indexes.base.Index'> >>> print(frame.values,'\n该数据类型为:',type(frame.values)) [['Jack' 18 'm'] ['Tom' 19 'm'] ['Marry' 20 'w']] 该数据类型为: <class 'numpy.ndarray'> # 查看数据，数据类型为dataframe # .index查看行标签 # .columns查看列标签 # .values查看值，数据类型为ndarray
# Dataframe 创建方法一：由数组/list组成的字典 # 创建方法:pandas.Dataframe() >>> data1 = {'a':[1,2,3], 'b':[3,4,5], 'c':[5,6,7]} >>> data2 = {'one':np.random.rand(3),'two':np.random.rand(3)} ## 这里如果尝试 'two':np.random.rand(4) 会怎么样？转为DataFrame会报错--> {'one': array([0.938673 , 0.90796881, 0.8890414 ]), 'two': array([0.37261493, 0.70430298, 0.24494145, 0.3924875 ])},转为DataFrame 则 ValueError: arrays must all be same length >>> print(data1,'\n',data2) {'a': [1, 2, 3], 'b': [3, 4, 5], 'c': [5, 6, 7]} {'one': array([0.76701471, 0.01005053, 0.09453216]), 'two': array([0.58442534, 0.14610703, 0.03588291]))
>>> df1 = pd.DataFrame(data1) >>> df2 = pd.DataFrame(data2) >>> print(df1,'\n',df2) a b c 0 1 3 5 1 2 4 6 2 3 5 7 one two 0 0.767015 0.573035 1 0.010051 0.892624 2 0.094532 0.228811 >>> # 由数组/list组成的字典 创建Dataframe，columns为字典key，index为默认数字标签 # 字典的值的长度必须保持一致！ >>> df1 = pd.DataFrame(data1,columns=['b','c','a','d']) >>> print(df1) b c a d 0 3 5 1 NaN 1 4 6 2 NaN 2 5 7 3 NaN >>> df1 = pd.DataFrame(data1,columns=['b','c']) >>> print(df1) b c 0 3 5 1 4 6 2 5 7 # columns参数：可以重新指定列的顺序，格式为list，如果现有数据中没有该列（比如'd'），则产生NaN值 # 如果columns重新指定时候，列的数量可以少于原数据 >>> df2 = pd.DataFrame(data2,index=['f1','f2','f3']) # 这里如果尝试 index = ['f1','f2','f3','f4'] 会怎么样？长度不一致，报错 >>> print(df2) one two f1 0.767015 0.573035 f2 0.010051 0.892624 f3 0.094532 0.228811 >>> # index参数：重新定义index，格式为list，长度必须保持一致
# Dataframe 创建方法二：由Series组成的字典 >>> data1 = {'one':pd.Series(np.random.rand(2)),'two':pd.Series(np.random.rand(3))} # 没有设置index的Series >>> data2 = {'one':pd.Series(np.random.rand(2),index=['a','b']),'two':pd.Series(np.random.rand(3),index=['a','b','c'])} # 设置了index的Series >>> print(data1,'\n',data2) {'one': 0 0.682455 1 0.282592 dtype: float64, 'two': 0 0.995054 1 0.781587 2 0.959304 dtype: float64} {'one': a 0.940915 b 0.792245 dtype: float64, 'two': a 0.609878 b 0.910182 c 0.245590 dtype: float64} >>> df1 = pd.DataFrame(data1) >>> df2 = pd.DataFrame(data2) >>> print(df1) one two 0 0.682455 0.995054 1 0.282592 0.781587 2 NaN 0.959304 >>> print(df2) one two a 0.940915 0.609878 b 0.792245 0.910182 c NaN 0.245590 >>> # 由Seris组成的字典 创建Dataframe，columns为字典key，index为Series的标签（如果Series没有指定标签，则是默认数字标签） # Series可以长度不一样，生成的Dataframe会出现NaN值
# Dataframe 创建方法三：通过二维数组直接创建 >>> ar = np.random.rand(9).reshape(3,3) >>> print(ar) [[0.43760945 0.3563898 0.16767573] [0.26565413 0.61673585 0.54037501] [0.95541978 0.05395517 0.02045977]] >>> df1 = pd.DataFrame(ar) >>> df2 = pd.DataFrame(ar,index=['a','b','c'],columns=['one','two','three']) >>> print(df1,'\n',df2) 0 1 2 0 0.437609 0.356390 0.167676 1 0.265654 0.616736 0.540375 2 0.955420 0.053955 0.020460 one two three a 0.437609 0.356390 0.167676 b 0.265654 0.616736 0.540375 c 0.955420 0.053955 0.020460 >>> # 通过二维数组直接创建Dataframe，得到一样形状的结果数据，如果不指定index和columns，两者均返回默认数字格式 # index和colunms指定长度与原数组保持一致
# Dataframe 创建方法四：由字典组成的列表 >>> data = [{'one':1,'two':2},{'one':5,'two':10,'three':20}] >>> print(data) [{'one': 1, 'two': 2}, {'one': 5, 'two': 10, 'three': 20}] >>> df1 = pd.DataFrame(data) >>> df2 = pd.DataFrame(data,index = ['a','b']) >>> df3 = pd.DataFrame(data,columns = ['one','two']) >>> print(df1,'\n',df2,'\n',df3) one three two 0 1 NaN 2 1 5 20.0 10 one three two a 1 NaN 2 b 5 20.0 10 one two 0 1 2 1 5 10 >>> # 由字典组成的列表创建Dataframe，columns为字典的key，index不做指定则为默认数组标签 # colunms和index参数分别重新指定相应列及行标签
# Dataframe 创建方法五：由字典组成的字典 data = {'Jack':{'math':90,'english':89,'art':78}, 'Marry':{'math':82,'english':95,'art':92}, 'Tom':{'math':78,'english':67}} df1 = pd.DataFrame(data) print(df1) # 由字典组成的字典创建Dataframe，columns为字典的key，index为子字典的key df2 = pd.DataFrame(data, columns = ['Jack','Tom','Bob']) df3 = pd.DataFrame(data, index = ['a','b','c']) print(df2) print(df3) # columns参数可以增加和减少现有列，如出现新的列，值为NaN # index在这里和之前不同，并不能改变原有index，如果指向新的标签，值为NaN （非常重要！） #在cmd或pycharm里边报错。AttributeError: 'list' object has no attribute 'astype' Jack Marry Tom art 78 92 NaN english 89 95 67.0 math 90 82 78.0 Jack Tom Bob art 78 NaN NaN english 89 67.0 NaN math 90 78.0 NaN Jack Marry Tom a NaN NaN NaN b NaN NaN NaN c NaN NaN NaN
2.2 Dataframe：索引
Dataframe既有行索引也有列索引，可以被看做由Series组成的字典（共用一个索引）
选择列 / 选择行 / 切片 / 布尔判断
df [ 'a' ] df [ ['a', 'b'] ] 选择列 、 df.loc [ 'one' ] 按index选择行
#选择行df.loc[] 与列 df[ ]
>>> df = pd.DataFrame(np.random.rand(12).reshape(3,4)*100,index=['one','two','three'],columns=['a','b','c','d']) >>> print(df) a b c d one 15.715854 86.084608 22.376152 66.760504 two 3.761389 63.610935 85.752549 19.065568 three 77.277233 24.776938 13.159774 46.518796 >>> data1 = df['a'] >>> data2 = df[['a','c']] >>> print(data1,type(data1)) one 15.715854 two 3.761389 three 77.277233 Name: a, dtype: float64 <class 'pandas.core.series.Series'> >>> print(data2,type(data2)) a c one 15.715854 22.376152 two 3.761389 85.752549 three 77.277233 13.159774 <class 'pandas.core.frame.DataFrame'> >>> # 按照列名选择列，只选择一列输出Series，选择多列输出Dataframe >>> data3 = df.loc['one'] >>> data4 = df.loc[['one','two']] >>> print(data3,type(data3)) a 15.715854 b 86.084608 c 22.376152 d 66.760504 Name: one, dtype: float64 <class 'pandas.core.series.Series'> >>> print(data4,type(data4)) a b c d one 15.715854 86.084608 22.376152 66.760504 two 3.761389 63.610935 85.752549 19.065568 <class 'pandas.core.frame.DataFrame'> >>> # 按照index选择行，只选择一行输出Series，选择多行输出Dataframe
2.2.1 df[ ] -- 选择列
#1. df[] - 选择列 # 一般用于选择列，也可以选择行 >>> df = pd.DataFrame(np.random.rand(12).reshape(3,4)*100,index=['one','two','three'],columns=['a','b','c','d']) >>> print(df) a b c d one 94.536247 33.478780 10.738060 52.679418 two 37.573186 95.915130 8.529743 11.367094 three 80.758763 0.000355 36.136580 95.739389 >>> data1 = df['a'] >>> data2 = df[['b','c']] # 尝试输入 data2 = df[['b','c','e']]会报错 >>> print(data1) one 94.536247 two 37.573186 three 80.758763 Name: a, dtype: float64 >>> print(data2) b c one 33.478780 10.738060 two 95.915130 8.529743 three 0.000355 36.136580 >>> # df[]默认选择列，[]中写列名（所以一般数据colunms都会单独制定，不会用默认数字列名，以免和index冲突） # 单选列为Series，print结果为Series格式 # 多选列为Dataframe，print结果为Dataframe格式 >>> data3 = df[:1] #data3 = df[0] #这两种都是错误的，0 'one' #data3 = df['one'] >>> print(data3,type(data3)) a b c d one 94.536247 33.47878 10.73806 52.679418 <class 'pandas.core.frame.DataFrame'> # df[]中为数字时，默认选择行，且只能进行切片的选择，不能单独选择（df[0]） # 输出结果为Dataframe，即便只选择一行 # df[]不能通过索引标签名来选择行(df['one']) # 核心笔记：df[col]一般用于选择列，[]中写列名
2.2.2df.loc[ ] - 按index选择行
#2. df.loc[] - 按index选择行 >>> df1 = pd.DataFrame(np.random.rand(16).reshape(4,4)*100,index=['one','two','three','four'],columns=['a','b','c','d']) >>> df2 = pd.DataFrame(np.random.rand(16).reshape(4,4)*100,columns=['a','b','c','d']) >>> print(df1,'\n',df2) a b c d one 36.881890 13.897714 5.237098 24.676327 two 42.183000 27.146129 49.074872 56.447147 three 6.935006 16.742130 5.955048 2.576066 four 49.843982 64.641184 70.038643 75.103787 a b c d 0 60.589246 60.305811 90.306763 46.761824 1 59.296330 6.039652 52.296003 97.149954 2 58.255476 13.837192 74.255506 84.082167 3 55.204207 17.340171 25.056553 84.518804 #单标签索引，返回Series >>> data1 = df1.loc['one'] #单标签索引返回Series >>> data2 = df2.loc[1] >>> print(data1,'\n',data2) a 36.881890 b 13.897714 c 5.237098 d 24.676327 Name: one, dtype: float64 a 59.296330 b 6.039652 c 52.296003 d 97.149954 Name: 1, dtype: float64 >>> #多标签索引,顺序可变 >>> data3 = df1.loc[['two','three','five']] >>> data4 = df2.loc[[3,2,1]] >>> print(data3) a b c d two 42.183000 27.146129 49.074872 56.447147 three 6.935006 16.742130 5.955048 2.576066 five NaN NaN NaN NaN #多标签索引，如果标签不存在则返回NaN >>> print(data4) a b c d 3 55.204207 17.340171 25.056553 84.518804 2 58.255476 13.837192 74.255506 84.082167 1 59.296330 6.039652 52.296003 97.149954 #切片索引 ，可以做切片对象 >>> data5 = df1.loc['one':'three'] #末端包含 >>> data6 = df2.loc[1:3] >>> print(data5) a b c d one 36.881890 13.897714 5.237098 24.676327 two 42.183000 27.146129 49.074872 56.447147 three 6.935006 16.742130 5.955048 2.576066 >>> print(data6) a b c d 1 59.296330 6.039652 52.296003 97.149954 2 58.255476 13.837192 74.255506 84.082167 3 55.204207 17.340171 25.056553 84.518804 # 核心笔记：df.loc[label]主要针对index选择行，同时支持指定index，及默认数字index
2.2.3 df.iloc[ ] - 按整数位置选择行
# df.iloc[] - 按照整数位置（从轴的0到length-1）选择行 # 类似list的索引，其顺序就是dataframe的整数位置，从0开始计 >>> df = pd.DataFrame(np.random.rand(16).reshape(4,4)*100,index=['one','two','three','four'],columns=['a','b','c','d']) >>> print(df) a b c d one 21.693396 38.203531 85.439983 9.740751 two 28.940287 57.861274 68.467893 60.788056 three 81.871777 57.813973 60.092876 1.637220 four 67.789269 95.648501 62.837383 65.794259 # 单位置索引; 和loc索引不同，不能索引超出数据行数的整数位置 >>> print(df.iloc[0]) a 21.693396 b 38.203531 c 85.439983 d 9.740751 Name: one, dtype: float64 >>> print(df.iloc[-1]) a 67.789269 b 95.648501 c 62.837383 d 65.794259 Name: four, dtype: float64 >>> print(df.iloc[4]) #索引超过行数了 IndexError: single positional indexer is out-of-bounds # 多位置索引，顺序可变 >>> print(df.iloc[[0,2]]) ##从0开始 ，第0行和第3行即末端包含 a b c d one 21.693396 38.203531 85.439983 9.740751 three 81.871777 57.813973 60.092876 1.637220 >>> print(df.iloc[[3,2,1]]) a b c d four 67.789269 95.648501 62.837383 65.794259 three 81.871777 57.813973 60.092876 1.637220 two 28.940287 57.861274 68.467893 60.788056 #切片索引 >>> print(df.iloc[1:3]) #末端不包含 a b c d two 28.940287 57.861274 68.467893 60.788056 three 81.871777 57.813973 60.092876 1.637220 >>> print(df.iloc[::2]) a b c d one 21.693396 38.203531 85.439983 9.740751 three 81.871777 57.813973 60.092876 1.637220 >>>
2.2.4布尔型索引
df < 20 df [ df < 20 ] 、 单列做判断df [ 'a' ] >20 、多列做判断df [ ['a', 'b'] ] >20 、多行做判断 df.loc[ ['one', 'three'] ] < 50
#布尔型索引 # 和Series原理相同 >>> df = pd.DataFrame(np.random.rand(16).reshape(4,4)*100,index=['one','two','three','four'],columns=['a','b','c','d']) >>> print(df) a b c d one 17.575951 66.534852 96.774872 94.415801 two 67.485820 11.871447 19.140092 9.634462 three 32.052532 8.891445 63.209949 92.451412 four 6.931403 0.622515 29.972335 24.438536 >>> b1 = df < 20 # 也可以书写为 df[df < 20] >>> print(b1,type(b1)) a b c d one True False False False two False True True True three False True False False four True True False False <class 'pandas.core.frame.DataFrame'> >>> print(df[b1]) a b c d one 17.575951 NaN NaN NaN two NaN 11.871447 19.140092 9.634462 three NaN 8.891445 NaN NaN four 6.931403 0.622515 NaN NaN >>> # 不做索引则会对数据每个值进行判断 # 索引结果保留 所有数据：True返回原数据，False返回值为NaN
>>> b2 = df['a'] > 50 >>> print(b2,type(b2)) one False two True three False four False Name: a, dtype: bool <class 'pandas.core.series.Series'> >>> print(df[b2]) #会把two为True的行保留，包括小于50的数 a b c d two 67.48582 11.871447 19.140092 9.634462 # 单列做判断，索引结果保留单列判断为True的行数据，包括其他列
>>> b3 = df[['a','b']] > 50 >>> print(b3,type(b3)) a b one False True two True False three False False four False False <class 'pandas.core.frame.DataFrame'> >>> print(df[b3]) a b c d one NaN 66.534852 NaN NaN two 67.48582 NaN NaN NaN three NaN NaN NaN NaN four NaN NaN NaN NaN # 多列做判断，索引结果保留所有数据：True返回原数据，False返回值为NaN
>>> >>> b4 = df.loc[['one','three']] < 50 >>> print(b4,type(b4)) a b c d one True False False False three True True False False <class 'pandas.core.frame.DataFrame'> >>> print(df[b4]) a b c d one 17.575951 NaN NaN NaN two NaN NaN NaN NaN three 32.052532 8.891445 NaN NaN four NaN NaN NaN NaN >>> # 多行做判断，索引结果保留 所有数据：True返回原数据，False返回值为NaN
2.2.5 多重索引：比如同时索引行和列
先选择列再选择行 ：df[ 'a' ].loc[ ['a', 'b', 'c'] ] df [ df [ 'a' ] < 50 ].iloc[ :2 ]
#多重索引：比如同时索引行和列 # 先选择列再选择行 —— 相当于对于一个数据，先筛选字段，再选择数据量 >>> df = pd.DataFrame(np.random.rand(16).reshape(4,4)*100,index=['one','two','three','four'],columns=['a','b','c','d']) >>> print(df) a b c d one 12.408141 98.202562 38.715980 62.978631 two 93.980397 39.455335 77.214844 42.495949 three 4.210569 48.999179 10.320513 51.919796 four 73.838276 72.854442 98.555301 27.902682 >>> print(df['a'].loc[['one','three']]) # 选择a列的one，three行 one 12.408141 three 4.210569 Name: a, dtype: float64 >>> print(df[['b','c','d']].iloc[::2]) # 选择b，c，d列的one，three行 b c d one 98.202562 38.715980 62.978631 three 48.999179 10.320513 51.919796 >>> print(df[df['a'] < 50].iloc[:2]) # 选择满足判断索引的前两行数据 a b c d one 12.408141 98.202562 38.715980 62.978631 three 4.210569 48.999179 10.320513 51.919796 >>>
2.3 Dataframe：基本技巧
数据查看、转置 / 添加、修改、删除值 / 对齐 / 排序
######数据查看（.head() .tail() ）与转置（ .T ）
>>> df = pd.DataFrame(np.random.rand(16).reshape(8,2)*100,columns=['a','b']) >>> print(df) a b 0 41.447858 93.937878 1 29.684415 58.637993 2 2.260561 23.601327 3 79.555013 55.611010 4 64.825361 92.444769 5 53.716091 40.166872 6 19.657354 47.842487 7 22.705715 26.977886 >>> >>> print(df.head(2)) a b 0 41.447858 93.937878 1 29.684415 58.637993 >>> print(df.tail()) a b 3 79.555013 55.611010 4 64.825361 92.444769 5 53.716091 40.166872 6 19.657354 47.842487 7 22.705715 26.977886
# .head()查看头部数据
# .tail()查看尾部数据
# 默认查看5条
>>> print(df.T) 0 1 2 3 4 5 6 7 a 41.447858 29.684415 2.260561 79.555013 64.825361 53.716091 19.657354 22.705715 b 93.937878 58.637993 23.601327 55.611010 92.444769 40.166872 47.842487 26.977886
# .T 转置
# 添加与修改 >>> df = pd.DataFrame(np.random.rand(16).reshape(4,4)*100,columns=['a','b','c','d']) >>> print(df) a b c d 0 40.395591 38.023720 64.954712 82.833601 1 69.405393 77.664903 76.566145 11.218753 2 61.793220 95.929196 15.415231 79.368691 3 29.482119 85.228170 94.134330 25.678733 >>> df['e'] = 10 >>> df.loc[4] = 20 >>> print(df) a b c d e 0 40.395591 38.023720 64.954712 82.833601 10 1 69.405393 77.664903 76.566145 11.218753 10 2 61.793220 95.929196 15.415231 79.368691 10 3 29.482119 85.228170 94.134330 25.678733 10 4 20.000000 20.000000 20.000000 20.000000 20 >>> # 新增列/行并赋值 >>> df['e'] = 20 >>> df[['a','c']] = 100 >>> print(df) a b c d e 0 100 38.023720 100 82.833601 20 1 100 77.664903 100 11.218753 20 2 100 95.929196 100 79.368691 20 3 100 85.228170 100 25.678733 20 4 100 20.000000 100 20.000000 20 >>> # 索引后直接修改值
# 删除 del / drop() ；inplace = False/True 、 axis = 0 为行 | axis = 1 为列 >>> df = pd.DataFrame(np.random.rand(16).reshape(4,4)*100,columns=['a','b','c','d']) >>> print(df) a b c d 0 76.082974 91.636219 70.831268 82.900443 1 16.328769 9.910538 36.670726 67.187492 2 96.234567 16.699254 0.257354 31.032239 3 16.659137 85.438085 91.993957 33.055454 >>> del df['a'] >>> print(df) b c d 0 91.636219 70.831268 82.900443 1 9.910538 36.670726 67.187492 2 16.699254 0.257354 31.032239 3 85.438085 91.993957 33.055454 >>> # del语句 - 删除列
>>> print(df.drop(0)) ##删除行 b c d 1 9.910538 36.670726 67.187492 2 16.699254 0.257354 31.032239 3 85.438085 91.993957 33.055454 >>> print(df.drop([1,2])) b c d 0 91.636219 70.831268 82.900443 3 85.438085 91.993957 33.055454 >>> print(df) ##原数据不改变 b c d 0 91.636219 70.831268 82.900443 1 9.910538 36.670726 67.187492 2 16.699254 0.257354 31.032239 3 85.438085 91.993957 33.055454 >>> # drop()删除行，inplace=False → 删除后生成新的数据，不改变原数据
>>> print(df1.drop(['1'], inplace=True)) #默认为 inplace=False, inplace = True是把原数据也删除了
None
>>> print(df.drop(['d'],axis = 1)) #axis = 1是删除列 ---> 用 [ ]，不改变原数据 ; axis = 0是删除行,不改变原数据。 b c 0 91.636219 70.831268 1 9.910538 36.670726 2 16.699254 0.257354 3 85.438085 91.993957 >>> print(df) b c d 0 91.636219 70.831268 82.900443 1 9.910538 36.670726 67.187492 2 16.699254 0.257354 31.032239 3 85.438085 91.993957 33.055454 >>> # drop()删除列，需要加上axis = 1，inplace=False → 删除后生成新的数据，不改变原数据
#对齐 + >>> df1 = pd.DataFrame(np.random.randn(10,4),columns=['A','B','C','D']) >>> df2 = pd.DataFrame(np.random.randn(7,3),columns=['A','B','C']) >>> print(df1) A B C D 0 -0.711905 1.102947 -0.203125 0.464160 1 -1.633976 -0.126530 1.437948 1.721049 2 1.323383 -0.277546 0.060134 0.207093 3 1.708294 0.815721 -0.151322 0.522937 4 0.263572 -0.674251 -1.325148 -2.702464 5 1.659823 -0.131172 -1.114735 -2.182527 6 -0.186723 -0.071455 -1.370213 0.513062 7 0.381603 1.265310 0.083247 1.084061 8 0.399770 0.765438 -1.066299 0.626402 9 0.781321 -1.612135 -0.387417 -0.673143 >>> print(df2) A B C 0 0.012025 -0.488556 0.243515 1 -0.751000 0.277448 0.013675 2 1.008712 -1.231084 -0.523329 3 0.663029 -0.752602 -0.724749 4 -0.755075 0.303930 1.288335 5 -1.233975 -1.241185 -0.414564 6 -0.251519 -1.384259 -0.996120 >>> print(df1+df2) #DataFrame对象之间的数据自动按照列和索引（行标签）对齐。 A B C D 0 -0.699879 0.614391 0.040390 NaN 1 -2.384977 0.150917 1.451622 NaN 2 2.332095 -1.508629 -0.463195 NaN 3 2.371323 0.063119 -0.876071 NaN 4 -0.491503 -0.370321 -0.036813 NaN 5 0.425847 -1.372357 -1.529299 NaN 6 -0.438242 -1.455714 -2.366333 NaN 7 NaN NaN NaN NaN 8 NaN NaN NaN NaN 9 NaN NaN NaN NaN >>>
# 排序1 - 按值排序 .sort_values # 同样适用于Series >>> df1 = pd.DataFrame(np.random.rand(16).reshape(4,4)*100,columns=['a','b','c','d']) >>> print(df) b c d 0 91.636219 70.831268 82.900443 1 9.910538 36.670726 67.187492 2 16.699254 0.257354 31.032239 3 85.438085 91.993957 33.055454 >>> print(df1.sort_values(['a'],ascending=True)) #升序； ascending参数：设置升序降序，默认升序。 a b c d 0 3.255012 35.188882 99.290551 67.897580 1 43.221583 36.144081 84.124544 18.844967 3 47.364524 41.530226 20.800088 22.597198 2 83.170528 1.550416 7.810286 61.375057 >>> print(df1.sort_values(['a'],ascending=False)) #降序 a b c d 2 83.170528 1.550416 7.810286 61.375057 3 47.364524 41.530226 20.800088 22.597198 1 43.221583 36.144081 84.124544 18.844967 0 3.255012 35.188882 99.290551 67.897580 # 单列排序
>>> df2 = pd.DataFrame({'a':[1,1,1,1,2,2,2,2],'b':list(range(8)),'c':list(range(8,0,-1))}) >>> print(df2) a b c 0 1 0 8 1 1 1 7 2 1 2 6 3 1 3 5 4 2 4 4 5 2 5 3 6 2 6 2 7 2 7 1 >>> print(df2.sort_values(['a','c'])) #多列排序，按列顺序排序。 a b c 3 1 3 5 2 1 2 6 1 1 1 7 0 1 0 8 7 2 7 1 6 2 6 2 5 2 5 3 4 2 4 4 >>>
# 排序2 - 索引排序 .sort_index >>> df1 = pd.DataFrame(np.random.rand(16).reshape(4,4)*100,index=[5,4,3,2],columns=['a','b','c','d']) >>> df2 = pd.DataFrame(np.random.rand(16).reshape(4,4)*100,index=['h','s','x','g'],columns=['a','b','c','d']) >>> print(df1) a b c d 5 70.899006 29.653652 38.273239 99.254931 4 68.173016 27.051275 43.236560 48.573018 3 35.870577 41.990773 78.055733 63.581352 2 20.946046 19.712039 33.906534 89.749668 >>> print(df1.sort_index()) a b c d 2 20.946046 19.712039 33.906534 89.749668 3 35.870577 41.990773 78.055733 63.581352 4 68.173016 27.051275 43.236560 48.573018 5 70.899006 29.653652 38.273239 99.254931 >>> print(df2) a b c d h 62.234181 32.481881 83.483145 39.145470 s 41.003081 16.515826 19.958257 30.331726 x 60.486728 20.206607 91.149820 31.731089 g 22.132468 61.116998 19.929379 98.976248 >>> print(df2.sort_index()) a b c d g 22.132468 61.116998 19.929379 98.976248 h 62.234181 32.481881 83.483145 39.145470 s 41.003081 16.515826 19.958257 30.331726 x 60.486728 20.206607 91.149820 31.731089 >>> # 按照index排序 # 默认 ascending=True, inplace=False
透视表(pivotTab)和交叉表(crossTab)
pandas.crosstab| pivotTab
- pandas.crosstab(index, columns, values=None, rownames=None, colnames=None, aggfunc=None, margins=False, margins_name='All', dropna=True, normalize=False)[source]#
-
Compute a simple cross tabulation of two (or more) factors.
>>> import numpy as np >>> import pandas as pd >>> from pandas import Series,DataFrame >>> >>> >>> df = DataFrame({'类别':['水果','水果','水果','蔬菜','蔬菜','肉类','肉类'], ... '产地':['美国','中国','中国','中国','新西兰','新西兰','美国'], ... '水果':['苹果','梨','草莓','番茄','黄瓜','羊肉','牛肉'], ... '数量':[5,5,9,3,2,10,8], ... '价格':[5,5,10,3,3,13,20]}) >>> >>> df 类别 产地 水果 数量 价格 0 水果 美国 苹果 5 5 1 水果 中国 梨 5 5 2 水果 中国 草莓 9 10 3 蔬菜 中国 番茄 3 3 4 蔬菜 新西兰 黄瓜 2 3 5 肉类 新西兰 羊肉 10 13 6 肉类 美国 牛肉 8 20 >>> df.pivot_table(index=['产地','类别']) 价格 数量 产地 类别 中国 水果 7.5 7 蔬菜 3.0 3 新西兰 肉类 13.0 10 蔬菜 3.0 2 美国 水果 5.0 5 肉类 20.0 8 >>> >>> df.pivot_table(columns=['产地','类别']) 产地 中国 新西兰 美国 类别 水果 蔬菜 肉类 蔬菜 水果 肉类 价格 7.5 3.0 13.0 3.0 5.0 20.0 数量 7.0 3.0 10.0 2.0 5.0 8.0 >>> >>> df.pivot_table('价格',index='产地',columns='类别',aggfunc='max',margins=True,fill_value=0) 类别 水果 肉类 蔬菜 All 产地 中国 10 0 3 10 新西兰 0 13 3 13 美国 5 20 0 20 All 10 20 3 20 >>> >>> pd.crosstab(df['类别'],df['产地'],margins=True) 产地 中国 新西兰 美国 All 类别 水果 2 0 1 3 肉类 0 1 1 2 蔬菜 1 1 0 2 All 3 2 2 7 >>>
div()
语法: DataFrame.div(other, axis=’columns’, level=None, fill_value=None)
参数:
other：系列，数据框架，或常数
axis：对于系列输入，axis要与系列的索引相匹配。
fill_value :用这个值来填补缺失（NaN）的值。如果两个DataFrame的位置都缺失，结果将是缺失。
level :跨层广播，与通过的MultiIndex层的索引值相匹配。
返回： result : DataFrame
例子#1：使用div()函数来寻找数据帧元素的浮动除法，其值为常数。同时处理数据帧中存在的NaN值。
>>> df = pd.DataFrame({"A":[5, 3, None, 4],
... "B":[None, 2, 4, 3],
... "C":[4, 3, 8, 5],
... "D":[5, 4, 2, None]})
>>> df
A B C D
0 5.0 NaN 4 5.0
1 3.0 2.0 3 4.0
2 NaN 4.0 8 2.0
3 4.0 3.0 5 NaN
>>> df.di
df.diff( df.div( df.divide(
>>> df.div(2,fill_value=50)
A B C D
0 2.5 25.0 2.0 2.5
1 1.5 1.0 1.5 2.0
2 25.0 2.0 4.0 1.0
3 2.0 1.5 2.5 25.0
使用div()函数来查找一个带有系列对象的数据框架在索引axis上的浮动分割。
输出是一个数据框架，其中的单元格包含了当前单元格元素与相应系列对象单元格的划分结果。
>>> df = pd.DataFrame({"A":[5, 3, 6, 4],
... "B":[11, 2, 4, 3],
... "C":[4, 3, 8, 5],
... "D":[5, 4, 2, 8]})
>>>
>>> df
A B C D
0 5 11 4 5
1 3 2 3 4
2 6 4 8 2
3 4 3 5 8
>>> seris_obj=pd.Series([2,3,1.5,4])
>>>
>>> seris_obj
0 2.0
1 3.0
2 1.5
3 4.0
dtype: float64
>>>
>>> df.div(seris_obj,axis=0)
A B C D
0 2.5 5.500000 2.000000 2.500000
1 1.0 0.666667 1.000000 1.333333
2 4.0 2.666667 5.333333 1.333333
3 1.0 0.750000 1.250000 2.000000
