Pandas学习1
pandas有两种自己独有的基本数据结构Series和DataFrame
Series
数据结构
data 100 300 500
index 0 1 2
或者
index data
0 100
1 300
2 500
创建series对象
In [1]: import numpy as np
In [2]: from pandas import Series,DataFrame
In [3]: import pandas as pd
传递list创建对象，默认创建整数索引
In [4]: s1 = Series([1,3,6,-1,2,8])
In [5]: s1
Out[5]:
0 1
1 3
2 6
3 -1
4 2
5 8
dtype: int64
传入列表自定义索引创建对象
In [9]: s2 = Series([1,3,6,-1,2,8],index = ["a","c","d","e","b","g"])
In [10]: s2
Out[10]:
a 1
c 3
d 6
e -1
b 2
g 8
dtype: int64
传入字典创建对象
In [11]: SD = {"python":100,"java":101,"scala":102}
In [12]: s3 = Series(SD)
In [14]: s3
Out[14]:
java 101
python 100
scala 102
dtype: int64
//显示数据值【values】和索引【index】
In [6]: s1.values
Out[6]: array([ 1, 3, 6, -1, 2, 8])
In [7]: s1.index
Out[7]: RangeIndex(start=0, stop=6, step=1)
In [17]: s1
Out[17]:
0 1
1 3
2 6
3 -1
4 2
5 8
dtype: int64
自定义索引名字
In [18]: s1.index = ["p1","p2","p3","p4","p5","p6"]
In [19]: s1
Out[19]:
p1 1
p2 3
p3 6
p4 -1
p5 2
p6 8
dtype: int64
根据索引查看值和修改值
In [20]: s1['p1']
Out[20]: 1
In [21]: s1['p1']=100
In [22]: s1
Out[22]:
p1 100
p2 3
p3 6
p4 -1
p5 2
p6 8
dtype: int64
判断值是否为空
In [29]: pd.isnull(s1)
Out[29]:
p1 False
p2 False
p3 False
p4 False
p5 False
p6 False
dtype: bool
In [30]: pd.notnull(s1)
Out[30]:
p1 True
p2 True
p3 True
p4 True
p5 True
p6 True
dtype: bool
运算
In [31]: s2
Out[31]:
a 1
c 3
d 6
e -1
b 2
g 8
dtype: int64
In [32]: s2[s2>5]
Out[32]:
d 6
g 8
dtype: int64
In [33]: s2*10
Out[33]:
a 10
c 30
d 60
e -10
b 20
g 80
dtype: int64
DataFrame
DataFrame几要素：index、columns、values等
通过传递一个list对象来创建一个Series，pandas会默认创建整形索引
In [34]: s = pd.Series([1,3,5,np.nan,6,8])
In [35]: s
Out[35]:
0 1.0
1 3.0
2 5.0
3 NaN
4 6.0
5 8.0
dtype: float64
通过传递一个numpy array,时间索引以及列标签创建DataFrame
In [48]: dates = pd.date_range("20170101",periods = 6)
In [49]: df = pd.DataFrame(np.random.randn(6,4),index=dates,columns=list("ABCD"))
In [50]: df
Out[50]:
A B C D
2017-01-01 0.198724 1.455237 -1.165803 -0.474382
2017-01-02 0.622154 -0.280253 -0.492515 0.002470
2017-01-03 1.764839 -1.734531 -0.195002 0.128216
2017-01-04 -0.520130 1.372930 -2.240510 0.362139
2017-01-05 1.530835 0.406480 -1.714226 -0.289591
2017-01-06 0.675166 0.210024 -0.773319 -1.410746
In [51]: dates
Out[51]:
DatetimeIndex(['2017-01-01', '2017-01-02', '2017-01-03', '2017-01-04',
'2017-01-05', '2017-01-06'],
dtype='datetime64[ns]', freq='D')
d1=DataFrame(np.arange(12).reshape((3,4)),index=['a','b','c'],columns=['a1','a2','a3','a4'])
比较常用的有导入等长列表、字典、numpy数组、数据文件等
In [61]: data = {'name':['zxx','lxx','gxx','hxx'],'age':[12,13,14,15],'addr':['JX','JS','BJ','SH']}
字典数据转换为DataFrame，并指定索引
In [62]: d2 = DataFrame(data)
In [63]: d2
Out[63]:
addr age name
0 JX 12 zxx
1 JS 13 lxx
2 BJ 14 gxx
3 SH 15 hxx
In [64]: d3 = DataFrame(data,columns=['name','age','addr'],index=['a','b','c','d'])
In [65]: d3
Out[65]:
name age addr
a zxx 12 JX
b lxx 13 JS
c gxx 14 BJ
d hxx 15 SH
df.dtypes 查看不同列的数据类型
df.Tab键 自动识别所有属性及自定义列
df.head(2) 查看前两行
df.tail(2) 查看尾部两行
df.index 查看索引值
df.columns 查看列名
df.values 查看底层numpy数据
In [56]: df.head(2)
Out[56]:
A B C D
2017-01-01 0.198724 1.455237 -1.165803 -0.474382
2017-01-02 0.622154 -0.280253 -0.492515 0.002470
In [57]: df.tail(2)
Out[57]:
A B C D
2017-01-05 1.530835 0.406480 -1.714226 -0.289591
2017-01-06 0.675166 0.210024 -0.773319 -1.410746
In [58]: df.index
Out[58]:
DatetimeIndex(['2017-01-01', '2017-01-02', '2017-01-03', '2017-01-04',
'2017-01-05', '2017-01-06'],
dtype='datetime64[ns]', freq='D')
In [59]: df.columns
Out[59]: Index([u'A', u'B', u'C', u'D'], dtype='object')
In [60]: df.values
Out[60]:
array([[ 0.19872446, 1.45523672, -1.16580285, -0.47438238],
[ 0.62215406, -0.28025262, -0.49251531, 0.00247041],
[ 1.76483913, -1.73453082, -0.19500168, 0.12821624],
[-0.52013049, 1.37292972, -2.24051045, 0.36213914],
[ 1.53083459, 0.40647992, -1.71422601, -0.28959076],
[ 0.67516588, 0.2100239 , -0.77331882, -1.41074624]])
获取数据
In [71]: d3=DataFrame(data,columns=['name','age','addr'],index=['a','b','c','d'])
In [72]: d3
Out[72]:
name age addr
a zhanghua 40 jianxi
b liuting 45 pudong
c gaofei 50 beijing
d hedong 46 xian
选择列
In [78]: d3[['name','age']]
Out[78]:
name age
a zhanghua 40
b liuting 45
c gaofei 50
d hedong 46
选择行
In [84]: d3['a':'c']
Out[84]:
name age addr
a zhanghua 40 jianxi
b liuting 45 pudong
c gaofei 50 beijing
选择行（利用位置索引）
In [87]: d3[1:3]
Out[87]:
name age addr
b liuting 45 pudong
c gaofei 50 beijing
使用过滤条件
In [90]: d3[d3['age']>40]
Out[90]:
name age addr
b liuting 45 pudong
c gaofei 50 beijing
d hedong 46 xian
obj.ix[indexs,[columns]]可以根据列或索引同时进行过滤
In [91]: d3.ix[['a','c'],['name','age']]
Out[91]:
name age
a zhanghua 40
c gaofei 50
In [93]: d3.ix['a':'c',['name','age']]
Out[93]:
name age
a zhanghua 40
b liuting 45
c gaofei 50
In [94]: d3.ix[0:3,['name','age']]
Out[94]:
name age
a zhanghua 40
b liuting 45
c gaofei 50
修改数据
In [95]: data={'name':['zhanghua','liuting','gaofei','hedong'],'age':[40,45,50,46],'addr':['jianxi','pudong','beijing','xian']}
In [96]: d3=DataFrame(data,columns=['name','age','addr'],index=['a','b','c','d'])
删除行
In [97]: d3.drop('d',axis=0)
Out[97]:
name age addr
a zhanghua 40 jianxi
b liuting 45 pudong
c gaofei 50 beijing
删除列
In [99]: d3.drop('age',axis=1)
Out[99]:
name addr
a zhanghua jianxi
b liuting pudong
c gaofei beijing
d hedong xian
添加一行，注意需要ignore_index=True
In [103]: d4 = d3.append({'name':'wxx','age':38,'addr':'HN'},ignore_index=True)
In [104]: d4
Out[104]:
name age addr
0 zhanghua 40 jianxi
1 liuting 45 pudong
2 gaofei 50 beijing
3 hedong 46 xian
4 wxx 38 HN
In [105]: d4.ix['4','age']=100
In [106]: d4
Out[106]:
name age addr
0 zhanghua 40.0 jianxi
1 liuting 45.0 pudong
2 gaofei 50.0 beijing
3 hedong 46.0 xian
4 wxx 38.0 HN
4 NaN 100.0 NaN
修改索引
In [111]: d3.index=['a','b','c','d']
In [112]: d3
Out[112]:
name age addr
a zhanghua 40 jianxi
b liuting 45 pudong
c gaofei 50 beijing
d hedong 46 xian
汇总统计
常用统计方法
count 统计非NA的数量
describe 统计列的汇总信息
min、max 计算最小值和最大值
sum 求总和
mean 求平均数
var 样本的方差
std 样本的标准差
导入和保存数据
读取csv文件/或者逗号分隔的txt文件
pd.read_csv('wu.csv')
读取HDFS数据
pd.read_hdf('wu.h5',df)
写入为csv文件
df.to_csv('wu.csv')
写入HDF5存储
df.to_hdf('wu.h5','df')
In [15]: inputfile = '/home/hadoop/wujiadong/wu1_stud_score.txt'
In [16]: data = pd.read_csv(inputfile)
In [40]: df = DataFrame(data)
In [41]: df.head(3)
Out[41]:
stud_code sub_code sub_name sub_tech sub_score stat_date
0 2015101000 10101 数学分析 NaN 90 0000-00-00
1 2015101000 10102 高等代数 NaN 88 0000-00-00
2 2015101000 10103 大学物理 NaN 67 0000-00-00
In [42]: df.count()
Out[42]:
stud_code 121
sub_code 121
sub_name 121
sub_tech 0
sub_score 121
stat_date 121
dtype: int64
In [43]: df['sub_score'].describe()
Out[43]:
count 121.000000
mean 78.561983
std 12.338215
min 48.000000
25% 69.000000
50% 80.000000
75% 89.000000
max 98.000000
Name: sub_score, dtype: float64
求学生成绩标准差
In [44]: df['sub_score'].std()
Out[44]: 12.338214729032906
应用函数和映射
In [45]: d1 = DataFrame(np.arange(12).reshape((3,4)),index=['a','b','c'],columns=['a1','a2','a3','a4'])
In [46]: d1
Out[46]:
a1 a2 a3 a4
a 0 1 2 3
b 4 5 6 7
c 8 9 10 11
处理每个元素
In [47]: d1.applymap(lambda x:x+2)
Out[47]:
a1 a2 a3 a4
a 2 3 4 5
b 6 7 8 9
c 10 11 12 13
处理行数据
In [54]: d1.ix[1].map(lambda x:x+2)
Out[54]:
a1 6
a2 7
a3 8
a4 9
Name: b, dtype: int64
列级处理
In [62]: d1.apply(lambda x:x.max()-x.min(),axis=0)
Out[62]:
a1 8
a2 8
a3 8
a4 8
dtype: int64
参考资料
1）10 Minutes to pandas：
http://pandas.pydata.org/pandas-docs/stable/10min.html
2）十分钟搞定pandas：
http://www.cnblogs.com/chaosimple/p/4153083.html
3）Pandas使用：
https://github.com/qiwsir/StarterLearningPython/blob/master/311.md
4）python cookbook：
http://pandas.pydata.org/pandas-docs/stable/cookbook.html#cookbook
