本部分为pandas的输入输出对象。
先上表,其中read_csv和read_table最常用。
这些函数的选项可以划分几类:
类型推断(type inference):不需要指定数据类型,可以自动识别。
In [286]: df = pd.read_csv('../_data/source/ch06/ex1.csv')In [287]: dfOut[287]: a b c d message0 1 2 3 4 hello1 5 6 7 8 world2 9 10 11 12 fooIn [289]: pd.read_table('../_data/source/ch06/ex1.csv',sep=',')Out[289]: a b c d message0 1 2 3 4 hello1 5 6 7 8 world2 9 10 11 12 foo
In [290]: cat ../_data/source/ch06/ex2.csv1,2,3,4,hello5,6,7,8,world9,10,11,12,fooIn [291]: pd.read_csv('../_data/source/ch06/ex2.csv',names=['a','b','c','d','message'])Out[291]: a b c d message0 1 2 3 4 hello1 5 6 7 8 world2 9 10 11 12 fooIn [292]: names=['a','b','c','d','message']In [293]: pd.read_csv('../_data/source/ch06/ex2.csv',names=names, index_col='message')Out[293]: a b c dmessage hello 1 2 3 4world 5 6 7 8foo 9 10 11 12
read_csv时可以使用skiprows跳过文件的某些行。
In [294]: !cat ../_data/source/ch06/ex4.csv# hey!a,b,c,d,message# just wanted to make things more difficult for you# who reads CSV files with computers, anyway?1,2,3,4,hello5,6,7,8,world9,10,11,12,fooIn [295]: pd.read_csv('../_data/source/ch06/ex4.csv',skiprows=[0,2,3])Out[295]: a b c d message0 1 2 3 4 hello1 5 6 7 8 world2 9 10 11 12 foo
In [296]: !cat ../_data/source/ch06/ex5.csvsomething,a,b,c,d,messageone,1,2,3,4,NAtwo,5,6,,8,worldthree,9,10,11,12,fooIn [297]: pd.read_csv('../_data/source/ch06/ex5.csv')Out[297]: something a b c d message0 one 1 2 3.0 4 NaN1 two 5 6 NaN 8 world2 three 9 10 11.0 12 fooIn [298]: result = pd.read_csv('../_data/source/ch06/ex5.csv')In [299]: result.isnull()Out[299]: something a b c d message0 False False False False False True1 False False False True False False2 False False False False False False
read_csv/read_table选项:
当文件太大时候,我们只想读取其中一部分。
In [303]: result = pd.read_csv('../_data/source/ch06/ex6.csv',nrows=5)In [304]: resultOut[304]: one two three four key0 0.467976 -0.038649 -0.295344 -1.824726 L1 -0.358893 1.404453 0.704965 -0.200638 B2 -0.501840 0.659254 -0.421691 -0.057688 G3 0.204886 1.074134 1.388361 -0.982404 R4 0.354628 -0.133116 0.283763 -0.837063 QIn [305]: chunker = pd.read_csv('../_data/source/ch06/ex6.csv',chunksize=1000)In [306]: chunkerOut[306]: <pandas.io.parsers.TextFileReader at 0x1147e5850>
In [323]: data = pd.read_csv('ex5.csv')In [324]: dataOut[324]: something a b c d message0 one 1 2 3.0 4 NaN1 two 5 6 NaN 8 world2 three 9 10 11.0 12 fooIn [325]: data.to_csv('out.csv')In [326]: cat out.csv,something,a,b,c,d,message0,one,1,2,3.0,4,1,two,5,6,,8,world2,three,9,10,11.0,12,fooIn [327]: data.to_csv('out.csv',sep="|")In [328]: cat out.csv|something|a|b|c|d|message0|one|1|2|3.0|4|1|two|5|6||8|world2|three|9|10|11.0|12|fooIn [329]: import sysIn [330]: data.to_csv(sys.stdout,na_rep='NULL'),something,a,b,c,d,message0,one,1,2,3.0,4,NULL1,two,5,6,NULL,8,world2,three,9,10,11.0,12,fooIn [331]: data.to_csv(sys.stdout,index=False,header=False)one,1,2,3.0,4,two,5,6,,8,worldthree,9,10,11.0,12,fooIn [332]: data.to_csv(sys.stdout,index=False,columns=list('abc'))a,b,c1,2,3.05,6,9,10,11.0
Series也有类似方法
In [333]: dates = pd.date_range('2/1/2017',periods=7)In [334]: s1 = Series(np.arange(7),index=dates)In [335]: s1.to_csv(sys.stdout)2017-02-01,02017-02-02,12017-02-03,22017-02-04,32017-02-05,42017-02-06,52017-02-07,6
CSV文件的形式有很多,只需要定义csv.Dialect的一个子类接口。
In [340]: import csvIn [341]: class my_dialect(csv.Dialect): ...: lineterminator = "\n" ...: delimiter = ";" ...: ...: In [342]: with open("mydata.csv",'w') as f: ...: writer = csv.writer(f,dialect = my_dialect) ...: writer.writerow(('one','two','three')) ...: writer.writerow(('1','2','3'))
JSON数据格式十分灵活
In [344]: import jsonIn [350]: obj = {'name':'alex','age':18,'job':'enginer'}In [353]: a = json.dumps(obj)In [354]: aOut[354]: '{"job": "enginer", "age": 18, "name": "alex"}'In [355]: json.loads(a)Out[355]: {u'age': 18, u'job': u'enginer', u'name': u'alex'}
lxml是python处理xml的库。
待续。。
实现二进制数据存储最方便的方式就是使用python标准库pickle序列化。
In [356]: frame = pd.read_csv('ex1.csv')In [357]: frameOut[357]: a b c d message0 1 2 3 4 hello1 5 6 7 8 world2 9 10 11 12 fooIn [358]: frame.save('frame_pickle') # 没这个方法了。---------------------------------------------------------------------------AttributeError Traceback (most recent call last)<ipython-input-358-f936768749d3> in <module>()----> 1 frame.save('frame_pickle')/Users/yangfeilong/anaconda/lib/python2.7/site-packages/pandas/core/generic.pyc in __getattr__(self, name) 2670 if name in self._info_axis: 2671 return self[name]-> 2672 return object.__getattribute__(self, name) 2673 2674 def __setattr__(self, name, value):AttributeError: 'DataFrame' object has no attribute 'save'
HDF5流行的工业库,支持多种压缩器的即时压缩,更有效的存储数据。对于那些很大,但是不能存放在内存中的数据,
hdf5是一个不错的选择。
python有两个库处理HDF5接口:
pandas有一个HDFStore的类,它通过PyTables存储pandas对象。
In [359]: store = pd.HDFStore('myhdf.h5')In [360]: frameOut[360]: a b c d message0 1 2 3 4 hello1 5 6 7 8 world2 9 10 11 12 fooIn [361]: store['obj1'] = frameIn [362]: store['obj1_col'] = frame['a']In [363]: storeOut[363]:<class 'pandas.io.pytables.HDFStore'>File path: myhdf.h5/obj1 frame (shape->[3,5])/obj1_col series (shape->[3]) In [367]: store['obj1']Out[367]: a b c d message0 1 2 3 4 hello1 5 6 7 8 world2 9 10 11 12 foo
In [368]: xls = pd.ExcelFile('data.xls')In [369]: table = xls.parse('Sheet1')
使用request库可以很容易的处理web api,简单的get一个url:
In [373]: import requestsIn [374]: url = "http://www.baidu.com"In [375]: resp = requests.get(url)In [376]: respOut[376]: <Response [200]>
In [388]: import sqlite3In [390]: con = sqlite3.connect(':memory:')In [394]: query = ''' ...: CREATE TABLE test ...: (a VARCHAR(20), ...: b VARCHAR(20), ...: c REAL, ...: d INTEGER);'''In [395]: con.execute(query)Out[395]: <sqlite3.Cursor at 0x117414dc0>In [396]: con.commit()In [399]: data = [('hello','123',1.25,6),('world','Beijing',2.0,3),('foo','shanghai',2.1,20)]In [400]: dataOut[400]:[('hello', '123', 1.25, 6), ('world', 'Beijing', 2.0, 3), ('foo', 'shanghai', 2.1, 20)]In [401]: stmt = "INSERT INTO test VALUES(?,?,?,?)"In [403]: con.executemany(stmt,data)Out[403]: <sqlite3.Cursor at 0x1174148f0>In [404]: con.commit()In [405]: cursor = con.execute('select * from test;')In [406]: cursorOut[406]: <sqlite3.Cursor at 0x1174145e0>In [407]: cursor.fetchall()Out[407]:[(u'hello', u'123', 1.25, 6), (u'world', u'Beijing', 2.0, 3), (u'foo', u'shanghai', 2.1, 20)] In [419]: cursor = con.execute('select * from test') In [420]: df = cursor.fetchall() In [421]: df Out[421]: [(u'hello', u'123', 1.25, 6), (u'world', u'Beijing', 2.0, 3), (u'foo', u'shanghai', 2.1, 20)] In [422]: DataFrame(df,columns=zip(*cursor.description)[0]) Out[422]: a b c d 0 hello 123 1.25 6 1 world Beijing 2.00 3 2 foo shanghai 2.10 20
同样DataFrame也可以:
In [423]: import pandas.io.sql as sqlIn [425]: sql.read_sql('select * from test',con)Out[425]: a b c d0 hello 123 1.25 61 world Beijing 2.00 32 foo shanghai 2.10 20
待续。。。
联系客服