[zz]Make Yahoo! Web Service REST calls with Python
http://developer.yahoo.com/python/python-rest.html
Python provides a number of modules for performing HTTP requests.
This HOWTO describes how to perform GET and POST requests using the urllib
and urllib2
modules from the Python standard library.
Simple GET requests
The simplest way of retrieving data from a URL uses the urllib.urlopen
function:
import urllib
url = 'http://developer.yahoo.com/'
u = urllib.urlopen(url)
# u is a file-like object
data = u.read()
This can be condensed in to a single line:
data = urllib.urlopen(url).read()
This is fine for quickly retrieving some data, but does not give
you the ability to easily distinguish between normal page retrievals and
404 or 500 errors. The urllib2
module offers a similar
function that throws an exception should one of these HTTP errors be
encountered:
import urllib2
try:
data = urllib2.urlopen(url).read()
except urllib2.HTTPError, e:
print "HTTP error: %d" % e.code
except urllib2.URLError, e:
print "Network error: %s" % e.reason.args[1]
An HTTPError
is thrown if the server returns a status
code other than 200 (OK) or an HTTP redirect. Redirects are followed
automatically.
Simple POST requests
Some APIs require you to make POST requests. The urlopen
method provided by both urllib
and urllib2
supports POST requests through an optional second argument:
import urllib
url = 'http://search.yahooapis.com/ContentAnalysisService/V1/termExtraction'
appid = 'YahooDemo'
context = '''
Italian sculptors and painters of the renaissance favored
the Virgin Mary for inspiration
'''
query = 'madonna'
params = urllib.urlencode({
'appid': appid,
'context': context,
'query': query
})
data = urllib.urlopen(url, params).read()
Authenticated requests
The del.icio.us API
requires you to make authenticated requests, passing your del.icio.us
username and password using HTTP authentication. urllib2
provides a mechanism for doing this.
import urllib2
delicious_user = 'Your del.icio.us username'
delicious_pass = 'Your del.icio.us password'
password_manager = urllib2.HTTPPasswordMgrWithDefaultRealm()
password_manager.add_password(
None, 'https://api.del.icio.us/', delicious_user, delicious_pass
)
auth_handler = urllib2.HTTPBasicAuthHandler(password_manager)
opener = urllib2.build_opener(auth_handler)
urllib2.install_opener(opener)
xml = urllib2.urlopen('https://api.del.icio.us/v1/posts/recent').read()
Using httplib2
If the urllib2
code for making authenticated requests
strikes you as over-complicated, you may be interested in Joe Gregorio's
httplib2 library.
It is not included in the Python standard library so you will need to
download and install it separately.
All requests are passed through an instance of the Http
class. You can optionally enable an HTTP cache by passing a directory
to that constructor; this is not necessary for the Yahoo! APIs as they
do not directly support this kind of caching.
A simple GET request looks like this:
import httplib2
http = httplib2.Http()
response, content = http.request('http://developer.yahoo.com/')
At this point, response is a dictionary-like object containing the
headers from the server response, while content is the body of the
response. You can check for error codes using response.status
.
POST requests are similar to urllib
and urllib2
,
but require you to explicitly set the Content-Type
header
of your request to application/x-www-form-urlencoded
.
import httplib2
http = httplib2.Http()
url = 'http://search.yahooapis.com/ContentAnalysisService/V1/termExtraction'
appid = 'YahooDemo'
context = '''
Italian sculptors and painters of the renaissance favored the Virgin Mary for inspiration
'''
query = 'madonna'
import urllib
params = urllib.urlencode({
'appid': appid,
'context': context,
'query': query
})
response, content = http.request(url, 'POST', params,
headers={'Content-type': 'application/x-www-form-urlencoded'}
)
Finally, an authenticated request can be set up using the add_credentials
method:
import httplib2
http = httplib2.Http()
delicious_user = 'Your del.icio.us username'
delicious_pass = 'Your del.icio.us password'
http.add_credentials(delicious_user, delicious_pass)
response, content = http.request('https://api.del.icio.us/v1/posts/recent')
Further reading
Related information on the web
- urllib and urllib2 documentation
- httplib documentation
- httplib2 documentation
- Dive Into Python Chapter 11: HTTP Web Services