a blog about programming and startups by Brian Brunner

Replacing the Standard Library - Python Requests

31 Mar 2014

Sometimes, a language’s standard library is so poorly exposed, undocumented or outdated that it warrants using an external package to replace it.

In Python, making HTTP requests is shockingly cumbersome. A basic GET request is easy enough.

import urllib2
response = urllib2.urlopen('')
html =

But once you start getting into more complicated things, you enter a world that grows increasingly more painful. For example, a post request looks like this.

import urllib
import urllib2

url = ''
values = {'username' : 'bbrunner',
          'fullname' : 'Brian Brunner',
          'requests' : True }

data = urllib.urlencode(values)
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)
the_page =

And if you need to get json back, that’s another import you need to remember. If you want to add in auth or coookies or read compressed data or do anything else that is totally fair game for interacting with HTTP services, urllib/urllib2 pretty much sucks. I’m not going to go into any more examples, because we’d be here all day trying to sort out issues.

For a modern language, Python is not the best at interacting with the web. Thankfully, there’s a solution that many of you probably know about. requests is a Python package that, as the name suggests, makes it easy to write HTTP requests. The above examples become single lines of code, and many, many things that are a nightmare with the standard library are made easy.

A GET request with json

jsonResponse = requests.get('').json()

A POST request

data = {'username' : 'bbrunner',
        'fullname' : 'Brian Brunner',
        'requests' : True }
jsonResponse ='', data=data).json()

For the sake of brevity, I’m going to stop there, but the documentation for requests is rather good, so I suggest you check it out.

So, in short, unless you have an abnormal, specific need to do so, don’t use urllib/urllib2. Just use requests.

subscribe to my mailing list