Sign in to follow this  

urllib2 - Website loads fine in browser but not in Python

Recommended Posts

TheComet    3897

I'm trying to download a page from this website:

If I click on the link from within Firefox, it loads fine.


If I try to download the page using urllib2, I keep getting this message:

    <span id="ctl00_mainContentPlaceHolder_lblInvalidRequest"><H2>There was a pr
oblem with your request.</H2>

We are unable to process your request.<br/> Please return to the previous page t
o try again or contact <a href=" Nu
mber: 18&#46;ce969d50&#46;1404818243&#46;20b8e71">Digi-Key Webmaster</a> if you
feel that you have received this message in error. Please reference the followin
g incident number so we may assist you with this error.

Here's the code:

import urllib2
import sys

if __name__ == '__main__':

    # optional proxy
    if len(sys.argv) > 1:
        proxy = {'http': str(sys.argv[1])}
        proxy = urllib2.ProxyHandler(proxy)
        opener = urllib2.build_opener(proxy)

    html = urllib2.urlopen('').read()
    error = html.find('request')
    if not error == -1:
        print html[error-400:error+400]
        print 'success'

Can anyone explain to me why it's doing this and how I can fix it?

Share this post

Link to post
Share on other sites
Key_46    540

In my case the webserver was not accepting the default User Agent from urllib2. Try changing it:


opener.addheaders = [('User-agent', 'Mozilla/5.0')]

Share this post

Link to post
Share on other sites
Key_46    540

Yes, that worked wonderfully. Thanks!


For the future, how were you able to diagnose what the problem was?

Since the request was working on my browser I tried to match the header exactly. I enabled the debug verbose for the opener to see what it was sending:


opener = urllib2.build_opener(urllib2.HTTPHandler(debuglevel=1))

Share this post

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this