econpy.org

Python for Economists

Change Your User-Agent using the Requests Module

Here's a quick snippet you can use to change your User-Agent using the requests module. The point of changing your User-Agent is to make your requests blend into the server logs so that you appear to be just another Firefox browser.

import requests
from lxml.html import fromstring
r = requests.get("http://econpy.pythonanywhere.com/ex/cpu.html")

# print the current user-agent to see the change yourself
print r.config['base_headers']['User-Agent']

# now set and print the new user-agent for comparison
r.config['base_headers']['User-Agent'] = 'Mozilla/5.0 (Windows; U; Windows NT 5.1; it; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11'
print r.config['base_headers']['User-Agent']