Change User Agent with curl to Get URL Source Code as Different OS & Browser

Using curl we can retrieve the HTML & CSS source code of any specified URL and even the http header info, but some sites serve completely different content or HTML to different OS and browser versions, this is done by detecting their user agent. Because of this, we can spoof the user agent of another browser version and operating system, and this allows web developers to quickly get access to those alternate variations of a sites source code. For the purposes here, we’ll achieve this from the command line by using curl.

The basic syntax for spoofing user agent with the curl command is as follows:

curl -A "UserAgentString" http://url.com

Of course you’ll replace UserAgentString with a legitimate user agent string that matches the browser you wish to mimic.

Let’s look at a few examples with various user agent strings.

One of the most common situations of different source HTML and CSS are for websites with stripped down mobile versions, you could retrieve iPhone-specific source code with:

curl -A "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5" http://www.apple.com

Some sites do this with other browsers too. This would be Chrome 12 in Mac OS X 10.6.8:

curl -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.112 Safari/534.30" http://microsoft.com

Here’s another that spoofs the Mac App Store and Mac OS X 10.6.7 as a user agent and is useful for querying the App Store from a script (more about that on TUAW):

curl -silent -A "iMacAppStore/1.0.1 (Macintosh; U; Intel Mac OS X 10.6.7; en) AppleWebKit/533.20.25" http://ax.search.itunes.apple.com/

Yet another spoofs Windows XP with Firefox 3:

curl -A "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3" http://yahoo.com

You can find user agent strings all over the web, just be sure to include them in quotes if you want to retrieve a sites source as that user agent. If you want to read more about user agents, Wikipedia has a good entry on the topic.

Note: This is intentionally done through the command line and aimed at those who like to work from the Terminal, but there are easy ways to do this through standard graphical applications and web browsers like Safari, Chrome, and Firefox. Safari is probably the simplest, because you can set different user agents directly from the Developer menu:

This screenshot was taken from an article about getting Facebook Video Chat calls to work in OS X Lion, which is accomplished by changing the browser user agent to a version that Facebook has deemed compatible.