How to Easily Mirror an Entire Web Site Locally

Mar 19, 2009 - 15 Comments

terminal It’s very easy to mirror an entire web site on your local machine thanks to the terminal command wget, this tutorial will show you how to do it via the command line. wget is available for Mac OS X, Linux, Unix, and many other popular operating system platforms, thus this is universally applicable assuming you have wget anyway.

To get started mirroring a web site with wget, launch the Terminal app and type the following command, replacing guimp.com (a tiny sample website) with the URL you desire to mirror locally.

How to Mirror a Web Site Locally with wget

wget and the -m flag will download and mirror an entire web site that is referenced. The syntax would be as follows, replacing the URL as desired:

wget -m http://www.guimp.com/

This will download the entire website on your local drive in a directory named the websites URL… note that this isn’t a particularly effective way to truly backup a website and it’s functionality, it’s simply mirroring it locally.

You can do this with just about any website, just fill in the appropriate site URL using the following syntax format:

wget -m [url]

Obviously this isn’t going to mirror ajax, databases, queries, dynamic content, or scripts, it’s only going to be a static HTML version of a site. To completely mirror a dynamic site locally, you would need access to the raw files through SFTP or otherwise, to which you could just download the entire site contents and then run in a mirrored version on a local machine through Apache, nginx, MAMP, or whatever other web server you’re prefer.

Note: various readers pointed out that wget is not installed in Mac OS X by default and that you’ll need to install it yourself. You can get wget for Mac OS X by building it yourself from source (recommended for most), or install it through Homebrew, or MacPorts. MacPorts or Homebrew are a very useful tools allowing you to easily install open source software packages and is highly recommended for moderate to advanced Mac OS X users, or those interested in the command line.

.

Related articles:

Posted by: David Mendez in Command Line, Mac OS, Tips & Tricks

15 Comments

» Comments RSS Feed

  1. […] you want to go a step further, you can also quickly mirror a website locally using wget or backup through FTP with […]

  2. […] You should note that this is not an effective way to backup a website, and it’s functionality. [Via] […]

  3. […] used wget to mirror a website locally before, it’s a powerful tool and is a worthwhile install if you’re an advanced user and […]

  4. This will download the entire site on your local drive in a directory named the websites URL… note that this isn’t a particularly effective way to truly backup a website and it’s functionality, it’s simply mirroring it locally.

  5. Joe says:

    httrack is much better for mirroring websites.
    Also available through MacPorts.
    Personally, I’ve downloaded the entire internet. It is *much*
    faster to browse that way.

  6. hi,

    I always used wget with -rxk options, result similar :)

  7. Great tip, I will try it, thanks!

  8. Jerid Hill says:

    Thanks! Adding products to a shopping cart takes a little too long and I’d rather do it locally!

  9. saintberry says:

    Jerid, if you want to sync local and remote directories use rsync. It is actually included in 10.5 by default. If you are game with the command line man rsync, otherwise check the internet for tutorials.

  10. Jerid Hill says:

    Couple of questions:

    1. If I mirror my site locally, can I make changes locally and sync it using terminal?

    2. What if my site is using a database on a separate server?

  11. james says:

    Of course first you have to istall macports

  12. CB says:

    Got Deep Vacuum… GUI to wget

  13. CB says:

    When I type sudo port install wget, I get

    sudo: port: command not found

  14. james says:

    sudo port install wget

  15. saintberry says:

    I didn’t think wget was included in 10.5 by default. Well it wasn’t for me at least, OS X is missing a tone of great *nix command line tools.

    If the above command gives you a command not found error you will need to download and install a wget binary or compile it from source. The latter can be done easily with a package manager like macports.

Leave a Reply

 

Shop on Amazon.com and help support OSXDaily!

Subscribe to OSXDaily

Subscribe to RSS Subscribe to Twitter Feed Follow on Facebook Subscribe to eMail Updates

Tips & Tricks

News

iPhone / iPad

Mac

Troubleshooting

Shop on Amazon to help support this site