How to easily mirror an entire web site locally
Command Line, Mac OS X, Tips & Tricks - March 19th, 2009 - 12 Comments
It’s very easy to mirror an entire web site on your local machine thanks to the terminal command wget, here’s how to do it:
Launch the Terminal and type the following command, replacing guimp.com (a tiny website) with the URL you desire to mirror locally.
wget -m http://www.guimp.com/
This will download the entire site on your local drive in a directory named the websites URL… note that this isn’t a particularly effective way to truly backup a website and it’s functionality, it’s simply mirroring it locally.
Note: various readers pointed out that wget is not installed in Mac OS X by default and that you’ll need to install it using MacPorts, you can check out our past article on installing MacPorts or visit the developer website at macports.org. MacPorts is a very useful tool allowing you to easily install open source software packages and is highly recommended for moderate to advanced OS X users, or those interested in the command line.
I didn’t think wget was included in 10.5 by default. Well it wasn’t for me at least, OS X is missing a tone of great *nix command line tools.
If the above command gives you a command not found error you will need to download and install a wget binary or compile it from source. The latter can be done easily with a package manager like macports.
sudo port install wget
When I type sudo port install wget, I get
sudo: port: command not found
Got Deep Vacuum… GUI to wget
Of course first you have to istall macports
Couple of questions:
1. If I mirror my site locally, can I make changes locally and sync it using terminal?
2. What if my site is using a database on a separate server?
Jerid, if you want to sync local and remote directories use rsync. It is actually included in 10.5 by default. If you are game with the command line man rsync, otherwise check the internet for tutorials.
Thanks! Adding products to a shopping cart takes a little too long and I’d rather do it locally!
Great tip, I will try it, thanks!
hi,
I always used wget with -rxk options, result similar
httrack is much better for mirroring websites.
Also available through MacPorts.
Personally, I’ve downloaded the entire internet. It is *much*
faster to browse that way.
This will download the entire site on your local drive in a directory named the websites URL… note that this isn’t a particularly effective way to truly backup a website and it’s functionality, it’s simply mirroring it locally.