Please help!

That said, I think it’s a good idea to have people use the command line, if only to get a feeling that under the hood there is something else than icons and windows.

The benefits of using Linux as an alternative is that it forces you to gain useful technical knowledge that also applies to the Mac. It’s like driving stick-shift.

Whereas… using Windows is a waste of time, I find.

Yikes, have the script kiddies taken over or something?!

Had to copy hundreds of thousands of small files from a Linux server to a Mac over the LAN. A normal cp operation is slow because it never gets a chance to increase the network bandwidth as it resets for each individual file. Watching a performance graph would show it peaking and falling for every individual file. So the solution would be to tar gzip the files into one huge file and then copy that file. Trouble was, there was not enough free disk space to handle the huge file.

Netcat to the rescue! Netcat is a very powerful tool it can open a raw network port and pass data through that port and it can listen for a raw data stream on the other end. So the trick is to tar gzip the files and pipe the output (no filename) into nc directing nc to a network port number of choice. Then on the receiving computer you listen with nc and pipe it’s data to untar and ungzip the files on the receiving end.

To give you an idea of how effective something like that is; the attempted copy operation over an SMB share ran for 5 hours and the Finder was still just calculating the number of files which at that point was well over 500,000 files! i.e. nothing had actually started to be copied! I tried using the cp command from terminal and it was only about a third of the way through the copy process having run overnight. I aborted the copy process and setup nc to listen on a particular port and piped it’s output to a tar gz command that would untar and ungzip the files to the required path. Then on the sending system I created a tar gzip command and piped it’s output to nc directing to the IP address of the receiving computer and the same network port. The copy operation finished in 2 hours and looking at the network activity the Gigabit Ethernet adapter was almost completely saturated at 100%. The performance graph rapidly peaked and stayed completely peaked the entire time the copy process ran.

So essentially, the sending system executed tar and gzip operations storing the data in RAM while it pushed that data over netcat to a waiting netcat listening on a pre-determined network port where the files were pushed to a tar gzip command that decompressed the files. Since there was a large continuous amount of data pumping through the network port, the amount of bandwidth increased exponentially until it maxed out the Ethernet card. This allowed it to go as fast as it possibly could have run.

The mind blowing part? These tools are built in to OS X and every Unix system but they have been around for more than 20 years. The whole Unix concept is small tools that do one thing very well and you connect the tools to one another via the pipe command.

The funny part? I’ve been around the block for years and Windows cannot do this at all. I don’t even know of a single third party application that comes close to the way I was able to use tar gzip and netcat to speed up copying an enormous amount of data. But I can boot a Windows box with a Linux USB flash drive and do it!

Let’s have a few more articles about interesting things you can do with the Unix pipe and some little known command line programs!

Disclaimer: The sending system was Linux running a newer GNU version of netcat while OS X includes the older FreeBSD version. I had to use homebrew to install the GNU version of netcat on OS X to get it to work with the Linux system. The command parameters vary a bit between the two netcat programs and I was unable to get the built-in OS X version of netcat to play nice with the Linux GNU version.

Homebrew is a way to install Linux tools on OS X it’s really simple and easy. http://mxcl.github.com/homebrew/ – for example OS X includes the curl command but maybe you want the wget command which is similar but maybe you are more familiar with wget. Homebrew installs the add-on tools into a location that won’t interfere with the normal OS X tools and if their names are the same, it will use the Homebrew one first. Homebrew will not mess up your Mac, but it might confuse someone else trying to work on your Mac.

e.g.
tar xzvf myTarFile.tar.gz

also you may find tarred/gzip files called .tgz
The file extension doesn’t actually matter on these, but your readers should know this lil’ factoid.