Using cURL to Download Remote Files from the Command Line

Feb 13, 2014 - 9 Comments

Download with curl The powerful curl command line tool can be used to download files from just about any remote server. Longtime command line users know this can be useful for a wide variety of situations, but to keep things simple, many will find that downloading a file with curl can often be a quicker alternative to using a web browser or FTP client from the GUI side of Mac OS X (or linux). This is is helpful for local situations, but there’s particular value if you’re in a situation where you need to download something to a remote Mac when connected through SSH.

For the purposes of this walkthrough, we’ll focus primarily on downloading files from the two commonly encountered HTTP and SFTP protocols, though it should be noted that cURL supports many more protocols. Though curl is easy to use, having some knowledge of the command line is recommended.

Download Exact Match Files with curl -O

Using the uppercase -O flag with curl downloads the file from the remote server while maintaining the exact file name, the basic syntax for this is the following:

curl -O [url]

This means if the specified URL file is named “sample.zip” it will download with the filename “sample.zip”, and if the file is named something enormous and complicated like “LongExampleFileNameForOSXDaily-v-1-3-51-revision-515b12-readme.txt” on the remote server, it will save with that exact name on the local machine. Longer file names are often better handled with the -o flag rather than -O, which we’ll address shortly.

Regular readers may recall that we used the curl -O command when explaining how to extract the actual audio content from an m3u streaming file.

Beginning any download with curl shows the percent transferred, time spent downloading and time remaining, and the transfer speed.

Downloading a file with curl

The screenshot offers a better representation than the pasted example below, but it looks something like this:

% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 10505 100 10505 0 0 79741 0 --:--:-- --:--:-- --:--:-- 142k

With transfer speed showing you could redirect the output of curl to /dev/null and use it to test internet connection speed, but the wget command has an easier to read and follow transfer bar thus wget is better suited for that task.

Saving a Remote File with a Different Name with curl -o

Using a lowercase -o flag will allow you to specify a different file name for the downloaded file than how it is named on the remote server. This can be helpful to reduce lengthly file names or just label something so that it’s easier to find on your own. The general syntax would be:

curl -o [shortname] [url]

For example, if you want to save an iOS IPSW file that you found listed on Apple servers without having the lengthy full name, you could use the following:

curl -o iPhone5C-704.ipsw http://appldnld.apple.com/iOS7/031-1828.20131114.P3wE4/iPhone5,3_7.0.4_11B554a_Restore.ipsw

This will download the “iPhone5,3_7.0.4_11B554a_Restore.ipsw” file but named shorter as the more meaningful “iPhone5C-704.ipsw”.

If you’d rather not save the file to the present working directory, specify a path as part of the file name like so:

curl -o ~/Desktop/localexample.dmg http://url-to-file/example.dmg

Downloading Multiple Files Concurrently with curl

cURL can easily download multiple files at the same time, all you need to do is specify more than one URL like so:

curl -O [URL 1] [URL 2] [URL 3]

For files with different names, or hosted on different servers, or within different directory paths, use the complete URL, for example:

curl -O http://ftp.gnu.org/gnu/Licenses/fdl-1.1.txt http://ftp.gnu.org/gnu/Licenses/lgpl-2.1.txt http://ftp.gnu.org/gnu/GNUinfo/Audio/index.txt

On the other hand, if the file names to be downloaded are using incremental naming, you can use brackets to specify a download range, like so:

curl -O http://ftp.gnu.org/gnu/Licenses/fd1-1[1-3].txt

This would grab the files fdl-1.1.txt, fd1-1.2.txt, and fd1-1.3.txt all at the same time without having to specify each unique URL. Of course this only works if the files are in the same directory together and at the same domain.

Authenticating with curl

You can also pass authentication with cURL by using the -u flag:

curl -u user:pass -O ftp://remote_url/file-to-download.zip

Keep in mind that bash history will store the password in plain text when using -u with a username and password specified, thus this is not recommended for most situations. You can get around that by placing a space in front of ‘curl’. If you don’t use the spacebar to prefix the command, you’ll probably want to empty the command history afterwards to be safe.

Supported curl Protocols & Usage Beyond HTTP & FTP

As mentioned earlier, cURL usage goes far beyond HTTP and FTP, as the curl manual page entry mentions additional protocols in the description:

curl is a tool to transfer data from or to a server, using one of the
supported protocols (DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP,
IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS,
TELNET and TFTP).

Additionally, you’ll find that curl can also be used for PUT and POST requests, cookies, proxies, tunnels, resume downloads, and even grabbing HTTP header information or changing a user agent (effectively spoofing) without the need to use a dedicated web browser.

Like most command line utilities, you can learn much more about curl by summoning the appropriate man page with the ‘man curl’ command.

.

Related articles:

Posted by: Paul Horowitz in Command Line, Mac OS, Tips & Tricks

9 Comments

» Comments RSS Feed

  1. Goldmind says:

    Publish npm install -g npm

  2. Adam says:

    I am using https://oshi.at to upload files using Curl:
    curl -T myfile.jpg oshi.at which gives a management and download links right after upload is finished

  3. Citizen J says:

    Alas, all works well until one wants to download a file with chinese characters in the filename. Then, giving the actual chinese.mp3 doesn’t work, it wants it url encoded. Go ahead, try explaining the use of the –data-urlencode because the manual is OBTUSE in explaining it and gives NO examples.

    • curly says:

      Escape the Chinese special characters in the file name and you can download it like any other file with curl.

      You an also try putting quotations around the file name like curl -O “filetodownload.pdf”

  4. SAM NG says:

    How do you download an XML or HTML file (as oppose to viewing it)? I need to pull some content that is stored as a XML file, and i get 404 File not found.. because it looks like the command is interpreting the contents of the file.

    Sam

  5. toan ho says:

    Awesome. This works great

  6. Jos says:

    Hey,

    I am trying to get the data from a PV solar power converter via WIFI using a password secured Seversolar-IP 160.190.0.1 connection. BUT I only get the html file and that file is converted in 7_zip format. No data like name, number or energy converted is present. When I copy/paste the content directly from Firefox by mouseclick then all data is present.

    Is there a command for cURL to download not only the html lines but the data variabels too?

  7. tygger says:

    Question,

    Can I use curl in a script in order to push our software to remote machines?

    I know I can use the -s switch to install, but that is local.

  8. TK says:

    Standard advice: use curl for single files, use wget for multiple files

Leave a Reply

 

Shop on Amazon.com and help support OSXDaily!

Subscribe to OSXDaily

Subscribe to RSS Subscribe to Twitter Feed Follow on Facebook Subscribe to eMail Updates

Tips & Tricks

News

iPhone / iPad

Mac

Troubleshooting

Shop on Amazon to help support this site