Curl Command
:-
Transferring data from one place to another is one of the main
task done using computers connected to a network. There are so many GUI tools
out there to send and receive data, but when you are working on a console, only
equipped with command line functionality, using curl is inevitable. A less
known fact is that curl can work with a wide range of protocols and can solve
most of your scripting tasks with ease. Before getting into details of curl use
cases and examples, let's see who is behind its development.
What
is curl and what makes it a superb command line tool?
CURL is
simply awesome because of the following reasons...
1.
CURL
is an easy to use command line tool to send and receive files, and it supports
almost all major protocols(DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS,
IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP,
SMTP, SMTPS, TELNET and TFTP) in use.
2.
Can be
used inside your shell scripts with ease
3.
Supports
features like pause and resume of downloads
4.
It has
around 120 command line options for various tasks
5.
It
runs on all major operating systems(More than 40+ Operating systems)
6.
Supports
cookies, forms and SSL
7.
Both
curl command line tool and libcurl library are open source, so they can be used
in any of your programs
8.
It
supports configuration files
9.
Multiple
upload with a single command
10.
Progress
bar, rate limiting, and download time details
11.
ipv6
support.
1. Download a Single File
The following command will get
the content of the URL and display it in the STDOUT (i.e on your terminal).
2.Fetch Multiple Files at a
time
We can
download multiple files in a single shot by specifying the URLs on the command
line.
Syntax:
$ curl -O URL1 -O URL2
3.Follow HTTP Location
Headers with -L option
By default CURL doesn’t follow
the HTTP Location headers. It is also termed as Redirects. When a requested web
page is moved to another place, then an HTTP Location header will be sent as a
Response and it will have where the actual web page is located.
For example, when someone types
google.com in the browser from India, it will be automatically redirected to
‘google.co.in’. This is done based on the HTTP Location header as shown below.
$ curl http://www.google.com
<TITLE>302 Moved</TITLE>
<H1>302 Moved</H1>
The document has moved
<A HREF="http://www.google.co.in/">here</A>
The
above output says that the requested document is moved to
‘http://www.google.co.in/’.
We can
insists curl to follow the redirection using -L option, as shown below. Now it
will download the google.co.in’s html source code.
$ curl -L http://www.google.com
4. Continue/Resume a
Previous Download
Using curl -C option, you can
continue a download which was stopped already for some reason. This will be
helpful when you download large files, and the download got interrupted.
If we say ‘-C -’, then curl will
find from where to start resuming the download. We can also give an offset ‘-C
<offset>’. The given offset bytes will be skipped from the beginning for
the source file.
Start a big download using curl,
and press Ctrl-C to stop it in between the download.
$ curl -O http://www.gnu.org/software/gettext/manual/gettext.html
############## 20.1%
Note: -# is used to display a
progress bar instead of a progress meter.
Now the above download was
stopped at 20.1%. Using “curl -C -”, we can continue the download from where it
left off earlier. Now the download continues from 20.1%.
curl -C - -O http://www.gnu.org/software/gettext/manual/gettext.html
############### 21.1%
5. Limit the Rate of Data
Transfer
You
can ask curl to limit download speed to your desired value. This is very
helpful, when you do not want your curl command to consume much of the
available bandwidth. This can be done using the below method.
$
curl --limit-rate 100k -O http://ftp.gnu.org/pub/gnu/libiconv/libiconv-1.10.tar.gz
6. Download a file only if
it is modified before/after the given time
We can get the files that are
modified after a particular time using -z option in curl. This will work for
both FTP & HTTP.
$ curl -z 21-Dec-11 http://www.example.com/yy.html
The above command will download
the yy.html only if it is modified later than the given date and time
$ curl -z -21-Dec-11 http://www.example.com/yy.html
The above command will download
the yy.html, if it is modified before than the given date and time.
Please refer ‘man curl_getdate’
for the various syntax supported for the date expression
7. Pass HTTP Authentication
in cURL
Sometime, websites will require a
username and password to view the content ( can be done with .htaccess file ).
With the help of -u option, we can pass those credentials from cURL to the web
server as shown below.
$ curl -u username:password URL
Note: By default curl uses Basic
HTTP Authentication. We can specify other authentication method using –ntlm |
–digest.
8. Download Files from FTP
server
cURL can also be used to download
files from FTP servers. If the given FTP path is a directory, by default it
will list the files under the specific directory.
$ curl -u ftpuser:ftppass -O ftp://ftp_server/public_html/xss.php
The above command will download
the xss.php file from the ftp server and save it in the local directory.
$ curl -u ftpuser:ftppass -O ftp://ftp_server/public_html/
Here, the given URL refers to a
directory. So cURL will list all the files and directories under the given URL
If you are new to FTP/sFTP, refer
ftp sftp
tutorial for beginners.
9. List/Download using
Ranges
cURL supports ranges to be given
in the URL. When a range is given, files matching within the range will be
downloaded. It will be helpful to download packages from the FTP mirror sites.
$ curl ftp://ftp.uk.debian.org/debian/pool/main/[a-z]/
The above command will list out
all the packages from a-z ranges in the terminal.
10. Upload Files to FTP
Server
Curl can also be used to upload
files to the FTP server with -T option.
$ curl -u ftpuser:ftppass -T myfile.txt ftp://ftp.testserver.com
The above command will upload the
file named myfile.txt to the FTP server. You can also upload multiple files at
a same time using the range operations.
$ curl -u ftpuser:ftppass -T "{file1,file2}" ftp://ftp.testserver.com
Optionally we can use “.” to get
the input from STDIN and transfer to the remote.
$ curl -u ftpuser:ftppass -T - ftp://ftp.testserver.com/myfile_1.txt
The above command will get the
input from the user from Standard Input and save the contents in the ftp server
under the name ‘myfile_1.txt’.
You can provide one ‘-T’ for each
URL and the pair specifies what to upload where.
11.Deleting files on FTP server
You can also delete files on your
FTP server using curl command, as shown below.
$ curl ftp://example.com/ -X
'DELE myfile.zip' --user username:password
11. More Information using
Verbose and Trace Option
You can get to know what is
happening using the -v option. -v option enable the verbose mode and it will
print the details
curl -v http://google.co.in
The about command will output the
following
* About to connect() to www.google.co.in port 80 (#0)
* Trying 74.125.236.56... connected
* Connected to www.google.co.in (74.125.236.56) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.21.0 (i486-pc-linux-gnu) libcurl/7.21.0 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.15 libssh2/1.2.6
> Host: www.google.co.in
> Accept: */*
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Date: Tue, 10 Apr 2012 11:18:39 GMT
< Expires: -1
< Cache-Control: private, max-age=0
< Content-Type: text/html; charset=ISO-8859-1
< Set-Cookie: PREF=ID=7c497a6b15cc092d:FF=0:TM=1334056719:LM=1334056719:S=UORpBwxFmTRkbXLj; expires=Thu, 10-Apr-2014 11:18:39 GMT; path=/; domain=.google.co.in
.
.
If you need more detailed
information then you can use the –trace option. The trace option will enable a
full trace dump of all incoming/outgoing data to the given file
12. Use Proxy to Download a
File
We can specify cURL to use proxy
to do the specific operation using -x option. We need to specify the host and
port of the proxy.
$ curl -x proxysever.test.com:3128 http://google.co.in
13. Send Mail using SMTP
Protocol
cURL can also be used to send
mail using the SMTP protocol. You should specify the from-address, to-address,
and the mailserver ip-address as shown below.
$ curl --mail-from blah@test.com --mail-rcpt foo@test.com smtp://mailserver.com
Once the above command is
entered, it will wait for the user to provide the data to mail. Once you’ve
composed your message, type . (period) as the last line, which will send the
email immediately.
Subject: Testing
This is a test mail
OR
$curl --url "smtps://smtp.example.com:465" --ssl-reqd --mail-from "user@example.com" --mail-rcpt "friend@example.com" --upload-file mailcontent.txt --user "user@example.com:password" –insecure
In the
above command, replace smtps://smpt.example.com:465 with your SMTP server
and port.
--mail-from:
This field contains the from address that the receiver should see.
--mail-rcpt:
This field contains TO address
--upload-file:
The file provided here should contain your message as a content
--user:
SMTP user@domain:password
--insecure
option used is exactly same as using -k option we saw earlier, to ignore
unknown SSL certificates.
14.Using CURL to send HTTP
POST, PUT, DELETE requests
Apart from the above seen
examples, curl can be used to send different types of HTTP request methods to
your server. This becomes very much handy if you want to configure an
application using REST API. I will show you some example's of sending REST API
requests using CURL.
Sending
POST request using CURL
If you use CURL to send request
to a server without any request method option in the argument, it will by
default use HTTP GET request. Which is a nice behavior. But using -X option in curl, you can
send your required HTTP method to the server. Let's see POST request.
$ curl -X POST -u admin:admin http://example.com/myconfigs/status
-Hcontent-type:application/xml -d @/home/user/file.xml
In the above example, PUT is the
request method, -u is used to
mention credentials to access that specific resource on the server, -H content-type is the type of
content format that we will be sending to the server (can be xml, normal text
etc). -d with
@/home/user/file.xml indicates to send the content of file.xml to the server.
This file will contain the configuration options with correct syntax that the
URL http://example.com/myconfigs/status
will accept.
Sending PUT request is exactly
same as the above shown example of POST. Simply replace POST with PUT in the
above example. If your web server does not accept these methods, you will get a
405 error as reply. HTTP 405 means that the server does not accept the HTTP
method on that specific URL.
You can also get HTTP unsupported Media Type error
as reply, if the server does not accept application/xml format. The HTTP status
code for unsupported media type is 415.
15.Ignore SSL Certificate
Error with CURL
16.Modify User Agent In
HTTP request
When a web browser sends an HTTP
request, it includes its own details in the request. This detail allows the
server to identify the browser details. The HTTP request header that includes
this information is called as User-Agent.
Basically server's can be configured to respond with different page layout for
different user agent's. The user agent information send by the client will be
visible in server logs.
A normal HTTP request send by a
Firefox web browser will have a user agent that looks something like "Mozilla/5.0 (Windows; U;
Windows NT 5.1; de; rv:1.9.2.3)"
Using curl you can modify user
agent value to your required string, as shown below.
|
|
17.Sending Cookies to
Server using CURL
CURL can be used to send
requests to the server using a previously got cookie. You can either use
VALUE=DATA format or give a file name as parameter, so that curl can use
cookie data from the file as input. You can also save the cookies you get
from the server to a file using -c
option as shown below.
$ curl -b
mycookies.txt http://example.com
|
|
|
|
