How to Use cURL with Proxy?

What is cURL?

cURL (pronounced curl) is a command line tool that facilitates the transfer of data to and from servers. With a few lines of commands in a terminal, you can complete the exchange of large chunks of data from any server to another.

Like any program command, cURL must follow a strict syntax to function properly. The syntax takes the following form.
cURL [seçenekler] [URL]

The ‘options’ part of the syntax specifies certain command parameters that indicate the desired result of the command. There are about 380 of these parameters or flags for cURL commands. The popular one is ‘-o’. It causes the content of a page to be delivered in a file.

cURL also requires a specific Uniform Resource Identifier (URI) type to identify the transport destination or source, hence the URL part of the syntax. This special URI format is the URL (Uniform Resource Locator). cURL does not support network protocols without URLs. (Protocols are a set of instructions on how a server should request and/or receive data on a network.

In a previous article, we gave a detailed explanation of what protocols are and the difference between two popular protocols: SOCKS and HTTP. You should check for further explanations).

cURL uses a transmission software called LibcURL. This is a library containing a list of multiple protocols that cURL supports. It is robust and versatile and is perhaps the most widely used network transport library.

libcURL is a traditional and portable network software. It works the same way on a wide range of operating systems, including Linux, Windows and macOS. It is installed by default on these operating systems. However, if not, you can download and install the software package from the CURL website.

Which Protocols does cURL Support?

cURL supports protocols that allow “data transfer” in both directions. Not all protocols support the use of URLs and such protocols are not compatible with cURL. Furthermore, cURL has a predetermined format for how each protocol it supports can be deployed.

HTTP is the default protocol for cURL usage. However, the command utility also works perfectly with many other network protocols, including the following, arranged in alphabetical order:

• Glossary Network Protocol [DICT]
• FILE
• File Transfer Protocol (FTP)
• GOPHER
• GOPHERS
• IMAP
• IMAPS
• LDAP
• LDAPS
• MQTT
• POP3
• POP3S
• RTMP
• RTSP
• SCP
• SFTP
• SMB
• SMBS
• SMTPS
• TELNET
• TFTP

Why use cURL?

cURL is widely used due to its versatility, simplicity and effectiveness. The main purpose of CURLs is to download and upload entire web pages using any of the supported protocols or formats. It is also used for various functions such as user authentication, HTTP POST, file transfer, Metalink support and proxy support for FTP uploads and SSL connections.

This widespread use of CURLs is due to the following features of the cURL:

• It provides detailed information about what has been delivered or sent and this is very useful for troubleshooting.
• Multiple network protocol support
• When the specified protocol does not work, it can automatically try to work with another protocol.
• Allows you to specify parts of a URL or multiple URLs by using URL parts in parentheses
• Compatible with different user environments (operating systems)

Using cURL with HTTP/HTTPS proxy

This website is particularly useful for testing proxies because the output of this page is the source IP address. If you are using a correct proxy, the page will return a different IP address than your machine, i.e. the IP address of the proxy.

There is more than one way to run curl with the proxy command. The next section discusses sending proxy details as a command line argument. We will also explore how to integrate HTTPS proxies.

NOTE All command-line options or switches are case sensitive. For example, -f instructs curl to fail silently, while -F specifies a form to submit.

Commands to set a proxy in cURL

Open the terminal and type the following command and press Enter:

curl --help

The output will be a large list of options. One of these will look like this:

-x, --proxy [protocol://]host[:port]  

Note that x is small and case sensitive. Proxy details can be provided using the -x or -proxy switch. They both mean the same thing. Proxy and curl commands have the same bot:

curl -x "http://user:pwd@127.0.0.1:1234" "http://httpbin.org/ip"

and

curl --proxy "http://user:pwd@127.0.0.1:1234" "http://httpbin.org/ip"

NOTE If there are SSL certificate errors, add -k (note the small k) to the curl command. This will allow insecure server connections when using SSL.

curl --proxy "http://user:pwd@127.0.0.1:1234" "http://httpbin.org/ip" -k

You may have noticed that both the proxy url and the destination url are enclosed in double quotes. This is a recommended practice to handle special characters in the url.

Another interesting thing to note here is that the default proxy protocol is http. Thus, following two commands will do exactly the same:

curl –proxy “http://user:pwd@127.0.0.1:1234” “http://httpbin.org/ip”

curl –proxy “user:pwd@127.0.0.1:1234” “http://httpbin.org/ip”

When you run these commands, any cURL command you run is automatically executed through the proxy server.

Using cURL with SOCKS proxy

curl -x "socks5://user:pwd@127.0.0.1:1234" "http://httpbin.org/ip"

Similarly, depending on the socks version, socks4://, socks4a://, socks5:// or socks5h:// can be used.

Alternatively, the curl socks proxy can be set using the -socks5 switch instead of -x. You can follow the same command, but use a different switch: username and password can be sent using the -proxy-user switch.

cURL is a very powerful tool for automation and is arguably the best command line interface in terms of proxy support. Finally, because libcurl works so well with php, many web applications use it for web scraping projects, making it a must-have for any web scraper.