cURL Use Case Summary

Posted on | Edited on | In

cURL stands for Client URL, it is a command-line tool for getting or sending data including files using URL syntax. Since cURL uses libcurl, it supports a range of common network protocols, currently including HTTP, HTTPS, FTP, FTPS, SCP, SFTP, TFTP, LDAP, DAP, DICT, TELNET, FILE, IMAP, POP3, SMTP and RTSP.

Here is a open-source book: Everything curl

About curl proxy environment variables(see man for more details):

  • http_proxy(only be lower case) to specify http proxy.
  • https_proxy to specify https proxy (curl to proxy over ssl/tls).
  • NO_PROXY or --noproxy '*' to skip proxy use for all hosts, or a list for some hosts.

When using curl download files, make sure it actually downloads it rather than the HTML page, you can check the file type via file command, for example:

1
2
3
curl -Ss <URL> -o xxx.tar.gz
# Check if it is a real tar gz file
file xxx.tar.gz | grep gzip

If the URL is wrong, curl will download the HTML page instead. wget will show 404 error and fail.

Date and Note

1
2
3
4
5
6
7
8
9
10
11
12
13
05/23/2019 download file
06/16/2019 http request verbose
06/22/2020 redirect, PUT/GET/DELETE
09/02/2020 check headers only
09/03/2020 use specific forward proxy
09/06/2020 resume download
09/07/2020 limit rate
09/08/2020 fetch headers only
09/09/2020 proxy tunnel
10/25/2022 name resolve tricks
04/07/2022 retry
04/01/2023 post request with json payload
04/03/2023 trace and see request body

05/23/2019

If the file server needs user name/password (usually will prompt when you open in browser).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# -O: downloads the file and saves it with the same name as in the URL.
# -u: specify "user:password"
# -k: explicitly allows curl to perform 'insecure' SSL connections and transfers.
# -L: allow redirect
# -o: custom file name
# -s: slient all output
# -S: used with slient, output error message
USER="username"
PASSWD="passwd"
USER_PWD="$USER:$PASSWD"
STREAMURL="https://xxx/target.tar.gz"
# Downlaod file with user/password.
curl -k -u ${USER_PWD} -LO ${STREAMURL}

# Download file with custom name /tmp/archive.tgz and slient.
curl -sS <url> -o /tmp/archive.tgz

If you don’t turn off the certificate check(-k), you will get error message and fail:

1
2
3
4
5
6
7
8
9
10
11
12
13
curl: (60) Peer's Certificate has expired.
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

06/16/2019

When I was working on Docker Registry API, I primarily used curl to do the job.

1
2
3
4
5
6
7
8
# -v: Makes the fetching more verbose/talkative. Mostly useful for debugging. A
# line starting with `>` means `header data` sent by curl, `<` means `header
# data` received by curl that is hidden in normal cases, and a line starting
# with `*` means additional info provided by curl.

# -X: (HTTP) Specifies a custom request method to use when communicating with
# the HTTP server.
curl -v -k -XGET http://localhost:5000/v2/_catalog

06/22/2020

1
2
3
4
5
6
7
8
9
10
11
12
# -L: redirect
# -i: include response header info to output
curl -iL http://...

# -v: verbose, to see better error message
curl -v http://...

# -d|--data: value to be put
# -X, --request
curl -X PUT -d '50s' http://localhost:8500/v1/kv/prod/portal/haproxy/timeout-server
curl -X DELETE http://localhost:8500/v1/kv/prod/portal/haproxy/timeout-server
curl -X GET http://localhost:8500/v1/kv/prod/portal/haproxy/timeout-server?pretty

09/02/2020

Add additional headers, only show header info:

1
2
3
4
5
6
7
8
9
# -H,--header: add header
# -L: redirect
# -I: fetch headers only
# -v: verbose, to see better error message
# -s: hide progress bar, slient
# > /dev/null: hide output, show only the -v output
curl --header "Host: chengdol.github.io" \
--header "..." \
-L -Ivs http://185.199.110.153 > /dev/null

09/03/2020

Using HTTP proxy (forward) with proxy authentication, learned from Envoy.

1
2
3
4
5
6
7
8
# -x, --proxy: use the specific forward proxy
curl -v -x <proxy:port> http://www.example.com
# the same as
http_proxy=<proxy:port> curl -v -x http://www.example.com

# -U, --proxy-user: user/password for proxy itself.
# this option overrides the existing proxy environment variable 
curl -v -U <user:password> -x <proxy:port>  http://www.example.com

Default to use basic authentication scheme, Some proxies will require another authentication scheme (and the headers that are returned when you get a 407 response will tell you which).

1
2
## --proxy-anyauth: ask curl to use any method the proxy wants
curl -U <user:password> --proxy-anyauth -x myproxy:80 http://example.com

09/06/2020

Resume the download:

1
2
3
4
curl <url> -o archive
# then break and resume
# -C -: automatrically resume from the break point
curl <url> -C - -o archive

09/07/2020

Download/upload speed limit if you have a limit bandwidth.

1
2
# limit rate as 1m/second, for example: 10k, 1g
curl <url> -O --limit-rate 1m

09/08/2020

Only fetch header information, no body:

1
2
3
# -I,--head: (HTTP FTP FILE) Fetch the headers only!
# notice that -i is to have response header to output, they are different
curl -I <url>

09/09/2020

Non-HTTP protocols over HTTP proxy Most HTTP proxies allow clients to “tunnel through” it to a server on the other side. That’s exactly what’s done every time you use HTTPS through the HTTP proxy.

1
2
3
4
# -p, --proxytunnel: make curl tunnel through the proxy
# used with -x, --proxy options
# here tunnel ftp protocol
curl -p -x http://proxy.example.com:80 ftp://ftp.example.com/file.txt

10/25/2020

The curl with name resolve tricks and another post about it.

This feature is primarily used for HTTP server development and testing the server locally to mimic real world situations.

1
2
3
# First start a simple http server within python virtualenv:
python3 -m http.server
Serving HTTP on :: port 8000 (http://[::]:8000/) ...
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
# Override the default Host header otherwise the Host header value
# will be localhost:8000
# This is not enough for HTTPS server due to SNI, see next --resolve.
curl  -vI -H "Host: www.example.com:80" http://localhost:8000
# The header info:
> HEAD / HTTP/1.1
> Host: www.example.com:80
> User-Agent: curl/7.84.0
> Accept: */*

# www.myfakelist.com does not exist and dns it to 127.0.0.1
# port 8000 must be the same.
curl -vI --resolve www.myfakelist.com:8000:127.0.0.1 http://www.myfakelist.com:8000
# The header info:
> GET / HTTP/1.1
> Host: www.myfakelist.com:8000
> User-Agent: curl/7.84.0
> Accept: */*
# works for HTTPS server as well
curl -kvI --resolve www.myfakelist.com:443:127.0.0.1 https://www.myfakelist.com

# www.myfakelist.com does not exist and map both the name:port
# to localhost:8000 points to the fake python http server.
curl -vI --connect-to www.myfakelist.com:80:localhost:8000 http://www.myfakelist.com
# The header info:
> GET / HTTP/1.1
> Host: www.myfakelist.com
> User-Agent: curl/7.84.0
> Accept: */*
# works for HTTPS server as well
curl -kv --connect-to www.myfakelist.com:443:127.0.0.1:443 https://www.myfakelist.com

04/07/2022

Retry until reaching condition limits:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# --retry: worked on a timeout, an FTP 4xx response code or an HTTP 408, 429,
# 500, 502, 503 or 504 response code.
# --retry-all-errors: retry on any errors. If you want to retry on all response
# codes that indicate HTTP errors (4xx and 5xx) then combine with `-f`, `--fail`.
# --retry-delay: if not set, retries it will double the waiting time until it
# reaches 10 minutes.
# --retry-max-time: given total time limit for retries.
# --retry-connrefused: connect refused retry.
# -m, --max-time: timtout for whole operation.
# --connect-timeout: timeout for curl's connection to take.
curl -kI \
--max-time 3.55 \
--connect-timeout 2.12 \
--retry-all-errors \
--retry 3 \
--retry-delay 1 \
--retry-max-time 10 \
https://no-op

# output
curl: (6) Could not resolve host: no-op
Warning: Problem : timeout. Will retry in 2 seconds. 3 retries left.
curl: (6) Could not resolve host: no-op
Warning: Problem : timeout. Will retry in 2 seconds. 2 retries left.
curl: (6) Could not resolve host: no-op
Warning: Problem : timeout. Will retry in 2 seconds. 1 retries left.
curl: (6) Could not resolve host: no-op

04/01/2023

For example, the POST request is like:

1
2
3
4
5
6
curl -vs -X POST \
--http1.1 \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $TOKEN" \
-d @payload.json \
https://xxx.com/v1/do-something

Regardless of the format in payload.json, curl will always remove the line returns and one-line the JSON content and send it, which is different from Postman who will send payload with line returns (which could be identified as security issue by firewall rules).

04/03/2023

To check the request payload sent, use --trace-ascii:

1
2
3
4
5
6
7
curl -vs -X POST \
--http1.1 \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $TOKEN" \
-d @payload.json \
https://xxx.com/v1/do-something \
--trace-ascii /dev/stdout
0%