SSL/TLS Demystify

Posted on | Edited on | In

This is all about securing servers with SSL/TLS certificates from udemy course SSL complete guide.

The quality of SSL varies, you have SSL setup doesn’t mean your site is good secured. The HTTPS may not work correctly, sub-optimal, you can test it here: https://www.ssllabs.com/index.html

If you click the lock icon at left of the website address, it will show you if the connection is secured or not, it’s certificates, cookies and so on. further click the certificate icon, you will see root CA, intermediate CA and certificate.

Install wireshark on Mac, go to download the stable version dmg package and double click to install.

You can use Chrome inspect -> Network to see traffic or use wireshark

For example, from Network select one item, check HEADER information you can get IP address of remote server, or just use host, nslookup commands to get IP address.

Interesting, from Network I see the the chrome browser sometime uses IPV6 address talk to server, for example facebook and some other sites. see this question

大概总结一下: openssl 目前有command 一次性生成(或分开生成 private key -> CSR(certificate signing request) -> self-signed certificate) private key 和 (self-signed) certificate, 一般用pem 的格式。这2个东西是放在web server上的，也可以把private key 和 certificate 合并到一个pem 文件中。还要注意的是，这里没有用到public key，但public key可以从private key中生成(其实里面已经包含了public key的信息)。此外，web server给 client的certificate 就只是certificate，不会有private key尽管它们可能在一个pem file中。

在client部分，对于self-signed certificate, 需要设置操作系统trust it，但如果是用的let’s encrypt（或其他well-known CA) 则很可能已经支持了。联想一下TLS handshake, 得到web server的certificate 后，会逐层验证到root CA(会下载所有相关的certificates), 由于client自身已经携带了well-known root CA的证书了，并且也知道CA的public key，所以就会知道这个root CA是否合法。如果合法，就会进行symmetric key的生成和交换，用于之后的数据传输。

具体命令操作可以参考: <<Set up Secure Docker Registry Container>> 如果要把这个过程自动化，比如自动给网站request安排证书，renew更新，revoke撤销等，需要用到一些automation，比如certbot, or cert-manager in K8s，见下面一章:

然后再来看看crt/key 文件和它们的内容, 一般来说会用到形如tls.crt, tls.key的文件，其中tls.key 一般就是private key的内容了, 比如:

1
2
3
-----BEGIN RSA PRIVATE KEY-----
///xxxxx
-----END RSA PRIVATE KEY-----

tls.crt 就是certificate, 它是一种PEM formatted file, PEM means Privacy Enhanced Mail (concatenated certificate container files), can have different extension like: tls.cert, tls.cer, tls.pem, etc. PEM is a container format that may include just the public certificate, or may include an entire certificate chain including public key, private key, and root certificates. Confusingly, it may also encode a CSR, for example, here it contains 2 block certificates:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
// can also contain private key
-----BEGIN RSA PRIVATE KEY-----
(Your Private Key: your_domain_name.key)
-----END RSA PRIVATE KEY-----
// trust chain
// 一般顺序是从上到下: your domain cert -> intermediate cert -> root cert
-----BEGIN CERTIFICATE-----
(Your Primary SSL certificate: your_domain_name.crt)
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
(Your Intermediate certificate: DigiCertCA.crt)
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
(Your Root certificate: TrustedRoot.crt)
-----END CERTIFICATE-----

阅读以下这些文章: PEM, DER, CRT, and CER: X.509 Encodings and Conversions Creating a .pem File for SSL Certificate Installations

Decode PEM encoded ssl/tls certificate to verify it contains correct information:

1
2
# 如果有多个certificate block in one file, 则解析的是第一个
openssl x509 -in <crt file> -text -noout

Cert-manager

Used in K8s to secure for example ingress, search this post: <<Cert-manager Light Note>>.

Certbot

Get free HTTPs certificates forever, it utilizes Let's Encrypt CA to automatically refresh the certificate for your web site: https://letsencrypt.org/docs/ https://certbot.eff.org/

How it works in detail: https://letsencrypt.org/how-it-works/ This is accomplished by running a certificate management agent on the web server.

Encryption

symmetric encryption, the same key is used by both sides, for example: AES. This algorithm is embedded in SSL with HTTPS protocol. asymmetric encryption, for example: RSA.

Hash

How does hash work to verify data integrity:

1
2
3
data + hash(data)  ---------> data + hash(data)
                                |         |
                                |--hash-->| (compare if they are the same)

Notice that in database the password is hashed, not plain text.

Hash algorithms: MD5, SHA…

  • MD5: 128 bits, echo 123 | md5

for SHA, use shasum command in linux or use inline tool.

  • SHA-1: 160 bits
  • SHA-256: 256 bits
  • SHA-512: 512 bits
1
2
## SHA-256
shasum -a 256 -t test.txt
  • HMAC: can be used with md5 or sha. In cryptography, an HMAC (sometimes expanded as either keyed-hash message authentication code or hash-based message authentication code) is a specific type of message authentication code (MAC) involving a cryptographic hash function and a secret cryptographic key. It may be used to simultaneously verify both the data integrity and the authenticity of a message, as with any MAC. Any cryptographic hash function, such as SHA-256 or SHA-3, may be used in the calculation of an HMAC.

Asymmetric Keys

Encryption

1
2
3
data -------------> code =========>  code -------------> data  (owner side)
      public key                          private key
      encryption                          decryption

Usually(but not necessarily), the keys are interchangeable, in the sense that if key A encrypts a message, then B can decrypt it, and if key B encrypts a message, then key A can decrypt it. While common, this property is not essential to asymmetric encryption.

Signature

1
2
3
4
5
6
            data                          |----------> hash value
             | (hash)     ==========>     |  compare      /|\
             |                            |                |
         private key encrypt              |           public key decrypte
            \|/                           | (hash)         |
data   +   encrypted hash              data    +   encrypted hash

Signing ensures the data is sent by the owner of private key and not has been modified inbetween.

What is the difference of digest and signature? https://www.ibm.com/support/knowledgecenter/SSFKSJ_9.2.0/com.ibm.mq.sec.doc/q009810_.htm A message digest is a fixed size numeric representation of the contents of a message, computed by a hash function. A message digest can be encrypted by the sender’s private key, forming a digital signature.

More readings: How does a public key verify a signature?

PKI

public key infrastructure is a set of roles, policies, hardware, software and procedures needed to create, manage, distribute, use, store and revoke digital certificates and manage public-key encryption. The purpose of a PKI is to facilitate the secure electronic transfer of information for a range of network activities such as e-commerce, internet banking and confidential email.

Certificate

A file with some contents:

  1. certificate owner
  2. certificate issuer
  3. signature (RSA created, made by issuer)
  4. public key (from owner, we then use this public key to HTTPS)

Self-signed certificate: issued and signed by the owner. The basic rule is we trust the CA (the issuer) so on the certificate owner.

Why we need intermediary CAs?

There are not so much public root CAs because of problem of trust. Actually anybody can create own root CA but nobody will trust it. That’s why there is limited set of global root CAs that are trusted worldwide by operating systems and browsers. You can view list of such global CAs with their root certificates in any browser or OS.

Such root CAs have certificates with long period of validity and their main responsibility is simple create “source of trust”. That’s why they don’t issue certificates to end users to avoid additional work and minimize risk that their private keys will be compromised. Intermediate CAs certificates don’t necessarily need to be in the list of trusted certificates in the OS or browser. They need simply be issued by trusted root CA.

Chain of Trust

Let’s see openssl command, generate RSA private key and public key: 这里没有谈到self-signed certificate,见后面

1
2
3
4
5
6
7
8
9
10
11
## check help for sub-command genrsa
openssl genrsa -h

## generate private.pem file private key with aes256 encryption method
## will ask you input pass phrase 
## 实际上private.pem 包含了public key 的信息了！
openssl genrsa -aes256 -out private.pem

## generate public key from above private key
## will ask you pass phrase from private key
openssl rsa -in private.pem -outform PEM -pubout -out public.pem

Root CAs in OS

How does web browser trust the root CAs and certificates? The OS ships a list of trusted certificates, in Mac search Keychain Access.

In Linux, see this link: On Red Hat/Centos, It includes all trusted certificate authorities under /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt. Just add your new certificate authority file(s) to the directory /etc/pki/ca-trust/source/anchors, then run /bin/update-ca-trust to update the certificate authority file.

Verify chain of trust

1
2
3
4
5
6
7
           root CA         |     intermediate CA       |        end user
----------------------------------------------------------------------------------------
  self-singed certificate  |     signed by root CA     |   signed by intermediate CA
----------------------------------------------------------------------------------------
   signature: encrpted     |    signature: encrpted    |       signature: encrpted
 by private key of root CA | by private key of root CA |  by private key of intermediate
                           |                           |  CA

CSR: certificate signing request (the root CA receive CSR from intermediate CA, the signature of intermediate CA is signed by root CA, the root CA also provide issuer info for intermediate CA. Similarly, end user is signed by intermediate CA)

Web server (the end user) sends you its certificate and all intermediate certificates. Then on your side start the verification process from end user certificates back to top intermediate certificate then root certificate.

这里如何验证的: To verify a certificate, a browser will obtain a sequence of certificates, each one having signed the next certificate in the sequence, connecting the signing CA’s root to the server’s certificate.

There is a online tool to check the certificates chain: https://www.geocerts.com/ssl-checker.

Create Self-signed Certificate

Openssl Essential, several ways to generate certificates

Before creating certificate, you need CSR, and before CSR, you need first generate asymmetric keys. (Becuase certificate needs to include signature from upstream and also your public key)

Choose common name (CN) according to the main domain where certificate will be used. (for example, in secure docker registry, CN is the registry address), actually CN is deprecated and Alternative Name(SAN) is used instead.

What is /etc/ssl/certs directory? Actually this is a softlink to /etc/pki/tls/certs.

Generate self-signed certificate, see this post

1
2
3
4
5
# this is CN certificate, if you want to have SAN(Subject Alternative Name), see below
openssl req \
        -newkey rsa:4096 -nodes -x509 -sha256\
        -keyout key.pem -out cert.pem -days 365 \
        -subj "/C=US/ST=CA/L=San Jose/O=Company Name/OU=Org/CN=<domain>"
  • -nodes: short for No DES, if you don’t want to protect your private key with a passphrase.
  • Add -subj '/CN=localhost' to suppress questions about the contents of the certificate (replace localhost with your desired domain).
  • For anyone else using this in automation, here’s all of the common parameters for the subject: -subj "/C=US/ST=CA/L=San Jose/O=Company Name/OU=Org/CN=<domain>"
  • Remember to use -sha256 to generate SHA-256-based certificate.

注意有时遇到的cert 文件中可能包含多个CERTIFICATE 块，其中有intermediate CA. 在kubernetes 中构造tls secret的时候，直接使用即可。

To generate SAN certificate, see this post. 主要是注意如何构造 san.cnf 文件给 openssl 使用. SAN主要是为了一证多用，注意这个和wildcard tls certificate不一样, wildcard tls certificate 主要是为了包容subdomains.

SSL/TLS and HTTPS

Both are cryptographic protocols used in HTTPS:

  • SSL: Secure Socket Layer
  • TLS: Transport Layer Security.

Difference between SSL vs TLS: TLS is an update and secure version of SSL. https://www.globalsign.com/en/blog/ssl-vs-tls-difference/ It’s important to note that certificates are not dependent on protocols. Sometimes you hear SSL/TLS certificate, it may be more accurate to call them Certificates for use with SSL and TLS, since the protocols are determined by your server configuration, not the certificates themselves.

Go to ssllab, you can check which version of TLS the web server use, input the web server address and scan, then click IP icon.

Why RSA is not used in data encryption?

  1. too slow.
  2. bi-directional data encryption requires RSA key pairs on both sides. We encrypt data use symmetic key after setup secure connection.

Why need rsa key pairs on both sides? Because the key is interchangeable, everyone has public key can decrypt data encrypted by private key!

Establish TLS Session

You can see verbose from curl command

1
2
3
# -I: fetch header only
# -v: verbose
curl -Iv https://www.google.com
  1. establish tcp session
  2. establish tls session (negotiate protocol)
  3. web server sends its certifiate (intermediate and others) to browser
  4. browser generate symmetic key secured by public key from server and send to server. Or use Diffie–Hellman key exchange.

Let’s see wireshark for wikipedia connection: The top 3 are TCP handshakes, then you see TLS client hello, then TLS server hello.

In client hello, there are lots of information to negoitate with server, here you see some supported version of TLS, also cipher suites.

In server hello, you see the the server has selected one of cipher suites. The TLS_ECDHE.._SHA256 means it uses Diffie–Hellman key exchange and sha256 as hash.

Other Readings

这个网站上的资源可以好好看看: https://www.cloudflare.com/learning/

–>> How Does SSL Work? The main use case for SSL/TLS is securing communications between a client and a server, but it can also secure email, VoIP, and other communications over unsecured networks.

–>> TLS handshakes During a TLS handshake, the two communicating sides exchange messages to acknowledge each other, verify each other, establish the encryption algorithms they will use, and agree on session keys.

SSL handshakes are now called TLS handshakes, although the “SSL” name is still in wide use.

A TLS handshake also happens whenever any other communications use HTTPS, including API calls and DNS over HTTPS queries.

这里讲到了2种不同的TLS handshake过程，一个是public-private key参与，另一个是Diffie–Hellman handshake. 对于第一种方式，提到了client从server cerficiate extract public key: https://stackoverflow.com/questions/17143606/how-to-save-public-key-from-a-certificate-in-pem-format

–>> How does proxy handle TLS handshakes (还记得envoy吗?) HTTPS knows how to tunnel the TLS handshake even through the proxy. 也就说TLS/SSL through proxy就是通过HTTP CONNECT tunnel去实现的, 特别是看一下第二个回答, comments:

So, the proxy is not MITM’ing the HTTPS connection, by replacing the server’s certificate with its own - it’s simply passing the HTTPS connection straight through between the client and the server. Is that right?

Normally, when HTTPS is done through a proxy, this is done with the CONNECT mechanism: the client talks to the proxy and asks it to provide a bidirectional tunnel for bytes with the target system. In that case, the certificate that the client sees is really from the server, not from the proxy. In that situation, the proxy is kept on the outside of the SSL/TLS session – it can see that some SSL/TLS is taking place, but it has no access to the encryption keys.

Diffie–Hellman

Diffie–Hellman uses one-way function, for example, mod operation. See, here a, b are priviate keys on both side, g, p are public key. A, B are mod result, K is the final result that both side can get use to encrypt the data.

Elliptic-curve cryptography is used in Diffie–Hellman.

Custom Domain

Purchase custom domain and use free hosting to setup our website.

0%