I figured that I never really understood certificates the way I want to, but knowing about them is of paramount importance to any software developer. This article is my effort to do a deep dive into the lifecycle of the certificate. I am not sorry for the length of the article as I have tried to move slowly and explain every nuance of the process. Still interested?

Grab a coffee and let’s start.

The end idea is to create a mechanism by which a server and a client can establish a secure connection to talk to each other.

Now it is not easy as it might sound. The process of establishing a secured channel between a server and a client involves creating an encrypted connection between the server and the client so that anyone sitting in between the server and the client may not know what is going on, thanks to encryption unless they have a way to parse the data.

Types of encryption

There are 2 types of encryption mechanisms used today.

Symmetric

Here the encryption and decryption are done using the same key. Even though it sounds like an easy way to achieve encryption, the real pain is the distribution of the encryption key from the one who is encrypting to the one who needs to decrypt the data.

This is a simple technique and the encryption can be done quickly.

We will not discuss this type of encryption in this article.

Asymmetric

Here the encryption is done using a key and decryption is done using another key, hence asymmetric.

The 2 keys are mathematically related. The encryption takes more time compared to the earlier process.

SSL encryption security works on asymmetric encryption, which is also called public-key cryptography/encryption. Asymmetric encryption works on two cryptographic keys, i.e., the public key and the private key. It is the basis of PKI (Public Key Infrastructure)

The private key is private to the one who is wanting to send data out to the public. The public key is shared with the world. Data encrypted using private key can only be decrypted using the public key and vice versa. This is how the distribution issue is avoided in the asymmetric encryption mechanisms.

The life cycle of SSL certificates

We will take a case study and discuss each step of how to use SSL and why that step is important.

Step 1: Generate public-private key

Let’s say Tom has a website and he wants to implement SSL to secure his website. To do so, he will first generate 2 keys.

• Private key: This key stays with the website and is never exposed. The server will use this key to encrypt data and send it forward to the clients.

openssl genrsa -out private.pem 2048

• Public key: This key is exposed to the outer world and is used by the clients of Tom’s website.

openssl rsa -in private.pem -outform PEM -pubout -out public.pem

Well, great. Tom is happy as he does not need to share his private key and the client can use his website’s public key to decrypt the data the website sends.

Now here is another problem. How do the users of Tom’s website know that it is Tom’s website? What is stopping wrongdoers from creating another website just like Tom’s and hosting it? The users may mistakenly go to the other website and provide their credentials and so on.

Here is where the Certificate Authorities come in. They are authorities who represent Tom’s website to the outer world so that users can trust it is Tom’s website they are visiting. The Certificate Authorities are well-known organizations across the internet.

Step 2. Generate a CSR

To get Tom’s certificate certified, we need to ask an authority (There are so many available like Godaddy, Verisign, Norton, etc) by creating a CSR request (Certificate Signing Request). The CSR file needs to be generated by the user, so here Tom uses his private key to generate a CSR.

This is how CSR can be generated.

openssl req -new -key private.pem -out certificate.csr

Here there will be several questions that we need to answer, like:

You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:New Jersey
Locality Name (eg, city) []:Maple Shade
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Stayingfoolish.org
Organizational Unit Name (eg, section) []:Tech
Common Name (e.g. server FQDN or YOUR name) []:stayingfoolish.org
Email Address []:...

After all these questions are answered, we will get a .csr file (Here certificate.csr).

The certificate.csr file is not the certificate that Tom can use on his website. It is the digital form( A standard way) of submitting a certificate signing request.

The most common format for CSR is the PKCS #10 specification; another is the Signed Public Key and Challenge SPKAC format generated by some web browsers.

This CSR file needs to be sent to a Certificate Authority of choice, who after verifying a lot of stuff will provide a certificate, which can be used by Tom on his website.

Step 3. Apply for a CSR(Certificate Signing Request)

After getting the CSR from Tom the certificate authority makes several tasks. Here is a link to view the roles of a certificate authority. Here are some major ones:

1. Receive, authenticate, and process certificate revocation requests.
2. Identify, and authenticate subscribers, here is Tom’s certificate and its authenticity.
3. Obtain the public key from the subscriber, here is the public key from Tom’s certificate.
4. Verify subscriber has the asymmetric private key corresponding to the public key.

After all, verification is done the Certificate Authority will finally provide the certificate(Typically in .crt format) which Tom can use for this website. The Certificate Authority charges Tom for their service and gives a date till the certificate is valid. After that Tom again needs to get a new certificate.

A few concepts

So what is a certificate?

A certificate is nothing but a simple document containing the public key and some information about the organization that is creating the certificate.

We can see the certificate in raw for using the OpenSSL library. Assuming the OpenSSL library is installed in the host machine, run this:

susamn@vulcan ~ openssl s_client -showcerts -connect medium.com:443
CONNECTED(00000003)
depth=2 C = IE, O = Baltimore, OU = CyberTrust, CN = Baltimore CyberTrust Root
verify return:1
depth=1 C = US, O = "Cloudflare, Inc.", CN = Cloudflare Inc ECC CA-3
verify return:1
depth=0 C = US, ST = California, L = San Francisco, O = "Cloudflare, Inc.", CN = medium.com
verify return:1
---
Certificate chain
 0 s:C = US, ST = California, L = San Francisco, O = "Cloudflare, Inc.", CN = medium.com
   i:C = US, O = "Cloudflare, Inc.", CN = Cloudflare Inc ECC CA-3
-----BEGIN CERTIFICATE-----
MIIFETCCBLigAwIBAgIQDcdN2TIZa+h9cFHogUeAEDAKBggqhkjOPQQDAjBKMQsw
CQYDVQQGEwJVUzEZMBcGA1UEChMQQ2xvdWRmbGFyZSwgSW5jLjEgMB4GA1UEAxMX
Q2xvdWRmbGFyZSBJbmMgRUNDIENBLTMwHhcNMjEwNTA2MDAwMDAwWhcNMjEwODAz
MjM1OTU5WjBqMQswCQYDVQQGEwJVUzETMBEGA1UECBMKQ2FsaWZvcm5pYTEWMBQG
A1UEBxMNU2FuIEZyYW5jaXNjbzEZMBcGA1UEChMQQ2xvdWRmbGFyZSwgSW5jLjET
MBEGA1UEAxMKbWVkaXVtLmNvbTBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABG8a
guZXgCdiy54ryMtywI8sybQNP40rrwEXqa25bVfxHhREIB8yAORIqRobEShes0rc
RTLvopxKY/NYCCCIdemjggNeMIIDWjAfBgNVHSMEGDAWgBSlzjfq67B1DpRniLRF
+tkkEIeWHzAdBgNVHQ4EFgQUGQyl1nRsBEHtV/ePtxr/UHtdCv8wIwYDVR0RBBww
GoIMKi5tZWRpdW0uY29tggptZWRpdW0uY29tMA4GA1UdDwEB/wQEAwIHgDAdBgNV
HSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwewYDVR0fBHQwcjA3oDWgM4YxaHR0
cDovL2NybDMuZGlnaWNlcnQuY29tL0Nsb3VkZmxhcmVJbmNFQ0NDQS0zLmNybDA3
oDWgM4YxaHR0cDovL2NybDQuZGlnaWNlcnQuY29tL0Nsb3VkZmxhcmVJbmNFQ0ND
QS0zLmNybDA+BgNVHSAENzA1MDMGBmeBDAECAjApMCcGCCsGAQUFBwIBFhtodHRw
Oi8vd3d3LmRpZ2ljZXJ0LmNvbS9DUFMwdgYIKwYBBQUHAQEEajBoMCQGCCsGAQUF
BzABhhhodHRwOi8vb2NzcC5kaWdpY2VydC5jb20wQAYIKwYBBQUHMAKGNGh0dHA6
Ly9jYWNlcnRzLmRpZ2ljZXJ0LmNvbS9DbG91ZGZsYXJlSW5jRUNDQ0EtMy5jcnQw
DAYDVR0TAQH/BAIwADCCAX8GCisGAQQB1nkCBAIEggFvBIIBawFpAHYA9lyUL9F3
MCIUVBgIMJRWjuNNExkzv98MLyALzE7xZOMAAAF5QFDoKAAABAMARzBFAiABzzxD
hWcvpGJFkO/vNr3f3lfQnL6emfIp2a703DAD9AIhAJACvhIkMj2pBwQZHz0bB1YD
kgasVxZdiRdgzPxE65YbAHcAXNxDkv7mq0VEsV6a1FbmEDf71fpH3KFzlLJe5vbH
DsoAAAF5QFDi5QAABAMASDBGAiEAgSW1T6TMhHGPcol5TjqPmaRoPSNcG3cA94/h
3/8Sv6oCIQDuwGUai6NRrq1s2CJFjoGAUFDDohAyUF/KWKKhZ4q+1gB2AESUZS6w
7s6vxEAH2Kj+KMDa5oK+2MsxtT/TM5a1toGoAAABeUBQ6HMAAAQDAEcwRQIgaTSM
OXNx4k52vjHqLZ5FYZQI8dNlcBG6etBeHpK2EFsCIQCMFalEWlozhhFqJ2IRlpZq
YioXeZMI+fD8WG7bCUCIbTAKBggqhkjOPQQDAgNHADBEAiBFoNUxOU6eQpbGOOz9
P2zqcDfCzlNV+4cUS0MjRn96nAIgfAy10yYeu+uwSQJw7ZrfGaNpEQUyTg+mTXyg
7v1MekU=
-----END CERTIFICATE-----
... more contents

The certificate is between — — -BEGIN CERTIFICATE — — — and — — -END CERTIFICATE — . It is encoded data.

Certificate Encodings

At its core an x.509(The standard for defining public key certificates) certificate that is encoded and/or digitally signed according to RFC 5280.

There is a lot of confusion about what DER, PEM, CRT, and CER are and many have incorrectly said that they are all interchangeable. While in certain cases some can be interchanged the best practice is to identify how the certificate is encoded and then label it correctly.

Encodings (also used as extensions)

• .DER = The DER extension is used for binary DER encoded certificates. These files may also bear the CER or the CRT extension. Proper English usage would be “I have a DER encoded certificate” not “I have a DER certificate”.
• .***PEM ***= The PEM extension is used for different types of X.509v3 files which contain ASCII (Base64) armored data prefixed with a “ — — BEGIN …” line.

Certificate Manipulations

View PEM encoded certificate

openssl x509 -in cert.pem -text -noout
openssl x509 -in cert.cer -text -noout
openssl x509 -in cert.crt -text -noout

View DER encoded Certificate

openssl x509 -in certificate.der -inform der -text -noout

Transform

#PEM to DER
openssl x509 -in cert.crt -outform der -out cert.der#DER to PEM
openssl x509 -in cert.crt -inform der -outform pem -out cert.pem

What is a signature?

A signature of a certificate is a piece of data inside the certificate which is the only thing that is used to validate the integrity and validity of the certificate.

We discussed that in Asymmetric Encryption one key encrypts and another key decrypt. This helped us with the issue of the distribution of the key.

Now here is another problem.

When Tom sends a CSR(The public key inside it) to the Certificate Authority, how can the Certificate Authority validate it is Tom? Anyone can send a CSR. Think for a while.

The first check the CSR needs to do before doing all other checks is that it is Tom who sent this CSR right? and the only way to do it is to know the user(Here it is Tom) who is sending the CSR owns the private key of the public key in the CSR.

To help in the above-said verification process, the signature is used. In anywhere:

Signature = encrypted with private key ( The checksum of the certificate itself in the mentioned algorithm)

The verifier needs to decrypt the signature by the provided public key in the certificate to get the checksum. Then the verifier will calculate the checksum of the cert itself. These 2 checksums must match to ascertain the integrity and validity of the user sending the certificate.

Every CSR or certificate will have a signature section. Here is the signature of the CSR we generated in Step 2.

The certificate authority on getting the CSR will do the same to validate that it is actually, Tom who is sending the CSR.

The verification of a certificate is done in the same way everywhere using the signature and it does not depend on the certificate. After verifying all the verifiable, the certificate authority then is ready to sign Tom’s certificate.

What is a self-signed certificate?

Self-signed certificates are those which are signed by the user itself. In our example, if Tom signs his certificate, then it is a self-signed certificate. With the CSR file in our hand we can very easily generate a self-signed certificate:

openssl x509 -req -sha256 -days 365 -in certificate.csr -signkey private.pem -out server.crt

Self-signed certificates are only used in tests and development as there is no need to go to a Certificate Authority and pay them money. It is never used in a production environment as they are very easy targets of the MITM(Man-In-The-Middle) attacks.

Step 3: The certificate authority signs the certificate

Now here is the question, what does it mean to sign a certificate?

Signing simply means that the certificate authority has verified the certificate, (here Tom’s certificate) and it can now vouch for Tom’s certificate’s authenticity. After signing the CA provides a certificate to the user, here Tom.

To sign, the certificate authority

1. Calculates the checksum of Tom’s certificate(in Pic 1) using some algorithm.
2. Encrypts the checksum using its (The certificate authority’s) private key
3. Adds the encrypted checksum in Tom’s certificate as a signature.
4. Adds its own information/metadata(The certificate authority’s) in the certificate

After this process, the certificate is created and the Certificate Authority hands the certificate to Tom and it looks like this.

If we expand the CloudFlare Inc ECC CA-3 we will see its information. It has its public key, validity(From and to) and so much more.

The Certificate Authority(CloudFlare Inc ECC CA-3) also has its private key which key keep secret and uses the same to create the signature for Tom’s certificate.

Notice, as this certificate authority(CloudFlare Inc ECC CA-3) certifies medium.com’s certificate that is why the subject name of Pic 5 matches with the Pic 2 issuer name.

But hang on a second. Why does the Certificate Authority(CloudFlare Inc ECC CA-3) have a signature also(Highlighted in blue in Pic 5)? Is someone also certifying Cloudflare CA?

Some more concepts…

Root Certificate and Intermediate Certificate

As it turns out, I lied. The real pic 4 is this:

There is another Certificate Authority on top of CloudFlare(Which certifies medium.com). This is the Root Certificate Authority. Here in this case the Root Certificate authority is Baltimore Cybertrust Root(Root CAs generally have Root at the end of their name to make them obvious)

Root certificate also works the same way. Just like CloudFlare Certificate Authority signed medium.com’s certificate using its private key, the Root Certificate Authority(Here Baltimore Cybertrust Root) uses its private key to sign CloudFlare’s certificate, Pic 5.

If we open the Certificate of the Root(Baltimore Cybertrust Root) we will see this:

Note, here the issuer and the certifier are the same, so the Root Certificate Authority does not need any other entity to certify themselves. The self sign their certificate(Uses own private key to sign) to make the signature(Highlighted purple)

This is another way to identify whether it is a ROOT or Intermediate certificate. See the issue and subject section, if they are the same then it is a ROOT otherwise Intermediate.

Root Certificates like this are the center of PKI. They are sacred and at the center of the trust model.

Every device comes with a so-called root store. A root store is a collection of pre-downloaded root certificates, along with their public keys, that reside on the device. Devices use either the root store built into its operating system or a third-party root store via an application like a web browser. The root stores are part of root programs, like the ones from Microsoft, Apple, Google, and Mozilla.

The browsers will automatically trust any certificate signed with its private key. A trusted root certificate is a special kind of X.509 digital certificate that can be used to issue other certificates like CloudFlare here.

Root certificates sign intermediate certificates and intermediate certificates sign users(medium.com) certificates.

While the intermediate certificates have a validity of 1–3 years, root certificates have a long lifespan. Notice the Baltimore Cybertrust Root is valid for 25 years.

Where are root certificates in my system?

In Windows, go to -> Manage user certificates

In MAC go to -> KeyChain Access

In Ubuntu, go to -> /etc/ssl/certs

Why do we need intermediate CAs?

Now here is a valid question. Why do we need intermediate CA like Cloudflare? Why not certify via the Root CA always? Surely that can do, right?

Yes, that surely can be done, but remember Root certificates are invaluable and few. If they have to certify all the businesses in the world, they will be overwhelmed in no time. Root CAs have to do a lot of scrutinies and put a lot of effort to keep their private keys unobtainable.

That is the reason Root Certificates delegate the responsibility to other CAs like CloudFlare to certify common business certificates.

This type of responsibility is called certificate chaining. In RFC5280 this is called “Certification Path”. This process can be repeated several times, where an intermediate root signs another intermediate and finally signs an end entity certificate

Here is the certificate chain of the medium website as of June 15, 2021.

How do application codes use certificates?

Java

In Java we have 2 files:

keystore.jks : This file contains the server certificate including the private key of the server(like Tom’s website server). The Keystore file is protected with a password, initially changeit. This file can be externally provided in the JVM arguments using -Djavax.net.ssl.trustStore= <Path>

cacerts ($JAVA_HOME/lib/security/cacerts): This file is the one containing all the root certificates and their public keys.

Python

Using the most popular package, requests:

Python looks inside these files:

python3/dist-packages/requests/cacert.pem

python2.7/site-packages/requests/cacert.pem

We can use the REQUESTS_CA_BUNDLE variable to point to a new location.

For a different certificate store to be used, we can specify this:

requests.get(url, verify="/path/to/cert")

Golang
According to__https://golang.org/src/crypto/x509/root_linux.go__

Here are the locations which are used by Golang.

"/etc/ssl/certs/ca-certificates.crt" // Debian/Ubuntu/Gentoo etc.
"/etc/pki/tls/certs/ca-bundle.crt"   // Fedora/RHEL 6
"/etc/ssl/ca-bundle.pem"             // OpenSUSE
"/etc/pki/tls/cacert.pem"            // OpenELEC
"/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem" // CentOS/RHEL 7
"/etc/ssl/cert.pem"                 // Alpine Linux

What is the SSL handshake process?

All SSL certificate work on TCP protocol to send data over the internet. SSL follows a handshake process that sets up a secure connection between the client and the server. The process is as follows:

• After the TCP connection is established, the client starts a workflow to achieve a secured channel. First, the client sends the SSL version, Cipher suites, and compression method it supports, to the server.
• The server sends the highest and most recent SSL version back which is supported by both the server and the client. The server also selects the compression method and cipher suite from the client’s response.
• After this, the server sends the certificate (with its public key) to the client.
• The client validates the certificate using the certificate chain from its certificate store.
• After the validation of the certificate, the client generates a pre-master secret, encrypts it with the server public key, and sends it.
• The server gets the encrypted pre-master key and decrypts using its private key.
• Both parties agree on a cipher suite and generate the symmetric session key.
• Finally, both client and server exchange encrypted messages establishing a secured channel.

Finally, we achieve a secured connection with the server. Phew !!! it was a journey. :)

Also Published Here