A Deeper Look into SSH and X.509 Certificates

As a developer or systems administrator, you're probably used to SSHing into servers with key pairs. What if I told you there is a better and more secure way to manage SSH access? Think SSH certificates (And no, they’re not the same as TLS certificates)

For most people, their knowledge of certificates lies within the confines of the X.509 digital certificates format — also known as TLS/SSL certificates — used by browsers to enable HTTPS. However, the X.509 certificate is not the only digital certificate format that exists; there is also the less-popular SSH certificate format created by OpenSSH.

While X.509 certificates enable identity authentication for browser-to-server communication, SSH certificates enable identity authentication for shell(terminal) to *nix server communications. This article will briefly discuss what SSH certificates are, why you should be using them, and the key distinctions between them and X.509 certificates.

If you’re reading this article, I assume you have some knowledge of X.509 certs work. Else, read Everything you should know about certificates and PKI but are too afraid to ask.

What are SSH Certificates?

When decoded, an OpenSSH certificate is just a data structure containing a public key, name, and optional information like an expiration date and associated permissions. SSH certs are a lot like X.509 certificates, but with fewer properties.

An OpenSSH certificate is the most secure way of granting SSH access. Yet, it’s not widely utilized likely due to developers simply not knowing about it. It also requires more up-front work than a typical SSH public/private key setup, which most people are wary of. However, once implemented, the benefits outweigh the trouble, especially when you have a lot of users, hosts, and keys to manage.

SSH authentication without certificates

Despite the release of SSH certificate-based authentication by OpenSSH in 2010, most people still carry out SSH authentication with passwords or public keys.

Password-based SSH authentication has long been criticized since humans are prone to using unavoidably predictable and weak passwords that are easy to hack. A 2019 Avast survey found that over 80% of average Americans use weak passwords. Additionally, passwords are directly transmitted to the server during an SSH connection initiation, thus making it vulnerable to brute-force attacks.

SSH Public key authentication provides a more secure alternative to passwords, but it is still easily misused. With public key authentication, all clients and servers must have their own public and private key pairs. When a client machine tries to initiate an SSH connection to a server, two things happen:

1. The client will try to confirm that it’s connecting to a trusted server by checking if the server’s public key is in its known_hosts file. If the client does not recognize the server’s public key, it throws an error (as shown) saying that the server’s authenticity can’t be proven but asking if it should connect anyway:

   
   

   The authenticity of host 'xxx' can't be established.
   RSA key fingerprint is SHA256:kfcwi9X8T4nMRw1OM0xDXETGcqjU26/zbM+KqNB6CKw.
   Are you sure you want to continue connecting (yes/no)?
   

   
   

   In this situation, the proper thing would be for the user to send the key fingerprint to the server administrator for verification. Because what if an attacker somehow managed to assign a well-known, trusted IP address to a rogue machine? The reality is that most people choose the easy way out and type yes. When you type “yes”, a connection to the server is established without authentication and the server’s public key is permanently added to the client’s known-hosts file. This presents the first flaw of public-key-based SSH authentication: it encourages trust on first use (TOFU) anti-pattern.

   
   

2. The server will also try to authenticate the user by checking if the user’s public key is present in its authorized_keysfile. Typically, team members send off their public key to an administrator for it to be added or deployed to the authorized_keysfile of every server they need access to. What happens when there are > 10s of servers to which these keys need to be deployed? This presents the second flaw of public-key-based SSH authentication: key management can easily become tedious, and time-wasting, and is not scalable for large organizations.

   
   

Finally, because keys never expire, SSH public key authentication exposes systems to security vulnerabilities. For example, when a developer leaves a company, their public key is supposed to be deleted from all hosts… but we all know in practice, that just isn’t the case. SSH academy recounts that in an analysis of 15000 servers, out of the three million SSH keys that granted access to live production servers, 90% were no longer used. Such compromised key bindings can go unnoticed for a long time, exposing the system to unauthorized access.

Some commercial and open-source services try to help tackle key management crises. The basic idea is that you maintain keys through that service, and when you remove a key, they will remove it from all your servers. However, the danger is that the service could become a single point of failure. If an attacker captures access to that service, they can gain access to all your servers. And in the worst-case scenario, if you lose access to that service, you will also lose access to all of your servers.

SSH authentication with certificates

SSH certificate-based authentication works similarly to X.509 certificates but with a considerably simpler implementation. With SSH certificate-based authentication, clients and hosts mutually authenticate each other, thereby eliminating TOFU.

First, set up an SSH CA server (aka an “online CA”) that will house two SSH CA key pairs: a host CA key pair for signing and authenticating certificates for host machines and a user CA key pair for signing and authenticating certificates for clients. For SSH, people use the term CA (certificate authority) to interchangeably refer both to the signing key pairs and the SSH CA server.

It’s best practice to create two separate CA key pairs to reduce the attack surface. If a private key is compromised, you only need to reissue the certificates for your hosts or users, not both at once.

After you generate the host CA and user CA key pairs, configure all host machines to trust certificates signed by the public key of the user CA. You will also need to configure all clients to trust certificates signed by the public key of the host CA.

To issue certificates to host machines, you’ll have to generate public-private key pairs for all host servers and sign them with the host CA keys. To issue client (user) certificates, You’ll also need to generate key pairs for all clients and sign them with the user CA keys. Ideally, the process of issuing user certificates is completely automated. For example, you could build a flow where users can automatically get signed certificates when they authenticate with your company’s identity provider.

When a client wants to access a host, it’ll present its certificate to the host, and the host will do the same. Host servers will only allow access to clients with certificates trusted by keys, and clients can rest assured that they’re connecting to a trusted host.

Granted, this is a bit of an oversimplification of the actual process. If you are looking for a practical walk-through of the process described above, check out our docs on Working with SSH Certificates.

Apart from eliminating mundane key management tasks and TOFU, SSH certificate-based authentication also offers some features which allow for fine-grained access:

• Certificates expire. Users can be issued short-lived certificates to eliminate the risk of abandoned compromised key bindings. For example, if a developer leaves without having their access revoked, their certificate will expire after a certain period of time, and they will also lose access automatically.
• Certificates are restricted to certain users (principals). This creates an opportunity to enable role-based access control.
• Certificates can contain SSH restrictions, e.g., forbidding PTY allocation or port forwarding.

If you want to read more in-depth on why SSH certificate-based authentication is the most secure and user-friendly way to grant SSH access and how to start using it, read our article If you’re not using SSH certificates, you’re doing SSH wrong.

How are SSH certificates different from X.509 certificates?

Now that we understand what SSH certificates are and how they are used, let’s look at how they differ from X.509 certificates.

Use Cases

X.509 certificates are designed to be ubiquitous. They can be used in SSL/TLS, S/MIME, EAP-TLS, and even in the SSH protocol for web applications, databases, VPNs, WiFi, digital signatures for document signing, and all kinds of other places. SSH certs are mainly used for SSH access & authentication.

Properties

In OpenSSH, SSH certs are much simpler and have only a few features. Unlike traditional X.509 certificates, with numerous options and complex encoding rules, the SSH certificate is just public and private key pairs with additional identity and constraints data.

Here’s what a decoded SSH user certificate and X.509 client certificate look like:

$ step ssh inspect id_ecdsa-cert.pub
id_ecdsa-cert.pub:
        Type: [email protected] user certificate
        Public key: ECDSA-CERT SHA256:O6M6oIjDm5gPm1/aTY619BgC3KSpS4c3aHVWxYh/uGQ
        Signing CA: ECDSA SHA256:EY2EXJGoPv2LA6yEbjH+sf9JjG9Rd45FH1Wt/6H1k7Y
        Key ID: "[email protected]"
        Serial: 4309995459650363134
        Valid: from 2022-07-11T14:50:01 to 2022-07-11T18:50:01
        Principals:
                linda
        Critical Options: (none)
        Extensions:
                permit-X11-forwarding
                permit-agent-forwarding
                permit-port-forwarding
                permit-pty
                permit-user-rc

step certificate inspect https://smallstep.com
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 312048233971325841563421249287934957707544 (0x39506ffca33cfa15b5325cace3c6c45e118)
    Signature Algorithm: SHA256-RSA
        Issuer: C=US,O=Let's Encrypt,CN=R3
        Validity
            Not Before: Jun 6 07:05:01 2022 UTC
            Not After : Sep 4 07:05:00 2022 UTC
        Subject: CN=*.smallstep.com
        Subject Public Key Info:
            Public Key Algorithm: ECDSA
                Public-Key: (256 bit)
                X:
                    98:e4:b1:ce:2f:54:13:50:ac:af:d0:31:54:79:12:
                    0c:48:83:23:0e:c3:92:92:5e:19:20:b0:a9:f1:5d:
                    63:a8
                Y:
                    29:0d:33:d7:6a:ab:c8:d0:20:03:15:0d:9f:62:8a:
                    b5:37:ff:2b:2c:ef:60:82:b5:fc:a3:b8:9d:c8:f6:
                    70:cd
                Curve: P-256
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature
            X509v3 Extended Key Usage:
                Server Authentication, Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier:
                09:5C:5C:F4:77:93:82:7E:E2:9E:C8:3A:EE:CD:4D:40:63:E0:85:67
            X509v3 Authority Key Identifier:
                keyid:14:2E:B3:17:B7:58:56:CB:AE:50:09:40:E6:1F:AF:9D:8B:14:C2:C6
            Authority Information Access:
                OCSP - URI:http://r3.o.lencr.org
                CA Issuers - URI:http://r3.i.lencr.org/
            X509v3 Subject Alternative Name:
                DNS:*.smallstep.com, DNS:smallstep.com
            X509v3 Certificate Policies:
                Policy: 2.23.140.1.2.1
                Policy: 1.3.6.1.4.1.44947.1.1.1
            RFC6962 Certificate Transparency SCT:
                SCT [0]:
                    Version: V1 (0x0)
                    LogID: QcjKsd8iRkoQxqE6CUKHXk4xixsD6+tLx2jwkGKWBvY=
                    Timestamp: Jun 6 08:05:01.371 2022 UTC
                    Signature Algorithm: SHA256-ECDSA
                      30:45:02:21:00:8e:77:7a:d9:bf:5c:50:59:60:10:bc:2d:c0:
                      f8:6e:9b:51:1d:31:85:4c:11:fe:89:10:ae:de:5c:e1:d7:0c:
                      a1:02:20:3a:d2:1b:78:67:2c:01:7b:04:07:45:3d:34:c8:30:
                      25:23:fe:f8:1e:ce:74:41:00:a0:88:64:32:de:4a:97:c9
                SCT [1]:
                    Version: V1 (0x0)
                    LogID: RqVV63X6kSAwtaKJafTzfREsQXS+/Um4havy/HD+bUc=
                    Timestamp: Jun 6 08:05:01.416 2022 UTC
                    Signature Algorithm: SHA256-ECDSA
                      30:45:02:20:6a:ca:bb:ec:e6:94:86:2b:2d:c2:57:5a:3c:f0:
                      c5:a1:c2:66:ce:eb:f3:83:4c:c3:94:04:31:57:5e:88:b0:e6:
                      02:21:00:e1:86:6e:0b:d0:c4:5f:18:1e:08:0c:a8:0f:8b:e0:
                      4a:da:b0:a3:33:b4:8f:ae:3a:57:91:aa:13:5d:a6:c3:c2
    Signature Algorithm: SHA256-RSA
         88:a3:d8:fb:86:33:83:a7:4c:cd:cc:e3:f2:f8:04:bf:f0:34:
         1c:f8:2f:f3:09:ec:0c:39:6a:96:f4:15:62:f6:7e:20:50:75:
         e2:f6:c3:79:69:f3:7a:78:71:83:7a:28:d0:c4:51:05:77:57:
         bc:5a:40:58:d0:1d:0e:36:5f:f4:2a:e7:70:7d:70:2f:0f:b1:
         91:f1:31:d2:bd:7d:e8:0b:c3:f9:b1:f3:81:08:b3:cc:a7:0a:
         7c:63:67:97:5c:2b:12:48:7b:39:1c:a0:6c:ae:7d:df:63:76:
         0a:40:27:ca:6a:99:d5:7d:67:d2:29:77:85:bf:b3:19:57:5f:
         c2:23:f4:67:c7:09:8b:d6:c4:fd:01:7e:78:66:eb:58:4f:2d:
         87:60:90:d4:27:b5:69:60:f4:ad:91:51:35:eb:25:01:0a:27:
         47:16:eb:47:18:5a:6e:28:e4:d1:5e:f1:78:81:e7:fa:e6:e4:
         45:a9:06:c5:d4:a6:95:97:58:57:1c:eb:d3:b9:e8:c7:c0:52:
         70:88:d4:71:c1:81:ca:9c:41:e0:f4:a0:88:0d:dd:18:b2:68:
         87:52:65:ea:3a:27:c1:d1:76:8c:61:b5:47:b5:e7:63:38:bb:
         3e:0d:f7:46:f3:00:96:91:df:d2:aa:7e:d4:50:63:79:32:4e:

Implementation

In X.509 implementation, certificate authorities always have their own certificates and can have intermediate certificate authorities (certificate chaining). SSH does not allow all that. Following a more simplistic approach allows SSH certificate configuration to be easier and minimizes the attack surface.

Also, there are two SSH certificate formats: host certificate and user certificate. In X.509 you specify whether a certificate allows for “server authentication” or “client authentication” (a single X.509 certificate can allow for both)

Can you use an X.509 CA to issue SSH certificates?

No. To support SSH certificate-based authentication, you will need to maintain another CA that can handle SSH certificate signing and issuance because OpenSSH uses a custom (simpler) certificate format.

SSH Certificates are the right way to do SSH

OpenSSH certificates have a lot of power to them. They allow you to drop complex key approval & distribution processes. They also offer ephemeral SSH access, making key management oversights fail-secure. You should be using them if you’re not already!

I know that one of the biggest headaches about using certificate-based authentication is setting up a certificate authority. Enter Smallstep! With our open source tool step-ca, you can easily set up an SSH CA to issue SSH certificates to users and hosts. You can even leverage your existing OIDC flow to authenticate users and make the issuance of personal certificates simple for your whole team.

But, if you don’t want to manage your own CA server,  check out our Smallstep SSH product.

To learn more about doing SSH right, read our other blog posts on SSH:

• Run an SSH CA and connect to hosts using SSH certificates
• DIY Single Sign-On for SSH
• SSH Tips & Tricks
• DIY SSH Bastion Host

Also Published here