top of page
Search

Secure remote access with Yocto

Securing an SSH remote access

The problem


Most embedded system vendors addressing the B2B market want to be able to remotely debug faulting devices to shorten their remediation delays and save some money. Unfortunately, this often results in allowing root (or any other privileged user) to log in via the SSH port. Obviously, this is not the best idea from a security point of view:


  • The root account is exposed to the internet.

  • Changing the password can be tricky.

  • Given that everybody uses the same account, you don't know who does what.


Aware of these problems most vendors will use a bastion or a VPN to protect the SSH service. Sometimes this is also the only way to access the device that would otherwise not be reachable from internet (e.g., device behind a NAT). This can give a false sense of security, and it seems better tackling the problem at the root.


An acceptable solution


Hereafter, considering the embedded systems context, we will assume that creating one account per user is not acceptable (this is a fairly acceptable assumption).

The first point is easy to tackle : you only need to create a new account (e.g., rdebug) with restricted privileges. Sudo and a sudoers file can be used to give some additional rights to this account if required.

It turns out that the second and third points can be easily solved by using SSH certificates :


  • Each user generates a key pair protected using a personal password, the public key is then signed by the CA whose certificate is stored on the device. In consequence, each user has his own password.

  • SSH logs the ID of the certificate that is unique per certificate.


In the remainder of this document, we implement this solution using Yocto Linux. To do that, you first need to create a custom Yocto layer meta-your-layer where you will put the required recipes and files.


Configuring OpenSSH


By default, authentication with certificate is disabled in OpenSSH so the first thing to do is to enable it. Let's create an openssh_%.bbappend recipe placed in meta-your-layer/recipes-connectivity/openssh/openssh_%.bbappend with the following content :

FILESEXTRAPATHS:prepend := "${THISDIR}/files:"
SRC_URI += "file://sshd.conf"

CA_FILE_PATH ?= "${TOPDIR}/ssh_user_ca.pub"

do_install:append() {
    install -d ${D}${sysconfdir}/ssh
    install -m 0644 ${CA_FILE_PATH} ${D}${sysconfdir}/ssh/ssh_user_ca.pub

    install -d ${D}${sysconfdir}/ssh/sshd_config.d
    install -m 0644 ${WORKDIR}/sshd.conf ${D}${sysconfdir}/ssh/sshd_config.d/sshd.conf

}

FILES_${PN} += "${sysconfdir}/ssh"

This recipe only installs the CA's public key and the daemon configuration in the SSH folder, nothing more. The CA's public key can be easily generated on your computer using the following command :

ssh-keygen -t ed25519 -f ssh_user_ca -C "User CA for devices"

Hereafter the meaning of the used parameters :


  • -t ed25519 : specifies the key type.

    • ed25519 is a modern elliptic curve algorithm that is faster and more secure than RSA (and produces shorter keys).

  • -f ssh_user_ca : the filename for the key pair.

    • Private key : ssh_user_ca

    • Public key : ssh_user_ca.pub

    • This naming indicates that the key will be used as a Certificate Authority (CA) key for signing SSH user certificates.

  • -C "User CA for devices" : an optional comment stored inside the public key.

    • Purely informational, used to describe the purpose of the key.


The ssh_user_ca.pub file can be placed in your build directory or in any other directory if you set the CA_FILE_PATH variable.


Then create the meta-your-layer/recipes-connectivity/openssh/files directory and add the following sshd.conf file :

# /etc/ssh/sshd_config.d/sshd.conf

# Disable root access through ssh
PermitRootLogin no

# Disable password authentication
PasswordAuthentication no
PermitEmptyPasswords no
ChallengeResponseAuthentication no

# Enable public key authentication
PubkeyAuthentication yes
AuthenticationMethods publickey

# Set the CA pubkey
TrustedUserCAKeys /etc/ssh/ssh_user_ca.pub

# Configure the path to the authorization file
AuthorizedPrincipalsFile /etc/ssh/%u_authorized_principals

Let's describe the last line of our sshd.conf file. First, you need to know that each SSH certificate contains at least one principal that is simply a label (e.g., rdebug), it is used by SSH to know whether a certificate should be accepted. The parameter AuthorizedPrincipalsFile allows to set the path to the list of the accepted principals.


It worth noting that, by default, a certificate containing a valid principal can login using any user account (including root).However in this example, to prevent a user to login to any account, the actual path to the authorized principals list is dynamically computed using the name of the requested user account thanks to %u. For example, if I try to connect to a device using the rdebug account the latest line of the file will be interpreted as AuthorizedPrincipalsFile /etc/ssh/rdebug_authorized_principals.


Configuring user accounts and SUDO


Now the SSH service is configured, we need to configure the rdebug user account and the sudo program that will allow to execute privileged commands (e.g., halt, reboot, etc.). Let's create the following file meta-your-layer/recipes-core/users/users-config.bb

SUMMARY = "Remote debug configuration"
DESCRIPTION = "Default secure configuration for remote debugging"
LICENSE = "CLOSED"

FILESEXTRAPATHS:prepend := "${THISDIR}/files:"
SRC_URI += "file://01-rdebug"

EXCLUDE_FROM_WORLD = "1"

RDEPENDS:${PN} += " sudo"

inherit useradd

USERADD_PACKAGES = "${PN}"

GROUPADD_PARAM:${PN} = "-g 870 power"

USERADD_PARAM:${PN} = "-u 1500 -M -s /bin/sh -p '*' -d /tmp -g power rdebug"

do_install() {
    install -m 750 -d ${D}${sysconfdir}/sudoers.d
    install -m 440 ${WORKDIR}/01-rdebug ${D}${sysconfdir}/sudoers.d/01-rdebug

    install -d ${D}${sysconfdir}/ssh
    echo "rdebug" > ${D}${sysconfdir}/ssh/rdebug_authorized_principals
    chmod 0644 ${D}${sysconfdir}/ssh/rdebug_authorized_principals
}

FILES:${PN} += "${sysconfdir}/ssh/rdebug_authorized_principals"
FILES:${PN} += "${sysconfdir}/sudoers.d/01-rdebug"

CONFFILES:${PN} = "${D}${sysconfdir}/sudoers.d/01-rdebug"

This recipe depends on sudo so Yocto will automatically build and install sudo. It also inherits from useradd to create the power group and the rdebug user account which belongs to the power group. The recipe also installs the following sudo configuration file meta-your-layer/recipes-core/users/files/01-rdebug:

# POWER is just a label for power-related commands.
Cmnd_Alias  POWER       =   /sbin/shutdown -h now, /sbin/halt, /sbin/poweroff, /sbin/reboot

# Users belonging to the group power can use sudo to execute commands from POWER label with root privilege without having to enter a password
%power      ALL         =   (root)  NOPASSWD: POWER

This simple sudo configuration allows rdebug to execute privileged command allowing to halt or reboot the device. Finally, this recipe also installs the principal list for rdebug. This list only contains the principal rdebug in consequence any certificate having rdebug as principal can be used to connect to the rdebug account.


Generating user's credential


First each user, Alice here, needs to generate its own keypair :

ssh-keygen -t ed25519 -f alice_key -C "alice key"

Then the public key of Alice needs to be signed using the CA's private key to generate the SSH certificate :

ssh-keygen -s ssh_user_ca -I alice -n rdebug -V +52w alice_key

This command generates a new certificate with alice as ID and rdebug as principal. This certificate will expire in 52 weeks. Changing Alice’s credential only requires to generate a new personal certificate.


NOTE : Please note that generating a new certificate does not revoke the older one that is still usable. There are two ways to tackle this problem: first, you can generate certificates with a short validity period (e.g., one day) only when it is really required. Second, you can use a revocation file placed in each device but in this case you will have to find a way to update the local revocation file each time a certificate is revoked.

Configuring Yocto


Now that we have everything, let's test our setup. Add the following lines to your local.conf file and build your image :

# Install tools allowing to reboot our device and our recipe
IMAGE_INSTALL:append = " util-linux users-config"

# Install OpenSSH
EXTRA_IMAGE_FEATURES:append = " ssh-server-openssh"

I our case we have built a vanilla Scarthgap Poky distribution with the modifications detailed above and debug-tweaks enabled for the qemuarm64 machine. By running the qemu machine and displaying the content of /var/log/messages, we can trace connection attempts.


Client side :

$ ssh -i alice_key rdebug@192.168.7.2
Enter passphrase for key 'alice_key': 

WARNING: Poky is a reference Yocto Project distribution that should be used for
testing and development purposes only. It is recommended that you create your
own distribution for production use.

rdebug@qemuarm64:/var/volatile/tmp$

Device side :

$ cat /var/log/messages | tail -n 4
Sep  2 13:31:53 qemuarm64 auth.info sshd[257]: Accepted publickey for rdebug from 192.168.7.1 port 42964 ssh2: ED25519-CERT SHA256:RsGR9ldVwklbQFJ7wgiNsTLy/FO+HT3tUZVe5OJQ2jM ID alice (serial 0) CA ED25519 SHA256:tDC/1PPRbu3LnUXAHvUU0FFnsI81WDH0Ym6TICwGkKA
Sep  2 13:33:51 qemuarm64 auth.info sshd[271]: Connection closed by authenticating user rdebug 192.168.7.1 port 44928 [preauth]
Sep  2 13:34:05 qemuarm64 auth.err sshd[273]: error: Certificate does not contain an authorized principal
Sep  2 13:34:05 qemuarm64 auth.info sshd[273]: Connection closed by authenticating user root 192.168.7.1 port 60086 [preauth]

By reading /var/log/messages we can see that Alice (i.e., certificate ID is alice) has successfully connected from 192.168.7.1. There is also another connection that failed because the used certificate does not contain an authorized principal (in this case Alice attempted to connect using root account).


Do I have to configure an SSH device certificate ?


I would say yes but if we do a quick risk analysis, this might not be the priority. Indeed, user certificates help to prevent password leakage and allow user revocation on the other hand, sever certificates prevent server spoofing and spoofing a server is certainly more complicated than leaking a shared password.


In addition, OpenSSH already uses by default a key pair for server authentication, so you could just put the hash of the server's public key in a whitelist. Though be aware that you will have to provision, sooner or later, a device identity anyway (e.g., to connect to a cloud).


First, you should generate a specific CA to sign server certificates to clearly split the server and user certificate generation processes. Then sign the server certificate using the CA private key.

ssh-keygen -t ed25519 -f ca_server -C "Server CA for devices"
ssh-keygen -t ed25519 -f server -N ""
ssh-keygen -s ca_server -I <device serial number> -h -n <device domain or IP> -V +52w server.pub

Just use -I to set the certificate ID and -n to set the list of principals (i.e., the thing behind @ in ssh user@addressordomain).


Put the server's key and certificate in /etc/ssh/ and add the following line to your sshd_config file:

# Server key
HostKey /etc/ssh/server

# Server certificate
HostCertificate /etc/ssh/server-cert.pub

Finally, in your ~/.ssh/known_hosts add the following line :

@cert-authority <principals> <ca_server content>

Where principals is a comma separated list of principals (i.e., the same principal as the one used to generate the server certificate). You will have something like that :

@cert-authority 192.168.7.2 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHN04EaNcO+9gg4ngp0tFmM1lebulDfEsPVkSpzkE3K0 Server CA for devices

When using server certificates, ssh client should not ask for public key validation when you connect to the server for the first time.


Do I have to disable the root account ?


Some internet resources advise to disable the root account to reduce the attack surface of a Linux based product. However, most of maintainer need to execute privileged commands and therefore need some kind of privileged admin account (much more privileged than our rdebug account). You could try to disable the root account and create a new admin account with reduced privileges using sudo however this is actually a risky approach.


Indeed, even if the root account is locked, some commands and services could be hijacked to gain root privileges. I believe properly configuring sudo to prevent privilege escalation is quite difficult. In my opinion it is pretty easy for an attacker to gain root privileges therefore locking the root account may give a false sense of security while complexifying the device setup.


If you still want to lock the root account, you can add the following lines to your local.conf file as explained in the Ejaaskel's blog :

INHERIT += " extrausers"

EXTRA_USERS_PARAMS += "\
    usermod -L -e 1 root; \
    usermod -s /sbin/nologin root; \
"

Conclusion


Using SSH certificates and sudo, it is possible to improve the security of a device by implementing remote access through SSH. This does not involve complex configuration steps and therefore can be easily deployed with Yocto. Locking the root account is a definite plus, but in this case, sudo must be carefully configured so that it is not useless.


To go further


TrustnGo is a company that provides software products, consulting, development, and training services in embedded cyber-security. Do not hesitate to contact us whether you have any questions about our offers or just want to discuss this blog post.

Comments

bottom of page