A Fake sshd on a Production Node: Why SSH Needs Keys

A monitoring alert told me that one of our four production nodes was extremely slow. The other three were fine.

Finding a binary that shouldn’t be there

I logged in and looked at what was eating the machine. It was sshd. Not the daemon I knew: this one was running as a non-root user, it was consuming a lot of resources, and its permissions were off. Even its modification date was wrong.

I got there with the usual tools: ps and top for the process, ss for the ugly traffic it was producing, ls for the permissions and the date.

I had a twin: another node of the four, installed the same way. I compared the two machines file by file. Only sshd had those problems. Everything else matched.

That comparison is the part I’d keep from the episode. When you suspect a machine, a sibling that you trust gives you a reference in minutes, with no tool to install. Without the twin I would have spent the day wondering what else had been touched. A clean reference doesn’t tell you how the compromise happened. It tells you what changed. Look, compare, and only then touch anything: I wrote about why that order matters in Don’t panic: the order matters more than the speed.

Seven years, one password

Someone had broken in and replaced the OpenSSH binary with a modified one that ran a cryptominer. I never proved how they got the first access, so this is a deduction.

The server had a service account that could use sudo, protected by a password of 16 characters. It had been on the internet for seven years, with SSH open to anyone, so that account had been exposed to years of automated attempts. Password guessing is one possible explanation. I simply don’t know.

I don’t remember whether root login was also enabled on that server. It hardly matters: that account had unrestricted sudo, with no rules at all in sudoers, so it was as exposed as root itself, and root can replace any binary.

A 16-character password is not a weak password. The problem was never its length. The problem is what password authentication gives an attacker on the internet: a secret they can test again and again, from anywhere, for as long as the port stays open. A private key is not something an attacker can realistically brute-force through repeated SSH login attempts. With public-key authentication the client proves it holds the private key, without sending the private key to the server, so the attacker doesn’t have something to guess: they have to own something.

A morning in the rack

The server could only be reached physically, so I went to the data centre. I shut the machine down and booted it from a rescue Linux. From there I restored the original sshd binary and cleaned up what the miner had left.

It was an easy cleanup for one reason: the machine ran a single Java application and nothing else. There was little to audit and nothing to untangle.

Afterwards I reset the passwords and moved every server in that rack to key-only SSH.

The downtime lasted a morning. The damage was small, because the other three nodes held the load without trouble.

Keys configured, passwords still accepted

A few years later I took over an infrastructure that included a few VPS with public SSH. Access was meant to be by key, and the keys were in place. But sshd still accepted passwords.

I noticed it in the authentication logs, which were full of login attempts: thousands a day, on two or three machines.

The machines were used by two or three customers, who logged in with a password. So I couldn’t just switch passwords off: first everyone needed a key. I warned that it had to be fixed right away, then I wrote to those two or three customers: “generate a key, thanks.”

They said yes immediately. That was the whole project.

The other road, the one where nobody asks, would have meant the same story in the cloud: a shutdown, a login from the recovery console with no network, and a restore.

What I’d set up today

After these two stories, my rule has three parts.

SSH is not exposed indiscriminately to the internet. Put a firewall, a security group, a VPN or some other access control in front of it. Not open to the whole internet, not even “temporarily”: a new server gets its first knock within minutes.

Access is by key only. Password authentication is off.

Passwords survive only where the job needs them, and only for accounts with a narrow scope. An SFTP user who can only reach its own directory is the typical case. An account that can run sudo is never one of them.

Then there is the key itself. My position: a key protected by a passphrase. A password is still a secret that, when password authentication is exposed, the server lets an attacker repeatedly try, and the cryptominer node is the case that made it concrete for me. The passphrase on a key works differently: it protects the file if it ever leaves the machine it was generated on, and it is never sent to the server. That is my position, not a law of nature.

RSA 1024, RSA 8k, ed25519

The last time I joined a new company, a few years ago, I looked at the SSH keys people used. Mine was an RSA key of 8k bits. The team’s were RSA keys of 1k.

Nobody had told them to use 8k, and I wasn’t there yet to nag them about it. Nobody had made a catastrophic mistake: they were simply old keys, generated under different assumptions. Today I would not generate a new 1024-bit RSA key.

After reading the best practices and the later updates, I decided to replace all of them with ed25519 keys, and I led the refactor.

Why ed25519 and not a bigger RSA? The keys are much smaller and the algorithm is efficient, which makes it a good default when the systems on both ends support it. I had also read about several other advantages of elliptic curves over RSA. And nearly every system we had supported them.

The exceptions were sporadic, and only where they were needed. The one I remember: the git service used by the devops team at that time didn’t support elliptic curve keys, so it stayed on RSA until it was decommissioned.

What to do

• Check password authentication on every server of the fleet, not only the ones you remember. sshd -T | grep -i '^passwordauthentication' takes seconds per machine, and it should print passwordauthentication no. It doesn’t show Match blocks that override the setting for some users or addresses, so read those too.
• Before you turn passwords off, make sure every user has a key. Ask them, it takes one message.
• Keep a second session open while you change the SSH config, so a typo doesn’t lock you out.
• Treat an account with unrestricted sudo as root, because it effectively is.
• Put access control in front of SSH (firewall, security group, VPN). Don’t leave the port open to the world.
• Protect your keys with a passphrase.
• Use ed25519, unless a system on the other end can’t.
• Keep a trusted twin of every machine you run, or at least a clean reference: it is what lets you spot the one binary that doesn’t match.