yourgeek.it_

~/ new-server-already-attacked

Your new server is already being attacked

#security#infrastructure#networking

A server nobody knew about

One morning I created a droplet on DigitalOcean from scratch, with Ubuntu 26.04, installed nginx and left everything open. No cloud firewall, no ufw, nothing in front of it. No domain pointed at it. A few hours later I added PostgreSQL, listening on every interface, with connection logging switched on.

This is not an incident. It is an experiment, and I ran it to show one thing: how easy it is to forget a port, and how quickly someone else finds it.

I already suspected the answer. A server left open on the internet gets checked sooner or later, and not because someone looked you up. Someone scans whole ranges of addresses, and yours is in one of them. Who, and why, you rarely find out. What I didn’t know was how soon. So I read /var/log/auth.log and the nginx access log for the first six hours.

One thing before the numbers. SSH accepted keys only: I switched password authentication off within the first hour, before the first password-guessing attempt arrived. Nothing below had a real chance against that port. The point is who showed up, how fast, and what they asked.

Port 22: industrial, not personal

sshd started listening at 07:51 UTC. The first connection from an address that wasn’t mine came 85 seconds later. It opened the connection, read the banner and left: a check that there is an SSH server here, and which one.

The first real attempt to log in as root came about 80 minutes after boot. From there the log shows three patterns.

  • One address, patient. A single IP tried root every two minutes for about an hour, some thirty attempts.
  • Many addresses, one rhythm. From mid-morning, one root attempt every five minutes, almost to the second, rotating across 8 addresses in 3 blocks. Each address came back only every 25 to 40 minutes. Eight machines keeping the same schedule look like one campaign to me, not eight people.
  • A dictionary. For twenty minutes one address went through 14 usernames, an attempt every 31 seconds or so: root, admin, user, test, guest, support, ftp, nobody, user1, sshadmin, ubuntu, debian, cirros, orangepi.

The last four names are the interesting ones. ubuntu and debian are the default users of cloud images. cirros is the default user of a tiny test image used in OpenStack. orangepi is the default on a family of single-board computers. That list wasn’t written for my server. It was written for anything out there still running with a default account, and my server was one more address to try it on.

Two scanners didn’t even get that far. They offered only classic Diffie-Hellman key exchanges, which recent OpenSSH no longer enables by default, and the server refused them. Some of the software scanning the internet is older than the servers it scans.

With passwords off, none of this could get in. If your servers still accept them, what happens next is in an earlier post.

Port 80: are you one of these?

nginx served its default page and nothing else. The access log filled up anyway, and almost nothing in it was looking for my page.

Even Googlebot came by, one second after I opened the page myself for the first time. It was the real one: the address resolves back to Google’s crawler. My guess is that the IP had served an indexed site for a previous tenant, since cloud providers recycle addresses. I can’t explain the timing.

Seventeen minutes after boot, someone asked nginx to fetch a page from another website on their behalf. It’s a test for an open proxy: a server that relays anyone’s traffic and hides where it came from. nginx answered with its own default page instead.

Then the questions got specific. Each of these paths exists only on a particular product:

  • /SDK/webLanguage: Hikvision IP cameras, the endpoint of a known command-injection flaw. The same address asked three times during the day.
  • /+CSCOE+/logon.html: the login page of Cisco’s VPN appliances.
  • /geoserver/web/: GeoServer, a map server with a remote code execution bug that was exploited widely.

Others I can’t pin to a single product: /admin/config.php, /webui/, and a POST /rpc that two different addresses sent with the same client. Each of them still asks the same kind of question.

None of this was aimed at me. It is a checklist run against every address it can reach, and every line asks the same thing: are you one of these? A freshly installed nginx answers no to all of them. The admin panel someone installed “just for a test” and forgot answers yes.

Postgres: found in about 18 minutes

In the afternoon I installed PostgreSQL. Ubuntu’s package listens on 127.0.0.1 and ::1 only: the machine can reach it, nobody else can. To expose it I had to change two things on purpose. listen_addresses in postgresql.conf, so it listens on every interface, and a line in pg_hba.conf that accepts password logins from 0.0.0.0/0.

Postgres started listening on every IPv4 address at 13:56. At 14:15, 18 minutes later, the first stranger connected. Within one second it sent three things:

  1. a startup packet claiming protocol version 0.0;
  2. another claiming 255.255;
  3. a valid one with no user name.

None of these is a login attempt. They are three ways to make the server talk, and it did: unsupported frontend protocol 0.0: server supports 3.0 to 3.2. From one error message the scanner learned that this is PostgreSQL, and roughly how recent, since protocol 3.2 only arrived with PostgreSQL 18. It needed no password for that. Half a minute later a second scanner, from a large cloud provider’s range, tried protocol 16.0.

The first one came from the same block of addresses that had asked my nginx for /webui/ and /geoserver/web/ that morning. My reading is that whoever runs it mapped port 80 first and came back to see what else had opened. I can’t prove it’s the same operation, only the same network.

An hour later, at 15:13, three addresses showed up within two seconds. Two dropped out halfway through the encrypted handshake. The third got further: it introduced itself as user postgres on database postgres, and hung up as soon as the server asked for a password. The server’s answer at that point says how it wants you to authenticate. Get “come in” and you’ve found the oldest mistake in the book, a superuser with no password. Get a password challenge and you’ve learned something worth writing down anyway. I don’t know which of the two this scanner was after. One of the three came from the same block as an address that had knocked on my SSH port that morning.

In the hour and a half or so that Postgres stayed open, nobody tried an actual password.

This is the step most people never see. Nobody guessed a password, but the server is now in someone’s catalogue: the product, the version, and the fact that it answers from anywhere. In my experience, from that catalogue to abuse is a very short step: the day a flaw comes out for that version, the list of servers to try it on already exists. That is my experience, not something this experiment showed.

What you expose is one line of config

ss -tlnp lists every TCP port the machine listens on, and which process holds it. This is the droplet at the end of the experiment, trimmed to the columns that matter:

Local Address:Port    Process
0.0.0.0:22            sshd
[::]:22               sshd
0.0.0.0:80            nginx
[::]:80               nginx
0.0.0.0:5432          postgres
[::]:5432             postgres
127.0.0.53%lo:53      systemd-resolve
127.0.0.54:53         systemd-resolve

Read the left column. 0.0.0.0 means every IPv4 address the machine has, [::] every IPv6 address: anyone who can reach the server can reach that port. 127.0.0.53 is loopback, the machine itself and nobody else. The DNS resolver lives there, and that is where a database should live too, unless something on another machine needs it.

Postgres went through three of these states in one afternoon, one line each time. The package default was 127.0.0.1 and ::1. With listen_addresses = '0.0.0.0' it showed up on IPv4 only. With '*' it showed up on IPv6 too, which is easy to miss when all you typed was an asterisk. And listening is only the first layer. pg_hba.conf decides who may authenticate, and mine only had a remote rule for IPv4: over IPv6 a client could still connect and talk to the server, and be turned away before the password. As the scanners showed, talking is already enough to end up in a catalogue.

The droplet had a public IPv6 address the whole time, and in six hours not a single probe arrived over it, on any port. On IPv4, finding an exposed server takes minutes, even when nobody knows its address. IPv6 is far too large to sweep blindly, so a scanner only finds an IPv6 address once it’s published somewhere, in DNS or in a list. That’s my explanation, not something I measured. Today it keeps you out of the trawl. It doesn’t close the port, and the day the address lands in a DNS record, the port is there to be found.

Containers add a trap of their own. On a typical Linux host, with Docker’s default firewall integration, publishing a port (say 5432:5432 in a compose file) adds Docker’s own rules, and traffic to that port is forwarded to the container without going through ufw’s rules. ufw says the port is closed; the internet disagrees. You can publish on 127.0.0.1:5432:5432, and you should. But I can’t stop someone from adding a ports: line one day, so I close the port where no file on the server can reopen it: outside the machine, at the cloud firewall.

Close it from minute zero

None of this is new. What I do on a new server is the usual list. What matters is the order, and that it’s done before the server spends its first 85 seconds on a public address.

  1. Cloud firewall first, attached at creation. On DigitalOcean a firewall can be applied by tag: every droplet created with that tag is born behind it. Everything is closed, and only what the server is for gets opened. For me this is a policy, not a good intention: a server without a firewall doesn’t get created.
  2. SSH with keys only, password authentication off from the first boot. Why, in detail.
  3. Databases on localhost or on the private network. If the app runs on the same machine, 127.0.0.1. If it doesn’t, the provider’s private network, never the public interface. Containers publish on 127.0.0.1, or not at all.
  4. fail2ban. It’s useful for cutting down noisy, repeated attempts, but it isn’t the security boundary. With the usual thresholds it would have banned the dictionary run and the address trying every two minutes, within minutes. The rotation across eight addresses, each one coming back every half hour or so, would most likely have stayed under any per-address threshold. That’s what the firewall and the keys are for.
  5. Check from outside. ss tells you what is listening; only a scan from another machine tells you what is reachable. Do both, for IPv4 and IPv6:
nmap -Pn -p- <server-ipv4>
nmap -6 -Pn -p- <server-ipv6>

Every open port in the output should have someone who can say why it’s there.

If you own compliance, the time to write this down is before the first server, not after the first audit. “Every new server is born behind a firewall, and every exposed port has a documented reason” is a rule you can check. “We’ll lock it down once it works” is how a test database stays on 0.0.0.0 long after the test.

What to take from it

  • A new server gets scanned within minutes. Nobody needs to know it exists.
  • The scans are a checklist asking “are you one of these?”. Make sure the answer is no.
  • 0.0.0.0 and [::] in ss -tlnp mean the whole internet. Each one needs a reason.
  • listen_addresses = '*' opens IPv6 too.
  • With default settings, ports published by Docker bypass ufw: close them outside the machine.
  • Cloud firewall at creation, keys only, databases on localhost or the private network, fail2ban on top.
  • Verify from outside with a port scan, on IPv4 and IPv6.
  • Make it a policy from day zero, not a step someone has to remember.

← all posts