Seven years, no HA, one sysadmin who made it work
For seven years, from 2014 to 2021, I ran the internal infrastructure of a mid-size company on a three-node Proxmox cluster. No high availability. No major version upgrade, ever. Backups on the same SAN as the VMs they were meant to protect, with a second copy of only the most critical ones in the office. One sysadmin, me, with Nagios alerting around the clock and no on-call rotation, because there was nobody to rotate with.
The part of the rack that made the money wasn’t on it. The company’s main production platform ran on bare metal, four front-end nodes and the servers behind them, and the rule was hands off. My job there was maintenance: purging database tables, replacing a failed disk in one of the front-end nodes. Everything else ran on the cluster: accounting, email, the internal CRM, Active Directory, the firewall and the phone system, in the same hosted rack, on a budget that didn’t stretch any further.
It worked. This post is about how it held together, the two times it didn’t, and why I wouldn’t build it the same way today.
One sacrificial VM, then an inherited cluster
Until 2013 everything in the rack was bare metal: one service, one box. The production platform stayed that way to the end. Proxmox came in for one reason, budget, and it came in carefully. We ran it off the free no-subscription repository.
The first node ran a single real workload: a Windows Server 2012 machine hosting a non-core service. If it had been down for a few hours, nothing would have happened. That was the point. It was a dry run on something we could afford to lose. Alongside it I built a new Linux web server (Debian, nginx, PHP, MySQL), the first machine in the rack that was born virtual.
The production cluster wasn’t designed, it was inherited. A sister company shut down its own rack, and we took in three of its servers and a Dell PowerVault MD3200, an iSCSI SAN with two controllers. In 2014 I reinstalled everything from scratch as a three-node cluster, copied the VMs over from the single node, and started moving the internal services onto it.
The original node didn’t retire. It went to the office as a single-node Proxmox host, running a replica domain controller and a backup VM.
What the cluster actually looked like
Three nodes with 16 GB of RAM each. The MD3200 with about 12 TB: roughly 9 for the VMs, 3 for backups. A 24-port switch with physical segmentation and dedicated VLANs, one for the internal network and one uplinked to the rack’s main switch.
On top of it:
- the accounting server and its SQL Server, both on Windows;
- the mail server, Postfix and Dovecot;
- the internal CRM, an ASP.NET application on IIS with its own Windows server;
- nginx as web server and reverse proxy, in front of both the PHP sites and the internal Java applications on Jetty;
- FTP and SFTP;
- an incubator server, where new products were tried out before going to market;
- the Active Directory master;
- pfSense, holding a slice of the company’s public IP addresses.
The office was tied in over IPsec between two pfSense firewalls: a VM in the rack and a dedicated machine in the office. Both sites shared a single 10.0.0.0/8. I picked a /8 because I had big plans for a lot of VMs. The AD master lived in the rack and the replica in the office, because the link between them was unreliable. I also migrated the office’s physical SIP PBX, on its own dedicated phone line, to a virtual SIP PBX with virtual fax, running as a VM in the rack. It worked until the day I left the company, and for all I know it still does.
At this scale, a shared SAN and VMs placed by hand is a legitimate design. The catch is that every shortcut in it is a risk you carry every day, and you need to know which ones you’re carrying.
No HA, no upgrades, backups on the same box
No HA. Every VM had a home node and resources were rationed. When something had to move, I moved it by hand. The rationing is what made that possible: with one node down, everything still fit on the other two.
No upgrades. In seven years the cluster got system updates, never a major Proxmox release. Once a cluster carries accounting, mail and AD, and the only person who can fix a failed upgrade is the one doing it, “I’ll do it when I have time” quietly becomes never.
Backups on the same box. A dedicated backup VM had a disk of about 3 TB on the SAN, and every server pushed its own data to it. It restored fine when I needed it. But it sat on the same storage, in the same rack, behind the same controllers as the data it protected. The backup VM in the office held a second copy of the critical services, accounting and mail among them, so losing the SAN would not have meant losing the company. Everything else only existed next to its original. That’s the one I’d change first today: a backup that shares a failure domain with the original is a copy, not a backup.
Quorum loss, split-brain, fencing, the things most Proxmox guides warn you about, never gave me a single problem. What gave me problems was a disk shelf, and me.
Two incidents
The first one was the hardware. A VM went offline, Nagios flagged the service, and the SAN console showed three disks in red. The hot spares had kicked in, two on the 16-disk shelf and one on the 12-disk one. I went to the datacenter, swapped the disks, let the arrays rebuild, restarted the VM, and everything came back.
Nagios was watching services, not storage. It told me a service was down. It didn’t tell me disks were failing. The SAN knew, and nobody was asking it.
The second one is the one I’m least proud of. I was experimenting with new VMs on the same cluster that ran production, and I wanted to merge two of them into one, by hand, editing config files and virtual disks. One moment of distraction and I wiped the mail server’s disk.
I found out when someone asked why email wasn’t working. I said there had been an infrastructure problem and I was fixing it. Nobody asked more: the users weren’t technical, and there was no incident process to ask on their behalf.
The recovery went like this. While the mailboxes were being restored from the backup VM, I brought up a clone of the mail server: configuration first, then webmail, with IMAP and SMTP closed to mail clients. I told people to use webmail while their mail came back. When the restore finished, I reopened client access. About an hour of work, maybe five minutes of visible downtime, no mail lost. It was late in the evening, and we didn’t get much email anyway: support and sales worked by phone, and administration used PEC, Italy’s certified email.
Both incidents ended well for the same reason: one person improvised a good recovery. Not a runbook, not a second pair of eyes, not a process. That works until the one person is on holiday.
Leaving the rack in 2021
In 2021 the whole rack was decommissioned. Half of it had already been retired, and it made no sense to keep paying for half a rack to host things that were already virtual. The rack was expensive. The managed VMware VPSs we moved to cost about a third of it.
The migration took a while, but less than I expected. Windows servers went over with backup and restore, Linux ones with rsync. Porting the firewall and NAT rules from pfSense to the VMware side was just as painless.
Would I run Proxmox on-prem today?
Only with a team that covers hardware, software, networking, infrastructure and security. One person with no change process decides on experience, knowledge and instinct, and nothing guarantees the result works. Seven years of changes without a single change request went well. That was luck for the company, not a method.
Without that team, I’d go managed, and which one depends on compliance, budget and the vendors the company already has contracts and processes with. For a small company, DigitalOcean or Hetzner. If it already runs on Microsoft 365, Azure, without a second thought. Managed VMware only if you already run VMware; otherwise I don’t see the point.
AI doesn’t close the gap, it moves it. Recently I watched an agent, working from incomplete context, try to add resources to infrastructure managed partly in Terraform and partly through the vendor’s web console. It wanted to create a network interface with a set of security rules that were already in place, created from the console, and they would have conflicted with existing ones. The agent was only as good as the context it got, and assembling that context is exactly the expertise a one-person setup is short on.
Takeaways
- Pilot on a workload you can afford to lose.
- Keep the lab off the production cluster.
- A backup on the same storage as the data is a copy, not a backup.
- Monitor the storage, not just the services running on it.
- Without HA, size and ration so that a node down still fits on the others.
- Plan upgrades, or plan the exit: seven years on one major release is debt.
- Give an AI agent the full picture, including what was changed outside your IaC, or don’t let it touch the infrastructure.
- Have a business continuity and disaster recovery plan that actually works: it costs up front, and always less than recovering without one.
- Ask what happens to your company if your one technician gets sick, is less experienced than you think, or can’t be trusted.
- Then ask them: “If this breaks, how do we bring it back? Can we try it now?”