Water on the floor
One morning I woke up to a loud noise and the sound of water pouring onto the floor. The inlet filter of the water softener had given out. That filter sits at the entry point of the whole house: every tap, every appliance, everything downstream of it.
I didn’t go looking for towels first. I followed the noise and the water backwards until I found where it was coming from. Then I shut the main valve, which I didn’t have to look for, because I already knew where it was. Only then did I dry the floor, go to the hardware store for a replacement, and fit it. About 45 minutes from the flood to the new part, and a quarter of an hour of that was the round trip to the store.
Nothing clever happened there. It’s the order that matters: trace the symptom to its source, contain it, clean up, repair. Anyone who has been paged at night for a production outage does it the same way. And the step that saved the floor was knowing where the valve was before I needed it.
The other thing that saved it was not panicking. Water on the floor pushes you to do something, anything, right now: grab a towel, call someone, start moving things. The towel can wait until the valve is shut. What happens to your head during an incident, at home or in production, deserves a post of its own.
Screws in plain sight
I’ve been taking things apart for as long as I can remember. First toys, the ones with screws you could see. Then electrical devices. Then, in 1994 or 1995, a 386 with 4 MB of RAM and a 200 MB hard drive, which is where I worked out how the pieces of a computer actually connect to each other. Over time I added a sound card and a CD-ROM drive to it, and upgraded the video card.
None of that was a career plan. It was curiosity, the itch to know what’s inside and why it stops working. That habit never stayed inside the computer case. It moved on to light bulbs, small bits of electrical wiring, plumbing and DIY jobs around the house. Same screwdriver, same questions: what changed, where does the flow go, what can I rule out?
That’s the whole premise of this post. The way of thinking you pick up fixing PCs doesn’t care whether the thing in front of you is a server or a sink. Mostly, that works in your favour. Sometimes it doesn’t, and I’ll get to that.
The same flood, in production
On a work project, an old web server stopped responding. It was so old that nobody had it in the inventory any more. I didn’t find out from the server itself. I found out because a separate alert, a secondary check on a resource that happened to be hosted on that machine, started failing.
Same sequence as the filter. I followed the symptom back to the machine and found the cause: a compromised PHP package was saturating the traffic. So I shut the valve first. I pulled the server off the network and got a shell on it directly through the VMware console, with no network path in or out. Then I cleaned it up. From the alert to a clean machine took minutes.
The repair came later, and it was the part that mattered. A server nobody knew about is a pipe behind the wall: you only find it when it leaks. I decommissioned it and migrated the services that still depended on it to newer, safer systems, small application functions instead of one forgotten box.
Teflon on brass
A few months after the filter, I noticed damp behind the water softener. One of the joints was badly sealed and dripping. It was one I had touched myself during the repair.
I took apart all four joints around the filter, one before it and three bends after, and resealed them with Teflon tape. I’d chosen Teflon because I had already done other connections at home with plastic pipes and fittings, and there it had been perfect.
These fittings were brass, and I had used too little tape. It didn’t hold. I spent a weekend of attempts on those four joints before I switched to hemp and green sealing paste, which fixed it. Later, talking to people who do this for a living, I learned that hemp was the better choice, but a generous amount of Teflon would have worked just as well. A plumber would have got it right on the first try.
The method transfers, the domain doesn’t
Look at what actually went wrong in that weekend. The method was fine: I found the leak, I traced it to the joint, I knew it was my own work. What failed was the knowledge of the trade: which sealant, and how much of it. I took something that had worked in a similar case and applied it to a case that only looked similar.
I’ve never had a Teflon-on-brass moment in IT. There, I have always read up thoroughly before touching anything. At home I trusted my previous job on plastic pipes and skipped that step.
So the troubleshooting mindset does transfer: isolate, trace the flow, contain before you repair. The domain doesn’t come with it. Today I’d do it the other way round: read first, and call an expert only when I can’t understand or fix it myself. Mostly because the expert, quite reasonably, wants to be paid.
Hire the ones who take things apart
If you hire engineers, this is the part for you.
Certifications are great, and genuinely useful. But without the right attitude, they give you an engineering graduate standing in front of a combustion engine that won’t start. They know every process inside it perfectly. And they will never get it running if they aren’t willing to get their hands dirty.
The domain knowledge can be taught, read up on, or borrowed from someone who has it. The urge to open the case and find out why it stopped working is much harder to teach. When you interview, ask about the last thing they took apart that nobody asked them to.
Takeaways
- Don’t panic: the towel can wait until the valve is shut.
- Know where the main valve is before the flood, for your house and for every system you run.
- Follow the symptom back to its source before you touch anything.
- Contain first, clean up second, repair third.
- An uninventoried server is a pipe behind the wall: you find it when it leaks.
- Secondary checks catch what primary monitoring doesn’t know exists.
- The method transfers, the materials don’t: read before you wrap.
- Hire for the urge to take things apart; certifications come after.