It was an extraordinarily busy and educational past year for me, working in the cloud computing space. In preparation for the ’20s, I spent the (not-so-)quiet time off of work at the end of last year rebuilding several home networks, installing and configuring personal servers, and otherwise leveling-up my homelabs - redoing and improving things now that I know better. This multiparter is a log of that!

Read the other articles in this series here:

Where we started

Note: when I say “site”, I mean “mine or a family member’s home”. If you read something here that makes you go “that’s not good enough!”, like my current lack of RAIDZ… it’s a HOMElab :)
I have 3 sites that I’m going to be refreshing during this process. Two have UniFi gear; the third has comparable small-business equipment that is not centrally/cloud managed. They all have several POE WAPs, dozens of wireless clients, and a handful of wired clients. More detail in the Networking post.

There’s only one existing “server” - aging, cheap, commodity hardware (AMD FX-6300, yikes!) and it recently failed an OS upgrade and has been stuck at:

grub>

for months. It’s “offsite” - several hundred miles from where I live - and no one technical is present there. It’s a challenge troubleshooting over the phone (reading out terminal commands letter by letter is painful), which was a major motivator for the real Supermicro server board going in to the Epyc build - it has IPMI! I’m going to attempt to save my data and rescue that install via an Ubuntu live USB, SSH, and chroot in the Rescuing post.

The FX server has several TBs of data on it in a 4 drive Btrfs RAID10, but one of drives has started failing and Btrfs is having trouble removing it. The one

$ sudo btrfs scrub

that I attempted ran for a week and then threw a few hundred million disk errors. Though degraded, the array is still readable; a priority is going to be offloading that data so that those disks can be wiped and replaced. To improve the filesystem integrity for the future, real server will have a ZFS pool because “either you care about your data and you use ZFS, or you don’t”. [Note: Yes, a 3-2-1 strategy is the only way to go if you really care about your data, but this isn’t production data, and that’s expensive $$$]

Further, there’s no existing point-to-point VPN set up, even though most of the gateways support IPSec or OpenVPN…I’ve been getting by with opening 22/TCP in the firewall and SSH port forwarding. Instead of setting that up, we’ll leapfrog it entirely and set up a secure WireGuard mesh - an encrypted overlay network that will let us peer with the other LANs to make remote access of these networks safe and secure.

The Epyc server is also going to host an always-on, GPU passthrough’d Windows virtual machine. This VM is going to replace a standalone Windows desktop and that hardware is going to be repurposed. Don’t worry, the server is going to have resources to spare and it should be quite performant even with the VM pegged out.

Next, let’s get on with the Epyc server build, or you can jump to any of the other write-ups in this series here.