Homelab: Overview

I am starting a series about my homelab and how it is all laid out. I have written this article a few times, with months in between. Each time the setup changes, but we seem to be at a stable-ish point where I will start this series. Since I wrote this whole article and now a while later am editing it, I will mark with italics and underline when present me is filling in. I think it will give a neat split of growth in the last year or so I have been working on this. Or it will make it illegible, we will see. My home setup gives me a good chance to test out different operating systems and configs in a domain environment before using that tech elsewhere like at work.


Starting off with virtualization technology, I settled a while ago on Microsoft Hyper-V instead of ESXi, the main reason behind it is I already had Windows Server, and Hyper-V allows for Dynamic memory, and allocating a range of memory for a VM. When something like an AD controller is idling, it doesn’t need much memory; when it starts it may, Dynamic memory allows me to take that into account. I will say one place that has bit me later is file storage, but that will be a later post.

The setup is technically “router on a stick”, where the Sophos XG firewall functions as the router, and the rest of the devices hang off of that. The Sophos XG machine is a old Dell Optiplex 990 (almost 10 years old!) with an Intel quad NIC in it. That way it can do hardware offloading for most of the traffic. I intend to do posts for networking, hypervisors, file storage, domain, and more; thus I will not get too in the weeds right now on the particulars.

The file storage is a FreeNAS box recently updated to 7, 3TB HDDS. I have had this box for about over 6 years (I just looked it up in November 2020, one of the drives has 55257 hours or 6.3 years of run time on it); it is older but has worked well for me so far.

The network backbone is a new switch I really like that I was able to get 2 of off eBay; they were broken but I was able to repair them, more on that later as well. They are Brocade, now Ruckus, ICX7150-12P; 12 1GB/s POE ports, 2 additional 1GB/s uplink port, and 2, 1/10GB/s SFP/SFP+ ports. These switches can run at layer 3, but I have the layer 2 firmware on them currently. They have a fiber connection between them, before that I was using 2 Unifi APs in a bridge, that didn’t work fantastic however because A. I am in NY, B. they were only 2×2 802.11AC Wave 1, and C. I am in NY. I custom ordered (so the significant other would not get mad) a white 50m fiber cable to go around the wall of the apartment.

With SSDs in the hypervisor boxes (I call them HV# for short) and iSCSI storage for VMs as well, which VMs are on which host doesn’t particularly matter. Flash forward 6 months or so, since that first sentence was written, I now still use the NAS for backups, but the hypervisors are running Storage Spaces Directed and doing shared storage now. This allows the hypervisors to move move VMs around during patching or pause during a system update if they are less critical. The Intel NUC and small Dell Inspiron are much under powered compared to the mid tower hypervisors, so they run usually only 1 or 2 things. The NUC runs the primary older domain controller, and that is it. It is an older NUC that I got about 7 years ago, so its not that fast. The “servers” in the hypervisor failover cluster are a Lenovo and 2 Dell Optiplex 5050s. I like these Dells because they go for about $200 on eBay, while having a Intel 7600 i5, can support 64GB of ram, and have expansion slots for things like 10gb SFP+ cards. These machines also idle at about 30 watts, which makes the power bill more reasonable.

Some of the services I run include:

  • 2 Domain Controllers (Server 2016, and 2019)
    • Including Routing and Access service for RADIUS and 802.1x on wifi on wired
  • Windows Admin Center Server (Windows Server 2019)
  • Windows Bastion (This box does Windows Management) (Server 2019)
  • Veeam Server (Server 2019)
  • Unifi Controller/Unifi Video for security camera (Ubuntu)
  • 3 Elastic Search boxes for ELK (CentOS 8)
  • Linux Bastion (CentOS 8)
  • Foreman Server (CentOS 8)
  • LibreNMS (This I grew to really like) (CentOS 8)
  • Nessus Server (CentOS 8)
  • Jira Server (CentOS 8)

That is the general overview, I will spend the next while diving into each bit and discussing how it is configured and what I learned in doing that.

Managing devices via SNMP v3 in LibreNMS

I started to deploy LibreNMS at home as a way to see all the systems on the network and any outstanding issues they may have. This is outside of log aggregation that I plan to do with ELK. Its been fairly smooth, running through the installation guide, https://docs.librenms.org/Installation/Install-LibreNMS/ for CentOS 8 (my standard Linux flavor at home for servers) was a breeze, then we just needed to add SNMP devices. I’ll post more when the whole system is together and I have a chance to put dashboard together, but for now I thought I would post some snags I hit.

Devices Screen in LibreNMS

SNMP v3 is considered a lot more secure than older versions, so I am sticking with that wherever possible. There are a few commands you need for SNMP v3 in its strongest mode that makes sure no one can read the data, authpriv. Username, password, crypto password; password and crypto password also have different modes available. AES or DES for crypt, and MD5, or SHA for password hashing. Some devices now offer SHA-256, or SHA-512; LibreNMS does not, so lookout for that, a few of mine had to drop down to MD5 to find a matching mechanism. Every device defaults to port 161 with UDP for SNMP. You can also run SNMP over TCP.

Before I dive into different devices and how to configure them to work with LibreNMS/SNMP, I suggest getting familiar with snmpwalk as a command. It lets you quickly test and figure out if your setup is working, since if the settings are wrong in LibreNMS device setup, it erases them for you to start again. Some devices have you white list or select which IPs will be reading SNMP, testing from your LibreNMS host can save you some pain as well.


Windows Servers easily support it, and have a guide on the site. This page walks you through it, https://docs.librenms.org/Support/SNMP-Configuration-Examples/ and a lot of other devices. This is one where the easy solution is SNMP v2, Windows does allow easy white listing for SNMP servers, that made me feel good enough about the security over not having SNMP v3.


snmpd via the link above. Your stock LibreNMS host gets a premade config, that can be easily copied.

Ruckus/Brocade Switch on FastIron Switching Firmware 8.0.90d

The below commands add SNMP v3 to the switch. You need to make a group that you specify which privileges it has, then add a user to that group with the required password and crypt password that SNMP v3 in authpriv mode needs. Here is my LibreNMS host, librenmsuser is my user, xxxxxxxxxxxx is my password and yyyyyyyyyyyy is my crypto password.

snmp-server host version v3 priv librenmsuser
snmp-server group librenms v3 priv read all write all notify all
snmp-server user librenmsuser librenms v3 encrypted auth md5 xxxxxxxxxxxx priv encrypted aes yyyyyyyyyyyy


The Unifi devices gave me the most difficulty. Ubiquti has a few different product lines that are fairly different, thus searching wasnt always the easiest. I saw a few people say they dont really support SNMP or they only supported v1. Unifi device do support full SNMP v3 đŸ˜€ They even have a cute icon in LibreNMS!

LibreNMS Device Setup

First I was in the Unifi controller attempting to set v3 info. As noted above there are several pieces of info you need to be able to enter into LibreNMS to get a SNMP v3 client to work. The Unifi interface was confusing because it just mentioned Username and Password?

Unifi SNMP Screen

I thought this may enable SNMP on the Controller itself to read data for devices, but checking netstat showed no new ports or anything changing. So what was happening when I change this setting? That’s when I noticed my AP was back in Provisioning Mode because a setting just changed. ITS CHANGING THE AP ITSELF! AH!

AP IN Provisioning Mode

The latest Unifi Access Points are just little Linux computers, if you ssh onto them and type “help” you get only a few commands, but a quick double tap of tab shows you have all the normal Linux commands.

BusyBox v1.25.1 () built-in shell (ash)

  ___ ___      .__________.__
 |   |   |____ |__\_  ____/__|
 |   |   /    \|  ||  __) |  |   (c) 2010-2020
 |   |  |   |  \  ||  \   |  |   Ubiquiti Networks, Inc.
 |______|___|  /__||__/   |__|
            |_/                  https://www.ui.com/

      Welcome to UniFi UAP-AC-Mesh!

UBNT-BZ.v4.3.13# cd ..
UBNT-BZ.v4.3.13# more snmp.conf
agentaddress udp:161,tcp:161
rouser DanIsTheBest authpriv

Going back one level I was able to quickly find snmp.conf, which had all the settings I need, and here they are for anyone who needs them.

Unifi APs Use the following for SNMP v3:

  • Port 161 in UDP and TCP
  • Username is the username you made on the Settings screen
  • Password is your password you set with SHA as the hashing method
  • Crypto is the same as your password, in AES mode
  • SNMP v3 auth mode is authpriv

Sophos XG

This one is also fairly straight forward, you need to go to Administration -> Device Access and set which vlans you want to have access to SNMP. Then go to SNMP at the top and setup your general info and then SNMP v3. This is one OS where for passwords they support MD5, SHA-256, and SHA-512; I had to use MD5 since I couldn’t get SHA mode on LibreNMS to connect to either of those newer SHA standards. I will also mention, when you commit your settings in Sophos XG it takes a few minutes for them to take effect. Set everything, then wait at least 2-3 minutes for it to start working, I was jumping around and couldn’t get my settings to take, then waiting a few minutes allowed it all to start working.

Fixing CentOS 6.6 Kickstart Issues

I recently have been working on a system automating CentOS 6 installs for servers. When upgrading to 6.6 my test environment (VMWare Fusion) stopped working. I got a hard kernel panic and halt on loading. Now VMware forums and CentOS site, have posts about work arounds for this. A bunch of them are complex and involve changing modules around, and other files. There is a very easy fix for this, and its detailed below.

NOTE: I am running VMware Fusion, so I will open a package, in Windows and Linux you dont have to do this, just go to the folder.

  1. Stop the VM
  2. Find the VM files
    1. For Fusion there will be a %Your VM%.vmwarevm file, you have to right click that and “Show package contents”
  3. There should be a %Your VM%.vmx file, open that with a text editor
    1. If you are on a Mac, or other machine that likes to do smart quotes, make sure to use a program like vim or Sublime Text that doest add “smart quotes”
  4. A line will read: ethernet0.virtualDev = “e1000e”, change that to ethernet0.virtualDev = “e1000”, just remove the last e. This changes the card from a E1000 in enhanced mode to a normal one. Now CentOS 6.6 will boot.

Here are some place people have discussed issues:


3D Printing Hard Drive Caddies

Sometimes terrible things happen, like when a laptop goes a place where they take the hard drive as it leaves. Then they keep the hard drive bracket because they forgot to take it off! You can get a new drive for ~$50 but where do you get these priceless holders? Well I ran into this situation and decided to 3D print one! Now all IT departments have a real reason to get a 3D printer!

I started a repo at https://github.com/daberkow/3d_harddrive_chassis, and the first laptop I added is a Dell Inspiron N7010, random but it works!

Hard Drive Caddy

Hard Drive Caddy