Quick Blurb

ESXi Migration & Lenovo ThinkCentre M710s

I have started a transition from Hyper-V and Storage Spaces Direct to VMWare vSphere and vSAN. I apologize that these blog posts order is all over the place. Part of the transition is upgrading the hardware on some of the hosts I have, including getting 250GB NVME drives for vSAN cache. I started the migration with one of the desktops that run in the cluster, a Lenovo ThinkCentre M710s. After finding the small slot the NVME drive goes in, I realized there is a manufacture piece of plastic you are supposed to get to install a NVME drive. Since I do not have that, and do not want to pay for it, I spent a good bit more than a hour the first day of the migration creating this bracket and 3D printing it. Then while that was printing, I realized one of the feet on the system had gone missing, so I made a small one of those.

This post is just a quick update and a preview of more to come.

NVME Drive Holder: Lenovo ThinkCentre M710s NVME Bracket by danberk – Thingiverse

Foot: Lenovo ThinkCentre M710s Foot by danberk – Thingiverse

Booting VMware vSphere ESXi 7.0 on Certain Dell Hardware

I recently attempted to boot a Dell Precision M6800 into ESXi 7.0u1 to test some functionality before going to prod. Unfortunately this was met with “Invalid Partition Table”, switching between UEFI and BIOS boot didn’t seem to fix it giving “No boot device available” instead. After searching online I found this, https://communities.vmware.com/t5/ESXi-Discussions/quot-Invalid-Partion-Table-quot-Error-booting-ESXi-7-from-USB/m-p/1823852 which had comments such as “just dont run on a laptop” which was not very helpful.
I spent a chunk of time playing with the partitions and seeing how they were configured. I noticed when I went into the UEFI on the laptop it said it couldn’t find any file systems available, but when I loaded Windows or Linux on the system, the UEFI could see those boot partitions. I tried updating the firmware like Dell recommended, with no change. I then realized the ESXi 7.0 image is FAT16 for the EFI partition, while all other EFI partitions I have seen are FAT32.

I copied the files and folder out of the boot partition, reformatted it with FAT32 instead of FAT16, marked it as EFI type (ESP in Gparted), and moved the files back. The system booted fine the first time, with ESXi running happily. If you need boot ESXi on a Dell M6800, or M4800, or other give that a try. If this worked or didn’t work for you leave a comment below.

3D Printing and Thingiverse Login Fix

I recently got a new 3D printer (Ender 3 Pro), and thought I would put up some of the small things I have recently printed. In trying to print things from Thingiverse, I couldn’t login even after making an account. I would get a spinning “Logging in” and it would never end. After looking at the network log, I saw it trying to reach out to https://accounts.thingiverse.com/unverified?username=danberk If you run into this issue, go to that URL with your username and it will send you an email to verify your account. Then the site will allow you to login.

I have been using Ruckus ICX 7150-12P switches at home recently, I wanted to have it more out of the way; so I designed and printed a mount that would mount the switch to the side. It came out well and looks good! I also printed a network cable comb to hold all the cables nicely together.

This is a nice little SD card, micro SD, and USB holder. I 3M stuck it to the shelf I have, next to the printer.

I have already posted about the Mac SE Battery mount I made. I put the design up if anyone is interested.

Email Alerts on Different Platforms

Different network gear I have has had many problems trying to get email alerts working. I thought I would document them. All of these systems use a service gmail address I made on free/public gmail to send alerts to me.

Sophos, and LibreNMS gave me no problems; if you have issues with them drop a comment below and I can post my settings.

Ruckus AP

The trick to getting Ruckus Unleashed, I used “smtp.gmail.com” and port 587. The issue I ran into is the service email I use to send emails had a long password. Ruckus Unleashed v200.8 supports a maximum of 32 character passwords. I would also mention it dumps the password raw into the logs, so make an account you dont care much about.

Unifi Controller

After digging through logs and getting lots of “There was an error sending the test email to x@gmail.com. Failed to send email for unknown reasons.”, I found one post that mentioned a fix for the console log of “fail to send email: api.err.SmtpSendFailed”. You need to once again use smtp.gmail.com, and port 587, but since its TLS, you need to counter intuitively UNCHECK “Enable SSL”.

Windows Server DNSSEC Error 9110

TL;DR; Check that your Domain Controllers are in the correct OU and that Microsoft Key Distribution Service is running

I ran into an issue recently when DNSSEC signing a dns zone where Windows Server 2019 gave a very vague error, and would only display that error after 10 minutes of timeout. This made iterating on it very slow since every change I made was a 10 minute wait. Every guide to setup DNSSEC mentioned right clicking the zone, then clicking sign and as long as you select the default it should just work. On another domain, that happened for me and it just worked; except the one original one that kept timing out.

In setting a custom DNSSEC signing policy I noticed that there were different keystores each of which gave a different error. This made me think it was something to do with the specific one I was using. It was time to troubleshoot the service itself not DNSSEC.

I got a list of the services from a known good, and signing, domain controller; then compared that to the bad one to see what was different. Part way down the list I noticed that Microsoft Key Distribution Service was failing to start, and if I tried to start it, there was an error.

Group Key Distribution Service cannot connect to the domain controller on local host Status 0x80070020.

Checking the Event Log showed an issue in finding the Domain Controllers on the network (error above), which was weird because it is a Domain Controller… In looking at where this system was placed in the domain tree, I saw it had been moved from the original OU for domain controllers to another place. I dragged it back, after applying all the GPOs that were on that other folder to the original Domain Controller folder. Then held my breath, hit start on the Key Distribution Service and it started right away.

After that DNSSEC signed with no issues. Long story short, dont move your DCs it’ll only end in pain. And to the one other person on the internet who has seen this problem and never solved it, 5+ years ago https://www.reddit.com/r/sysadmin/comments/3dedwm/dnssec_will_not_sign/ there is your answer!

How to use AD users as Admins on Sophos XG v18

As I will be speaking about more on this site soon, I use Sophos XG Home for my homelab (just upgraded to v18). I was attempting to have specific a OU in AD to be able to login and administer the firewall but kept hitting issues. That’s when I found this one support thread, https://community.sophos.com/products/xg-firewall/f/authentication/10879/add-domain-user-account-as-administrator and thought it was worth amplifying.

Setting up AD auth in the product is straight forward, set your domain search as wide as you are comfortable with, because next you import groups that are under that search. Next, make sure to hit the little icon that imports all the AD groups you want, it is easy to overlook.

Now go to the Services tab, and include your new AD servers in your group for Admin Authentication methods. The guides say to make AD first, and in testing I just put one of the servers above local; but this shouldn’t matter too much, local auth still works.

Now here is the trick that got me. TO HAVE THE USER SHOW UP IN THE USER AREA OF AUTHENTICATION, YOU MUST HAVE THEM LOGIN TO THE USER PORTAL FIRST. Thus the User Portal needs to also be setup to allow AD auth. After that, the user will appear like below, and you can click in to edit them.

Clicking into the user you can make them an Admin, and set their group. You have to provide a email at this point for the user. BEWARE, MAKING THE USER AN ADMIN IS NOT REVERSIBLE! IF YOU WANT TO MAKE THEM A NORMAL ACCOUNT AGAIN YOU NEED TO DELETE THE USER, AND IF THIS USER IS USED IN ANY FIREWALL RULE OR SETTINGS THIS WILL BE BLOCKED UNTIL THEY ARE REMOVED FROM ALL OF THEM. One fix for this is to make them part of a Admin group that has no rights to anything, but that doesn’t feel like the proper way.

Error if you try to delete a user tied to policies

Then you should be good to go!

Troubleshooting

Some troubleshooting techniques I used while fixing this: if you don’t have the user imported into Sophos XG, and attempt to login to the Admin panel, you will get “Wrong username/password” and looking at the logs in Sophos you will see “Wrong credentials entered for x@domain”. This is not exactly true and can throw you off. If you login to AD and look at your servers Security logs, it says “User login successful”. That is a good indicator that at least your login is working correctly, don’t get fooled by AD saying success, while Sophos says wrong; the user just needs to login to the User panel first to link the accounts.

Credential Guard on an AMD/Gigabyte system

Recently at work we have been rolling out Credential Guard on our Windows clients. I didn’t know that much about it, so I did some research: https://www.youtube.com/watch?v=urqXgBbVyWY this is a decent video that goes over what Credential Guard does. The high level bits are; it uses Hyper-V to create a secure container that holds your credentials. Then if your main Windows environment is compromised, in theory, the badie cant see your network hash and use it to gain access to stuff. This is just a quick post in case you haven’t heard or dug into a cool new security feature.

There are a few requirements to run Credential Guard, the first is you need Intel or AMD hardware support for virtualization which basically any system in the last 5+ years should easily have. You also have to be running with UEFI and Secure Boot enabled. Both are a good idea anyway, its 2020. This Microsoft page has a PowerShell script you can use to test if your machine is ready and enable bits you need on Windows, https://docs.microsoft.com/en-us/windows/security/identity-protection/credential-guard/credential-guard-manage .

The easiest way to check if its working, or even configured is to type “msinfo32” in the start menu. Then you can see which security tools are running and which are just configured. This is a nice panel because you can easily see if SecureBoot and Credential Guard are working. There are lots of guides on how to get this working, I want to go over some of the caveats to running this.

Caveat 1: Credential Guard breaks Single Sign On for 802.1x connections. This forces you to use certificate auth with User/Machine level certs. https://www.neighborgeek.net/2016/08/windows-10-credential-guard-breaks-wifi.html for more on that.

Caveat 2: Be careful with your motherboard. I have an AMD system I deployed this on, to get SecureBoot working I had to disable CSM (Compatibility Support Module), and after rebooting not only did my keyboard not want to work, but I had to enter my Bitlocker recovery key. That I should have remembered since I made a UEFI change. The keyboard issue seems to be the B350 motherboard in Fast Boot mode has issues with some USB keyboards. After disabling FastBoot that I got it working happily. With an NVME drive, letting the machine fully load each time and not using fast booting only delays the system a couple of seconds, but lets all the devices initialize.

Homelab: Ubiquiti Mesh Link

In my apartment I needed to get wired networking with VLANs across the apartment. I didn’t want to run a wire since I thought my roommate would not appreciate that. I wanted to have a switch near my desk, that allowed different devices I have like file server, desktop, and a few other things to have a wired link; then, connect to the modem/firewall and rest of the networking gear across the apartment.

Long story short, I ended up using a trick I didn’t know would work till I tried it. I have 2 x UAP-AC-M, they work decently well, topping out at 867Mbps and 2×2 MIMO; as well as being able to get them on sale in a 2 pack for a decent price made them a great deal. I have run 1 of them for 4 years as my main access point. Then when I wanted to get this wire connection in a new room configuration I tried to do a wireless uplink to the second one. This makes it mesh with the first access point. Now the important item I don’t seem written anywhere but works well (caveats below):

Ubiquiti access points in wireless uplink/mesh will bridge that network to the wired port on the device

This means if you have a trunk port going into your original/base mesh AP, you will have the same trunk port coming out the other end. This also means anyone who is running mesh points, and hasn’t secured the wired port may want to think about doing so. I am will skip over HOW to set this up, Ubiquiti has a good guide https://help.ui.com/hc/en-us/articles/115002262328 to walk you through it, and most APs can do wireless uplink at this point; this is more about saying it can be done, and works well from my experience to anyone thinking about implementing this or wants a solution for their home/apartment that is not powerline networking. The APs I have are 2×2 802.11AC, I’m sure with a 4×4 AP like the AC-Pro as your base you may see better performance on higher trafficked lines.

This setup has worked well for me for over 6 months now, I can easily hit the 300Mbps I get from my internet connection on a desktop plugged into this meshed AP’s port; I also get 6ms pings to servers while playing games. You get the benefit of real commercial grade antennas and radios in the APs you are using instead of a tiny wifi chip in a laptop, desktop, or device. This also lowers the number of wireless devices (since all the wired devices would have been wireless instead). I also disabled the secondary AP from hosting any of the SSIDs I have in the apartment, so it just works as a wireless uplink. My apartment is not big enough for 2 AP’s for devices.

Caveats

I am looking to move away from this setup for a few reasons. It has worked well and if you are in a pinch I would recommend this setup much more than powerline networking which I have also tried and used several times. I am hoping to move to 10gb/s networking at home with my growing homelab setup; thus, no more wireless link. The other limitation that 99% of people probably would not care about is that you can not do jumbo packets over wireless, so that means it can not be done from all I have read over a wireless link of this type.

Network Topology

The first caveat is that this configuration slightly confuses the access point when it first starts up. The first 60 seconds or so when the access point is online it will think the wired connection is its uplink and attempt to ping out over it. After that it realizes it cant hit anything and will go to wireless uplinking. Sometimes everything just works then, sometimes I have had my switch be confused about where traffic should go and had to power cycle it; in this case it was just a Netgear Prosafe switch with VLANs, not especially smart, but not the dumbest switch. This is similar to a enterprise networks re-converge time when a link is downed. Overall it is rarely a problem and these APs are solid and can go months between restarts, but this is something to lookout for.

Remember that if a Ubiquiti AP cant get an IP, then it doesn’t broadcast SSIDs; this is important since if the base AP boots (like after a power outage) and doesn’t get a DHCP address quick enough, it wont broadcast, then the mesh side will never find an uplink to connect to.

Management

With the earlier mentioned topology issues you can run into, that can make management difficult. You need to make sure the base side of the network is stable. You can get into a position where you did a bad config push or a setting is wrong on the secondary/mesh side and the only way to fix the config is bringing that AP back to the original wired network and pushing a config to it, before the secondary AP can go back into wireless uplink mode.

US Patent US10530642B1

One of the projects I currently work on at work, and have for the last few years is how to go from a blank stack of servers to a fully configured cluster with my companies software running on it. While some projects were starting and getting going in the open source field when I started this project 5+ years ago, a lot of them kept rewriting their API every minor version rev. That started my down a path that has now become a decently large internal network booting infrastructure, and managing interconnects to our inventory system as well as other systems such as Tenable Nessus. I recently was awarded my first patent! This one is specifically about how my system interacts with the inventory to dynamically assign systems as they come online to clusters.

https://patents.google.com/patent/US10530642B1/en?oq=US10530642

My part of the code was all written in Java and continues to evolve as a platform, I hope to open source a good amount of it down the road. I started the project by reading the RFCs for DHCP/PXE and then writing code. I have grown to enjoy writing libraries and some project this way of adhering to the standard (more on that some other time). The general platform can handle ProxyDHCP PXE booting, and then uses iPXE to create menus and boot systems. I spent many hours debugging different vendors PXE code and BIOS vs UEFI to get all the systems to work. The platform now supports plugins for many different aspects of server configuration.

I could write page about small details I have learned a long the way; one issue that has been driving me crazy recently, if you want to ProxyDHCP instead of using your main DHCP stacks these days is Secure Boot. iPXE does not have a Secure Boot signed image, I have tried to get Microsoft to sign it but they will not unless you are selling a product using that the sign iPXE. I am not I just wanted it for internal use. That means you may want to use grub2 as your loader, but there is a bug that has been outstanding for over 6 years and makes ProxyDHCP with grub basically impossible, https://savannah.gnu.org/bugs/?55636 which is sad.

Dell Inspiron 3050 Ram Upgrade

This is a short post about a Dell Inspiron 3050 I upgraded a little bit ago. This is a tiny pc, similar to an Intel NUC. Its a Intel Celeron, and came with a 32GB SSD. I got it for around $150, with an Office 365 subscription; thus it was worth it to me. It came with 2GB of ram, and a 32GB SSD, these days those are not expensive to swap; I wanted to swap the components for 8GB of ram and a 512GB SSD. Below is a short guide with some photos of opening this thing up.

First we needed to remove the case, this involves flipping it over, and taking the 4 screws out that are in the little feet.

That gives you access to the RAM DIMM. Easy to swap if you want to do just that. Now there are 4 screws at the outer corners, those come out then the board can fold out keeping the antenna and other cables connected. Flipping that over and putting on the table shows the CMOS battery, as well as the SSD.

After replacing the SSD its just a matter of flipping the board back onto the posts, and screwing it all back together. Fairly easy to do, but I couldn’t find a ton of photos online so I thought I would put some up. I ended up installed Hyper-V 2019 on it, the box is fairly slow with its Celeron dual core J1800 processor; but can run a Linux VM or two. Plus its a cute little computer that uses very little power.

One last note about putting it back together, there are little metal spokes that stick out from the top metal mount, those need to line up with the motherboard the system wont go back together correctly.