Quick Blurb

Cisco ISR 4451 Serial Password Recovery

I had to password recover a Cisco ISR 4451, and kept having issues getting into the ROMMON prompt. Every guide mentioned sending a BREAK character during startup, but I could not get that to work. I was using the mini-USB port in the front, and as far as I knew did not have password recovery disabled. It turns out there is a problem with the mini-USB port and the Mac driver, I switched to using a traditional serial cable with a DB-9 connector/RJ45 serial port and suddenly I could get into ROMMON. I wanted to post incase anyone else runs into this.

Below is the startup process, at the end there you should be able to send a BREAK character.

Initializing Hardware ...

System integrity status: 00000610
Rom image verified correctly


System Bootstrap, Version 15.3(3r)S1, RELEASE SOFTWARE
Copyright (c) 1994-2013  by cisco Systems, Inc.

Current image running: Boot ROM0

Last reset cause: PowerOn
Cisco ISR4451-X/K9 platform with 4194304 Kbytes of main memory


Warning: filesystem is not clean
File size is 0x1d482044
Located isr4400-universalk9.03.16.04b.S.155-3.S4b-ext.SPA.bin 
<SEND BREAK HERE>

Using a Custom User-Agent with Google OAuth Client in Java

I have been using the Google OAuth for some of my projects at work for a while. A recent request was to add custom user-agent strings to different apps for the people doing analytics on which apps are using the authentication servers. I have some functions that do custom HTTP Get calls using the Bearer token we get from the OAuth flow, then the library also does its own calls behind the scene. I was able to add a user-agent to my calls easily, but the under the hood ones the library does kept coming up as “Google-HTTP-Java-Client/1.34.2 (gzip)”. I tried a few different ways, and at the same time was searching online, and didn’t see anyone speaking about this. Below is a quick block to put into your app if you want to set the user-agent.

These are the current versions of the OAuth library, and the http client I have been using to do auth.

compile group: 'com.google.oauth-client', name: 'google-oauth-client', version: '1.31.4'
compile group: 'com.google.oauth-client', name: 'google-oauth-client-servlet', version: '1.31.4'
compile group: 'com.google.http-client', name: 'google-http-client', version: '1.39.0'
compile group: 'com.google.http-client', name: 'google-http-client-jackson2', version: '1.39.0'

For my setup, I have the OAuth Servlet that initializes the OAuth flow, then a second servlet which handles the callback; as documented here. I added to the “class OauthCallback extends AbstractAuthorizationCodeCallbackServlet” the following ConnectionFactory under the override for the initializeFlow() function. Replace “myApp-v1.0.1” with your app name. Hope this helps someone!

@Override
protected final AuthorizationCodeFlow initializeFlow() throws IOException {
    ConnectionFactory connectionFactory = url -> {
        HttpURLConnection httpURLConnection = (HttpURLConnection) url.openConnection();
        httpURLConnection.setRequestProperty("user-agent", "myApp-v1.0.1");
        return httpURLConnection;
    };
    return new AuthorizationCodeFlow.Builder(BearerToken.authorizationHeaderAccessMethod(),
            new NetHttpTransport.Builder().setConnectionFactory(connectionFactory).build(),
            new JacksonFactory(),
            .... (code removed);
}

ESXi Migration & Lenovo ThinkCentre M710s

I have started a transition from Hyper-V and Storage Spaces Direct to VMWare vSphere and vSAN. I apologize that these blog posts order is all over the place. Part of the transition is upgrading the hardware on some of the hosts I have, including getting 250GB NVME drives for vSAN cache. I started the migration with one of the desktops that run in the cluster, a Lenovo ThinkCentre M710s. After finding the small slot the NVME drive goes in, I realized there is a manufacture piece of plastic you are supposed to get to install a NVME drive. Since I do not have that, and do not want to pay for it, I spent a good bit more than a hour the first day of the migration creating this bracket and 3D printing it. Then while that was printing, I realized one of the feet on the system had gone missing, so I made a small one of those.

This post is just a quick update and a preview of more to come.

NVME Drive Holder: Lenovo ThinkCentre M710s NVME Bracket by danberk – Thingiverse

Foot: Lenovo ThinkCentre M710s Foot by danberk – Thingiverse

Booting VMware vSphere ESXi 7.0 on Certain Dell Hardware

I recently attempted to boot a Dell Precision M6800 into ESXi 7.0u1 to test some functionality before going to prod. Unfortunately this was met with “Invalid Partition Table”, switching between UEFI and BIOS boot didn’t seem to fix it giving “No boot device available” instead. After searching online I found this, https://communities.vmware.com/t5/ESXi-Discussions/quot-Invalid-Partion-Table-quot-Error-booting-ESXi-7-from-USB/m-p/1823852 which had comments such as “just dont run on a laptop” which was not very helpful.
I spent a chunk of time playing with the partitions and seeing how they were configured. I noticed when I went into the UEFI on the laptop it said it couldn’t find any file systems available, but when I loaded Windows or Linux on the system, the UEFI could see those boot partitions. I tried updating the firmware like Dell recommended, with no change. I then realized the ESXi 7.0 image is FAT16 for the EFI partition, while all other EFI partitions I have seen are FAT32.

I copied the files and folder out of the boot partition, reformatted it with FAT32 instead of FAT16, marked it as EFI type (ESP in Gparted), and moved the files back. The system booted fine the first time, with ESXi running happily. If you need boot ESXi on a Dell M6800, or M4800, or other give that a try. If this worked or didn’t work for you leave a comment below.

3D Printing and Thingiverse Login Fix

I recently got a new 3D printer (Ender 3 Pro), and thought I would put up some of the small things I have recently printed. In trying to print things from Thingiverse, I couldn’t login even after making an account. I would get a spinning “Logging in” and it would never end. After looking at the network log, I saw it trying to reach out to https://accounts.thingiverse.com/unverified?username=danberk If you run into this issue, go to that URL with your username and it will send you an email to verify your account. Then the site will allow you to login.

Ruckus ICX 7150 sideways shelf mount

I have been using Ruckus ICX 7150-12P switches at home recently, I wanted to have it more out of the way; so I designed and printed a mount that would mount the switch to the side. It came out well and looks good! I also printed a network cable comb to hold all the cables nicely together.

SD card and USB holder!

This is a nice little SD card, micro SD, and USB holder. I 3M stuck it to the shelf I have, next to the printer.

Mac SE with battery mount

I have already posted about the Mac SE Battery mount I made. I put the design up if anyone is interested.

Email Alerts on Different Platforms

Different network gear I have has had many problems trying to get email alerts working. I thought I would document them. All of these systems use a service gmail address I made on free/public gmail to send alerts to me.

Sophos, and LibreNMS gave me no problems; if you have issues with them drop a comment below and I can post my settings.

Ruckus AP

The trick to getting Ruckus Unleashed, I used “smtp.gmail.com” and port 587. The issue I ran into is the service email I use to send emails had a long password. Ruckus Unleashed v200.8 supports a maximum of 32 character passwords. I would also mention it dumps the password raw into the logs, so make an account you dont care much about.

Unifi Controller

After digging through logs and getting lots of “There was an error sending the test email to x@gmail.com. Failed to send email for unknown reasons.”, I found one post that mentioned a fix for the console log of “fail to send email: api.err.SmtpSendFailed”. You need to once again use smtp.gmail.com, and port 587, but since its TLS, you need to counter intuitively UNCHECK “Enable SSL”.

Windows Server DNSSEC Error 9110

TL;DR; Check that your Domain Controllers are in the correct OU and that Microsoft Key Distribution Service is running

I ran into an issue recently when DNSSEC signing a dns zone where Windows Server 2019 gave a very vague error, and would only display that error after 10 minutes of timeout. This made iterating on it very slow since every change I made was a 10 minute wait. Every guide to setup DNSSEC mentioned right clicking the zone, then clicking sign and as long as you select the default it should just work. On another domain, that happened for me and it just worked; except the one original one that kept timing out.

In setting a custom DNSSEC signing policy I noticed that there were different keystores each of which gave a different error. This made me think it was something to do with the specific one I was using. It was time to troubleshoot the service itself not DNSSEC.

I got a list of the services from a known good, and signing, domain controller; then compared that to the bad one to see what was different. Part way down the list I noticed that Microsoft Key Distribution Service was failing to start, and if I tried to start it, there was an error.

Group Key Distribution Service cannot connect to the domain controller on local host Status 0x80070020.

Checking the Event Log showed an issue in finding the Domain Controllers on the network (error above), which was weird because it is a Domain Controller… In looking at where this system was placed in the domain tree, I saw it had been moved from the original OU for domain controllers to another place. I dragged it back, after applying all the GPOs that were on that other folder to the original Domain Controller folder. Then held my breath, hit start on the Key Distribution Service and it started right away.

After that DNSSEC signed with no issues. Long story short, dont move your DCs it’ll only end in pain. And to the one other person on the internet who has seen this problem and never solved it, 5+ years ago https://www.reddit.com/r/sysadmin/comments/3dedwm/dnssec_will_not_sign/ there is your answer!

How to use AD users as Admins on Sophos XG v18

As I will be speaking about more on this site soon, I use Sophos XG Home for my homelab (just upgraded to v18). I was attempting to have specific a OU in AD to be able to login and administer the firewall but kept hitting issues. That’s when I found this one support thread, https://community.sophos.com/products/xg-firewall/f/authentication/10879/add-domain-user-account-as-administrator and thought it was worth amplifying.

Setting up AD auth in the product is straight forward, set your domain search as wide as you are comfortable with, because next you import groups that are under that search. Next, make sure to hit the little icon that imports all the AD groups you want, it is easy to overlook.

Import groups button

Now go to the Services tab, and include your new AD servers in your group for Admin Authentication methods. The guides say to make AD first, and in testing I just put one of the servers above local; but this shouldn’t matter too much, local auth still works.

Admin Authentication Methods

Now here is the trick that got me. TO HAVE THE USER SHOW UP IN THE USER AREA OF AUTHENTICATION, YOU MUST HAVE THEM LOGIN TO THE USER PORTAL FIRST. Thus the User Portal needs to also be setup to allow AD auth. After that, the user will appear like below, and you can click in to edit them.

User admin panel

Clicking into the user you can make them an Admin, and set their group. You have to provide a email at this point for the user. BEWARE, MAKING THE USER AN ADMIN IS NOT REVERSIBLE! IF YOU WANT TO MAKE THEM A NORMAL ACCOUNT AGAIN YOU NEED TO DELETE THE USER, AND IF THIS USER IS USED IN ANY FIREWALL RULE OR SETTINGS THIS WILL BE BLOCKED UNTIL THEY ARE REMOVED FROM ALL OF THEM. One fix for this is to make them part of a Admin group that has no rights to anything, but that doesn’t feel like the proper way.

User panel making a user an admin
Error if you try to delete a user tied to policies

Then you should be good to go!

Troubleshooting

Some troubleshooting techniques I used while fixing this: if you don’t have the user imported into Sophos XG, and attempt to login to the Admin panel, you will get “Wrong username/password” and looking at the logs in Sophos you will see “Wrong credentials entered for x@domain”. This is not exactly true and can throw you off. If you login to AD and look at your servers Security logs, it says “User login successful”. That is a good indicator that at least your login is working correctly, don’t get fooled by AD saying success, while Sophos says wrong; the user just needs to login to the User panel first to link the accounts.

Credential Guard on an AMD/Gigabyte system

Recently at work we have been rolling out Credential Guard on our Windows clients. I didn’t know that much about it, so I did some research: https://www.youtube.com/watch?v=urqXgBbVyWY this is a decent video that goes over what Credential Guard does. The high level bits are; it uses Hyper-V to create a secure container that holds your credentials. Then if your main Windows environment is compromised, in theory, the badie cant see your network hash and use it to gain access to stuff. This is just a quick post in case you haven’t heard or dug into a cool new security feature.

There are a few requirements to run Credential Guard, the first is you need Intel or AMD hardware support for virtualization which basically any system in the last 5+ years should easily have. You also have to be running with UEFI and Secure Boot enabled. Both are a good idea anyway, its 2020. This Microsoft page has a PowerShell script you can use to test if your machine is ready and enable bits you need on Windows, https://docs.microsoft.com/en-us/windows/security/identity-protection/credential-guard/credential-guard-manage .

The easiest way to check if its working, or even configured is to type “msinfo32” in the start menu. Then you can see which security tools are running and which are just configured. This is a nice panel because you can easily see if SecureBoot and Credential Guard are working. There are lots of guides on how to get this working, I want to go over some of the caveats to running this.

Caveat 1: Credential Guard breaks Single Sign On for 802.1x connections. This forces you to use certificate auth with User/Machine level certs. https://www.neighborgeek.net/2016/08/windows-10-credential-guard-breaks-wifi.html for more on that.

Caveat 2: Be careful with your motherboard. I have an AMD system I deployed this on, to get SecureBoot working I had to disable CSM (Compatibility Support Module), and after rebooting not only did my keyboard not want to work, but I had to enter my Bitlocker recovery key. That I should have remembered since I made a UEFI change. The keyboard issue seems to be the B350 motherboard in Fast Boot mode has issues with some USB keyboards. After disabling FastBoot that I got it working happily. With an NVME drive, letting the machine fully load each time and not using fast booting only delays the system a couple of seconds, but lets all the devices initialize.

Homelab: Ubiquiti Mesh Link

In my apartment I needed to get wired networking with VLANs across the apartment. I didn’t want to run a wire since I thought my roommate would not appreciate that. I wanted to have a switch near my desk, that allowed different devices I have like file server, desktop, and a few other things to have a wired link; then, connect to the modem/firewall and rest of the networking gear across the apartment.

Long story short, I ended up using a trick I didn’t know would work till I tried it. I have 2 x UAP-AC-M, they work decently well, topping out at 867Mbps and 2×2 MIMO; as well as being able to get them on sale in a 2 pack for a decent price made them a great deal. I have run 1 of them for 4 years as my main access point. Then when I wanted to get this wire connection in a new room configuration I tried to do a wireless uplink to the second one. This makes it mesh with the first access point. Now the important item I don’t seem written anywhere but works well (caveats below):

Ubiquiti access points in wireless uplink/mesh will bridge that network to the wired port on the device

This means if you have a trunk port going into your original/base mesh AP, you will have the same trunk port coming out the other end. This also means anyone who is running mesh points, and hasn’t secured the wired port may want to think about doing so. I am will skip over HOW to set this up, Ubiquiti has a good guide https://help.ui.com/hc/en-us/articles/115002262328 to walk you through it, and most APs can do wireless uplink at this point; this is more about saying it can be done, and works well from my experience to anyone thinking about implementing this or wants a solution for their home/apartment that is not powerline networking. The APs I have are 2×2 802.11AC, I’m sure with a 4×4 AP like the AC-Pro as your base you may see better performance on higher trafficked lines.

This setup has worked well for me for over 6 months now, I can easily hit the 300Mbps I get from my internet connection on a desktop plugged into this meshed AP’s port; I also get 6ms pings to servers while playing games. You get the benefit of real commercial grade antennas and radios in the APs you are using instead of a tiny wifi chip in a laptop, desktop, or device. This also lowers the number of wireless devices (since all the wired devices would have been wireless instead). I also disabled the secondary AP from hosting any of the SSIDs I have in the apartment, so it just works as a wireless uplink. My apartment is not big enough for 2 AP’s for devices.

Caveats

I am looking to move away from this setup for a few reasons. It has worked well and if you are in a pinch I would recommend this setup much more than powerline networking which I have also tried and used several times. I am hoping to move to 10gb/s networking at home with my growing homelab setup; thus, no more wireless link. The other limitation that 99% of people probably would not care about is that you can not do jumbo packets over wireless, so that means it can not be done from all I have read over a wireless link of this type.

Network Topology

The first caveat is that this configuration slightly confuses the access point when it first starts up. The first 60 seconds or so when the access point is online it will think the wired connection is its uplink and attempt to ping out over it. After that it realizes it cant hit anything and will go to wireless uplinking. Sometimes everything just works then, sometimes I have had my switch be confused about where traffic should go and had to power cycle it; in this case it was just a Netgear Prosafe switch with VLANs, not especially smart, but not the dumbest switch. This is similar to a enterprise networks re-converge time when a link is downed. Overall it is rarely a problem and these APs are solid and can go months between restarts, but this is something to lookout for.

Remember that if a Ubiquiti AP cant get an IP, then it doesn’t broadcast SSIDs; this is important since if the base AP boots (like after a power outage) and doesn’t get a DHCP address quick enough, it wont broadcast, then the mesh side will never find an uplink to connect to.

Management

With the earlier mentioned topology issues you can run into, that can make management difficult. You need to make sure the base side of the network is stable. You can get into a position where you did a bad config push or a setting is wrong on the secondary/mesh side and the only way to fix the config is bringing that AP back to the original wired network and pushing a config to it, before the secondary AP can go back into wireless uplink mode.