Projects

CentOS/Rhel 8 Auto login Fix

I have a PXE environment that requires systems to boot up, then automatically login and start a program on boot. All of a sudden this stopped working after years of working. It took me a while to figure it out so figured I would post in case anyone else ran into this.

I have been doing auto login the recommended systemd for a while, as shown: https://wiki.archlinux.org/title/Getty. I copied /lib/systemd/system/getty@.service into /etc/systemd/system/getty@tty1.service. Then with a script edited it using sed in the build pipeline. In the end the line was:

ExecStart=-/usr/bin/agetty --noclear %I $TERM --autologin username

This worked for YEARS, then suddenly stopped. In investigating, I saw another file was being written next to mine at /etc/systemd/system/getty@tty1.servicee ; with another e added to the end of service, making it servicee. After a lot of playing around with it and looking at other guides I figured out, there was a update to systemd/getty and now it cares that all options are before the terminal variable is presented. Changing that line to the following fixed it.

ExecStart=-/usr/bin/agetty --noclear --autologin username %I $TERM 

Booting Dell Precision 7910 and Assassins Creed Valhalla

I recently was setting up a Dell Precision 7910 at work that was in storage for a while. I could not get the system to POST at all. I would give some power on lights, but then sit there and eventually turn off. I searched online and didn’t see any reference to this, so I wanted to make a quick note for the internet. In the system there is a big plastic air guide, that air guide holds the CPU fan. If the cable for the CPU fan gets unplugged, the system will not post.

On a completely unrelated note, I recently got Assassins Creed Valhalla and was enjoying it except every major cut scene or sometimes between areas the game would crash. This was leading to several crashes a hour. No one online seemed to have such a bad time. Long story short, I realized the game was also having issues with cloud saves. It turns out the game frequently attempts to reach out to Ubisoft servers, and instead of failing gracefully, hard crashes. I happen to have a bunch of the Pi-Hole lists in my firewall (Firebog seemed to be the culprit) and the following domains kept being blocked:

*.ubi.com
ubiservices.data.ubi.com

I believe only the “ubiservices.data.ubi.com” domain is the one that needs to be allowed, but to be save I just allow-listed the whole ubi.com domain. Since then the game has not crashed once. Maybe their developers should learn about try/catch blocks…

Homelab: 802.1x 2021

One technology I have played around with a little at work but wanted to get a better handle on is 802.1x. I have taken and passed the Cisco ISE cert a few years back and have used that with other services at work, but for the home setup I mostly wanted to be able to put different wireless devices onto different vlans based on device and user. Windows Server natively makes this possible with Network Policy Server (NPS).

An example of me playing with Network Policy Server

NPS is a Radius server in Windows at the end of the day. It gives you the conditions and a rules system to respond to different Radius calls, as well as a way to setup Accounting. It is fairly simple compared to something like ISE that can also do Posture and Profiling for devices; but for a quick free solution works well for home. You can say of a client is attempting to authenticate over a system like wireless, then accept x methods, vs if its wired or a switch login then accept other forms. Instead of going point by point of how to set it up, which you can find elsewhere online, I want to give some high level edge cases you may run into. First NPS need Windows Server with the Desktop experience, if you are running member servers or domain controllers as Server Core to simplify the environment then it will not work. NPS also does not easily HA. You can run multiple servers with it running, and export the config from one, then import it in another, but there is not a good system for dynamically syncing these (less you call random peoples PowerShell scripts a good system).

Some good reasons to use NPS is the simple AD integration, you can have users use their domain auth and easily get access. Or do as I do, really too much for home, or possibly anywhere, setup a domain CA, have a GPO that creates certs for each machine, then use cert based auth via 802.1x deployed via GPO. If anyone has questions about this I am happy to answer, but there are many places online that will talk about each of those configs and how to do them. Another place to integrate Radius other than 802.1x for wired and wireless is network device login. I use Radius for the stack of Ruckus switches (2 is considered a stack (like when you run k3s as a “cluster” of 1)) I have at home.

This is one of those Windows Services that works well, but also has not been touched in YEARS by Microsoft; like WSUS, or any other service that is useful. To backup this point, I installed several old versions of Windows Server in ESXi that I had laying around. Lesson 1 that I learned, the web console doesn’t work well with some of the legacy mouse support systems. Second you may need legacy VMware tools iso VMware Tools support for Windows 2000, Windows XP, and Windows Server 2003 (81466). The internet seems to say it came out in Server 2008.

https://social.microsoft.com/Forums/getfile/51145/

Building a Pac Man Battle Royale Table

This is a post I was working off and on for several years. It is something I always was meaning to finish, and got very near the end, the Covid and life happen. Instead of throwing it out I figured I would do some small edits then put out as is.

(2017) Friends and I always found the game Pac-Man Battle Royale to be fun, but when we went to look at the price of a cabinet they were $5,000! Worth it if you are a bar or arcade, but for a few friends playing games a bit over the top. After a evening at Barcade, I started the trek to see if I could build one myself for less.

Original Table

The first step was figuring out where I could legally get the game, a number of sites offered the ROM but that is not what I was looking for. Next I found it was ported and available on Steam! We are in luck! The game is available for less than $10! But once I start it, I am greeted by a “fun” border and changes they made to the screen for the Windows version.

Steam Version
Steam Version
Arcade Version
Arcade Version

On the left you can see the Steam version; there is a border, along with all the player text is facing one direction. On the right is the original version, where the screen goes to the edge, and the 3rd and 4th players face the other direction for when players are standing around the table. This version also has been made to play with Xbox 360 wired controllers, it works with keyboard but a lot of the interface seems to be built around that.

The solution, a brave hero on Github made a modified DirectX9 driver that edits the game screen as you play! https://github.com/vikbez/pacbrcade After installing this file, I was able to get the game to look just like the original! Below are the before and afters from the repo, and I can attest it does a great job! Throw in a script to start the game at startup of a Intel Compute Stick (a full pc on a HDMI dongle), and this was ready to go.

(2021) Now it came time to build the cabinet, I hadn’t built something thing big before and didn’t really have a large plan. The hope was to have this live at the office. Being in NYC it needed to be able to fold up, and then be put somewhere when not in use. Part of my plans were to give it folding legs, and one side of the table should be a rest, so the legs can be folded and it can be put on its side somewhere out of the way.

Most of the construction was actually done in one weekend in 2017, I got some 2x4s and went to the maker space I am a member of fat cat FAB LAB – NYC Hackerspace to cut the boards down to the sizes. Now this was a bit of a rough day because I was in NYC, and I am literally grabbing large 2x4s (some are 6 or 7 feet long) then walking a few blocks to the woodshop as people are drinking on a Saturday around me on the streets, then cutting them and walking several blocks to the office. After all is said and done, I used a normal drill and some wood screws to put it all together. I then used yellow vinyl wrap to make the table a bit more dressed up. Getting bubbles out of the vinyl wrapper, on a wood surface that isn’t completely flat was a bit of a challenge. I had a area for the screen to go in, then 4 sides; 2 with cup holders, and 2 with joysticks.

I used Teensy micro controllers as the joysticks, they emulate joysticks on a computer and you an make any input trigger any signal you want. I wanted to add some more style to the unit so I made Player 1 through 4 acrylic panels to go around the joystick and button. These were laser cut at the same woodshop I used before. I also ended up getting craft beer labels, and putting it on the inside of the buttons; that gave the unit a little more character. I put the rubber molding along the side of the unit, as you would expect from any good arcade system. After installing an old screen I found around I toped it with a sheet of clear plexiglass. This was a learning experience of plexiglass scratches easily and can crack if too much pressure is put in say a screw hole.

The whole thing worked, it booted up on the Compute Stick, auto loaded Steam, and started the game. The main issue that had me put it away for a while was the joystick handling. The joysticks I had were 8 way joysticks; they could go to the 4 sides but also to all the corners, we didn’t want the corners. In a game like Pac-Man, going to the corners of the joystick made the character either not move or go in one of the 2 directions you were facing.

The bottom of the joysticks had a plastic piece which allowed the rod of the joystick to go in certain directions. On the bottom of the units I had it was a empty square. I wanted it to be a diamond, this would have forced the player in going one of 4 directions instead. The plan was to design a piece then 3D print it and attach it to the bottom of all the joysticks.

With that I put the system into a closet, where it sat for several years. There was the issue on top of all this of being in NYC and there was no place to put the thing where it would not be in the way. I worked on this before I had a 3D printer and could have put the piece together quickly.

In the end it was fun to put something together quickly like this. I got to do some bigger wood working and vinyl wrap something. One take away I have from it was the momentum of a project cant be a very motivating thing. I started and put most of the system together in a single weekend. Then the last few percent of the project, getting the joysticks correct, I lost the energy (and didn’t have a place to put the thing) and went onto other projects. I try to use that motivation to push through projects when possible, and use this project as a reminder to do that.

Briel Computers Replica 1 Plus

Kit

I recently ordered the Briel Computers Replica I Plus, a Apple I clone. Instead of the originals big board to do a lot of NTSC generation, it uses a more modern single chip. The shipment came in a small box, and with everything I needed. The creator of the kit did a great job including everything you need, down to including an anti-static strap! The project came with some solder, but not nearly enough for everything, I think it was thicker to go with the structural points. Briel Computers sells the kit through ReActiveMicro.com. At $135 it is one of the less expensive kits I have had, but also comes with just the board. If you want a case that needs to be 3D Printed (more on that later).

ReActiveMicro points you over to the Project Wiki for more information, there is a ton there and a link to someone putting the project together. I found this easier to follow along and do than reading the instructions.

The kit was fairly easy and straight forward; I ran into a few small issues around the PS/2 port since the solder points are close together. Getting the few connector ports in can be a bit difficult with a few tiny pins and getting them in the board. As long as you have patience, then you can get through it.

I got it all together, and the board started the first try. I did have the same issue the person who made the video had; I was getting a lot of noise and characters added to the screen. I reflowed a lot of the sockets, and made sure all the socketed chips were fully seated. That cleared up the garbage at startup. The wiki also has some other notes on noise issues the board can show.

I also could not find a PS/2 keyboard in the house, and all the USB keyboards I had didn’t seem to like the USB->PS/2 Adapter. I am not very surprised by this because I didn’t have any very simple, older keyboards.

The USB port that is used for power is also a serial device for a PC/Mac. I plugged into that and got the serial driver working from SparkFun website, they produce the module. Then the output worked well, and I could enter BASIC on the board!

Case

I wanted to put the board in some sort of case, and after searching online I couldn’t find any. I thought I would throw something together quickly that I could put the board in. I took some measurements and threw together a V0 of the case. One small issue was I didn’t account for the RCA jack the video comes out of little let that sticks out. Instead of spending another 7 hours printing a new one, I used a little saw I have to cut a hole out.

Part of my thought of creating a case was to have something I could put the board in, then store it in a cabinet or shelf and not be worried that the board would get damaged. I also made a case that can go over the entire unit to protect it in storage.

Again, looking back small design things could have been changed, like flip the name of the project in the case, so looking at it in the protective cover, the text would be right right way. Getting the scaffolding out of the protective case was not the easiest of things. I designed the protective case with a rail that brings the edge of the mounting board into a locking position when you slide it in. I have to say, that was a nice aspect to the design. It took over 6 hours to print though.

Cisco ISR 4451 Serial Password Recovery

I had to password recover a Cisco ISR 4451, and kept having issues getting into the ROMMON prompt. Every guide mentioned sending a BREAK character during startup, but I could not get that to work. I was using the mini-USB port in the front, and as far as I knew did not have password recovery disabled. It turns out there is a problem with the mini-USB port and the Mac driver, I switched to using a traditional serial cable with a DB-9 connector/RJ45 serial port and suddenly I could get into ROMMON. I wanted to post incase anyone else runs into this.

Below is the startup process, at the end there you should be able to send a BREAK character.

Initializing Hardware ...

System integrity status: 00000610
Rom image verified correctly


System Bootstrap, Version 15.3(3r)S1, RELEASE SOFTWARE
Copyright (c) 1994-2013  by cisco Systems, Inc.

Current image running: Boot ROM0

Last reset cause: PowerOn
Cisco ISR4451-X/K9 platform with 4194304 Kbytes of main memory


Warning: filesystem is not clean
File size is 0x1d482044
Located isr4400-universalk9.03.16.04b.S.155-3.S4b-ext.SPA.bin 
<SEND BREAK HERE>

MisteRdeck MIDI Control Desk

I have been enjoying 3D printing projects recently. I saw a little control board for changing audio levels, and having hotkeys while playing games. The printing took a good long while, and I had to edit some of the parts to work with the parts I found currently on Amazon. I will post the parts list below. The soldering was straight forward, and the project came with a PDF that had good instructions. This also turned into a good opportunity for me to use the new Wiring Pencil, which worked surprisingly well.

For hardware, I am using a Teensy; the Teensy can be a USB keyboard or MIDI device or joystick or serial over the USB connection. The project comes with a premade Arduino file to run it as a MIDI controller. I had not worked before with MIDI input like this, but it seemed the best path forward compared to trying to emulate a keyboard and hitting odd key combinations. Or the alternative of writing something that output serial data then finding, or writing, a daemon for my PC to listen to that device.

For software, I looked at several pieces of software to use the keys and sliders with. I looked at software like VoiceMeeter. While overall that worked, it was very inflexible, and had a giant interface for things I didn’t want to use. Then I found Midi-Mixer, a passion project by a single dev and it is EXACTLY what I needed. The sliders can control single app volume, which is easy to select. And the buttons can be programmed for anything! And easily with a GUI instead of conf files like some other open source projects.

Overall I am enjoying the finished project. It sits next to my keyboard, and allows easy changing of levels while playing games. I added little rubber feet I had laying around so the plastic housing doesn’t slide around on the desk.

To fit the sliders, I needed to modify the knobs, here is the modified versions that work with the sliders I ordered below: MisteRdeck Knob Remix by danberk – Thingiverse.

Make: Makes of MisteRdeck – Arduino-based MIDI Stream Deck by danberk – Thingiverse

Example courtesy of midi-mixer.com

Parts

PartURLPrice
Teensy v3.2Teensy USB Development Board (pjrc.com)$19.80
Gateron Yellow SwitchesGateron Yellow Linear Switches | Kinetic Labs$16.10
Slidershttps://www.amazon.com/gp/product/B079ZP3LS5/$11.99 x 2
Diodeshttps://www.amazon.com/gp/product/B06XB1R2NK/~$6
Key Capshttps://www.amazon.com/gp/product/B01M023NFK/$7.50 x 2

Converting .heic on Windows With Open Source Tools and a Context Menu Shortcut

While taking photos and uploading them places, like this blog, I get the photos in .heic format from the iPhone, then need to convert them into JPEG for WordPress. There are a few paid, and some questionable freeware out there to do it, but I wanted to use open source tools. ImageMagick is an open source tool that can do the conversion, but that requires the command line, so I found the registry keys needed to add a right click context menu to convert the images!

This context menu only shows up when selecting a .heic file as well, which is a nice way to do it. How to install:

  • Install ImageMagick (link), the version I got was “ImageMagick-7.0.11-4-Q16-HDRI-x64-dll.exe”
  • Copy the following lines into a text document
Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\SystemFileAssociations\.heic\Shell\convertojpeg]
@="Convert To JPEG"

[HKEY_CLASSES_ROOT\SystemFileAssociations\.heic\Shell\convertojpeg\command]
@="\"C:\\Windows\\System32\\cmd.exe\" /C magick.exe mogrify -verbose -format jpg \"%1\""

  • Name it install_imagemagick.reg (or really anything.reg)
  • Open that file in file explorer

After install you should be able to right click a .heic photo and “Convert To JPEG”. I did not need to restart/logout/restart explorer. I am calling cmd.exe first instead of the program directly, because this allows you to easily update ImageMagick and not need to directly link to the file.

Using a Custom User-Agent with Google OAuth Client in Java

I have been using the Google OAuth for some of my projects at work for a while. A recent request was to add custom user-agent strings to different apps for the people doing analytics on which apps are using the authentication servers. I have some functions that do custom HTTP Get calls using the Bearer token we get from the OAuth flow, then the library also does its own calls behind the scene. I was able to add a user-agent to my calls easily, but the under the hood ones the library does kept coming up as “Google-HTTP-Java-Client/1.34.2 (gzip)”. I tried a few different ways, and at the same time was searching online, and didn’t see anyone speaking about this. Below is a quick block to put into your app if you want to set the user-agent.

These are the current versions of the OAuth library, and the http client I have been using to do auth.

compile group: 'com.google.oauth-client', name: 'google-oauth-client', version: '1.31.4'
compile group: 'com.google.oauth-client', name: 'google-oauth-client-servlet', version: '1.31.4'
compile group: 'com.google.http-client', name: 'google-http-client', version: '1.39.0'
compile group: 'com.google.http-client', name: 'google-http-client-jackson2', version: '1.39.0'

For my setup, I have the OAuth Servlet that initializes the OAuth flow, then a second servlet which handles the callback; as documented here. I added to the “class OauthCallback extends AbstractAuthorizationCodeCallbackServlet” the following ConnectionFactory under the override for the initializeFlow() function. Replace “myApp-v1.0.1” with your app name. Hope this helps someone!

@Override
protected final AuthorizationCodeFlow initializeFlow() throws IOException {
    ConnectionFactory connectionFactory = url -> {
        HttpURLConnection httpURLConnection = (HttpURLConnection) url.openConnection();
        httpURLConnection.setRequestProperty("user-agent", "myApp-v1.0.1");
        return httpURLConnection;
    };
    return new AuthorizationCodeFlow.Builder(BearerToken.authorizationHeaderAccessMethod(),
            new NetHttpTransport.Builder().setConnectionFactory(connectionFactory).build(),
            new JacksonFactory(),
            .... (code removed);
}

Homelab: NAS 2021

One piece that sits at the heart of my Homelab is the NAS I have. This is actually the same NAS I have written about years ago, looking back on that post brought back memories of the pervious system and Server 2008 that I didn’t recall. In the last year I have added several drives and a new network card to this box, I thought that as well as my experience running FreeNAS, now TrueNAS Core, over 8+ years was worth discussing.

When I built out that box, I had 5x3TB drives, each around $125 dollars. Now those same drives are $40. The rough rule of thumb I was always told is 1GB of RAM for ZFS for every TB of storage you have. So I maxed the mini-itx motherboard out at 16GB of RAM to get as close as I could get. This let met run basic services and I was running a few small VMs/Jails on the box. This did cut into the RAM I had available, but was a nice feature. This allowed me to run the Unifi controller without another system running. Back then, Raspberry Pis came with 256MB of RAM, making it not ideal to run too many services. I later would end up moving all of those to dedicated Raspberry Pis then later VM hosts.

These 5 disks served me well for a while; I every year or two would have a drive die, and it got cheaper and cheaper to replace them. I use this NAS for backups from my Windows desktop, and Macbook. Time Machine backups over the network to Macs works very well with TrueNAS. I ended up getting a smaller version of this box for my parents home, and sister, you can run the OS off a USB with a single or 2 small hard drives in a box like an Intel NUC, then have it always backup their PCs. Reminding people “plug in that USB drive” to backup seems to never stick. TrueNAS offers one click updates, with optional automatic checkin; this makes keeping the system up to date easily.

There have been reports of recent corruption with 12.0, but I have not seen that. Also there was a bug where you could get a banner saying “THIS IS A TESTING RELEASE NOT FOR PRODUCTION” on a production branch, so that is fun. These days those backups, and my Veeam backups are done to the NAS. I tried to use it as a iSCSI and then a NFS target, but the IO was a bit too much for these old spinning drives. Now I use vSAN, as mentioned, which has performed well for VMs, that leaves the NAS just as dumb storage for Veeam. Veeam is a good product that makes it very easy to backup VMs, I will probably write an article on it later. The software has a free 10 VM backup license for Homelabs.

In 2020 I was using a high percentage of the storage for backups and VMs, and was pondering upgrading. I didn’t want to throw down enough money to build a whole new system, and I liked this case a lot, so I started to look at what I could do to add to it. I was using 5 drives, but the case technically supports 7, with 2 being on the bottom. The issue was, I didn’t have enough SATA ports to add to the system. This brings me to one of the scariest, worst, best, cards I have bought. This card, adds 4 ports through a mini PCI-E connection. It actually works really well, with the drives coming up like any other, it gives you 1 PCI E Lane at roughly 2.5Gbp/s for my version. I have 2 drives of the now 7 I have in a RAIDZ2 (RAID 6), and for over a year it has worked well. The one other thing I added to the box was a 10GB networking card, I did a push a bit ago to move most of the Homelab server stuff to 10GB, and this box was part of that. TrueNAS is built on FreeBSD, and has good hardware compatibility, I got an old Intel X520 for compatibility and ease. I have seen it get near 5gbit/s, averaging closer to 2gb/s with writes.

First of all, yes the card is at a slight angle, but it works fine and is secure, so we will ignore that. I also used this time to upgrade the CPU. If you look for 7 year old CPUs on eBay, they are actually not that much money. I went from a Celeron from when I bought the system to a i5-4590. With this new CPU (and breaking a leg on the stock cooler) I ordered a new CPU cooler. That turned into an issue because they sent me the wrong version for an AMD instead of the Intel mount. You can see the very very tiny clearance that the CPU cooler has to the chipset heatsink. I also had this system in the office, since with adding disks to ZFS you need to destroy the pool and rebuild. I had to move all the data off to another system, destroy the array, then move it all back. Dynamically adding disks is always a dream ZFS has had and is always around the corner. Hopefully with OpenZFS 2.0, and the merging of the Linux and Unix code bases, we will get shiny new features like that.

Overall the system has worked well for the last 8 or so years, I have 4TB which is about 30% free still. I could probably clean it out more if I tried. I also have been using OneDrive to backup critical things like family photos, which slightly lowers my need for the system. The homelab AD has all the machines automount a chunk of storage as a shared drive, which makes normal home things and transferring files easier. I will continue to run this, and see how vSAN works for me going forward. I am a bit wary of vSAN running into issues on the consumer level gear I have, so having a whole backup of my VMs on the NAS gives me some peace of mind.

The years of using FreeNAS/TrueNAS were a good jumping off point as we recently got new Netapp Appliances at work, and I was tasked with learning them. Netapp ONTAP uses very similar concepts; instead of zVol you have FlexVol, instead of Datasets you have FlexGroups. Netapp also does some weird things like using Raid-4 or Raid-4 with added protection, instead of a traditional Raid-5/Raid-Z. If you work for a company that has a Netapp and want to learn more about it, I would push you to get the Netapp Simulator. It is a VM image that contains a virtual Netapp to play around with. It’s much better to break a virtual Netapp than a production one.