Building a Pac Man Battle Royale Table

This is a post I was working off and on for several years. It is something I always was meaning to finish, and got very near the end, the Covid and life happen. Instead of throwing it out I figured I would do some small edits then put out as is.

(2017) Friends and I always found the game Pac-Man Battle Royale to be fun, but when we went to look at the price of a cabinet they were $5,000! Worth it if you are a bar or arcade, but for a few friends playing games a bit over the top. After a evening at Barcade, I started the trek to see if I could build one myself for less.

Original Table

The first step was figuring out where I could legally get the game, a number of sites offered the ROM but that is not what I was looking for. Next I found it was ported and available on Steam! We are in luck! The game is available for less than $10! But once I start it, I am greeted by a “fun” border and changes they made to the screen for the Windows version.

Steam Version
Steam Version
Arcade Version
Arcade Version

On the left you can see the Steam version; there is a border, along with all the player text is facing one direction. On the right is the original version, where the screen goes to the edge, and the 3rd and 4th players face the other direction for when players are standing around the table. This version also has been made to play with Xbox 360 wired controllers, it works with keyboard but a lot of the interface seems to be built around that.

The solution, a brave hero on Github made a modified DirectX9 driver that edits the game screen as you play! https://github.com/vikbez/pacbrcade After installing this file, I was able to get the game to look just like the original! Below are the before and afters from the repo, and I can attest it does a great job! Throw in a script to start the game at startup of a Intel Compute Stick (a full pc on a HDMI dongle), and this was ready to go.

(2021) Now it came time to build the cabinet, I hadn’t built something thing big before and didn’t really have a large plan. The hope was to have this live at the office. Being in NYC it needed to be able to fold up, and then be put somewhere when not in use. Part of my plans were to give it folding legs, and one side of the table should be a rest, so the legs can be folded and it can be put on its side somewhere out of the way.

Most of the construction was actually done in one weekend in 2017, I got some 2x4s and went to the maker space I am a member of fat cat FAB LAB – NYC Hackerspace to cut the boards down to the sizes. Now this was a bit of a rough day because I was in NYC, and I am literally grabbing large 2x4s (some are 6 or 7 feet long) then walking a few blocks to the woodshop as people are drinking on a Saturday around me on the streets, then cutting them and walking several blocks to the office. After all is said and done, I used a normal drill and some wood screws to put it all together. I then used yellow vinyl wrap to make the table a bit more dressed up. Getting bubbles out of the vinyl wrapper, on a wood surface that isn’t completely flat was a bit of a challenge. I had a area for the screen to go in, then 4 sides; 2 with cup holders, and 2 with joysticks.

I used Teensy micro controllers as the joysticks, they emulate joysticks on a computer and you an make any input trigger any signal you want. I wanted to add some more style to the unit so I made Player 1 through 4 acrylic panels to go around the joystick and button. These were laser cut at the same woodshop I used before. I also ended up getting craft beer labels, and putting it on the inside of the buttons; that gave the unit a little more character. I put the rubber molding along the side of the unit, as you would expect from any good arcade system. After installing an old screen I found around I toped it with a sheet of clear plexiglass. This was a learning experience of plexiglass scratches easily and can crack if too much pressure is put in say a screw hole.

The whole thing worked, it booted up on the Compute Stick, auto loaded Steam, and started the game. The main issue that had me put it away for a while was the joystick handling. The joysticks I had were 8 way joysticks; they could go to the 4 sides but also to all the corners, we didn’t want the corners. In a game like Pac-Man, going to the corners of the joystick made the character either not move or go in one of the 2 directions you were facing.

The bottom of the joysticks had a plastic piece which allowed the rod of the joystick to go in certain directions. On the bottom of the units I had it was a empty square. I wanted it to be a diamond, this would have forced the player in going one of 4 directions instead. The plan was to design a piece then 3D print it and attach it to the bottom of all the joysticks.

With that I put the system into a closet, where it sat for several years. There was the issue on top of all this of being in NYC and there was no place to put the thing where it would not be in the way. I worked on this before I had a 3D printer and could have put the piece together quickly.

In the end it was fun to put something together quickly like this. I got to do some bigger wood working and vinyl wrap something. One take away I have from it was the momentum of a project cant be a very motivating thing. I started and put most of the system together in a single weekend. Then the last few percent of the project, getting the joysticks correct, I lost the energy (and didn’t have a place to put the thing) and went onto other projects. I try to use that motivation to push through projects when possible, and use this project as a reminder to do that.

Briel Computers Replica 1 Plus

Kit

I recently ordered the Briel Computers Replica I Plus, a Apple I clone. Instead of the originals big board to do a lot of NTSC generation, it uses a more modern single chip. The shipment came in a small box, and with everything I needed. The creator of the kit did a great job including everything you need, down to including an anti-static strap! The project came with some solder, but not nearly enough for everything, I think it was thicker to go with the structural points. Briel Computers sells the kit through ReActiveMicro.com. At $135 it is one of the less expensive kits I have had, but also comes with just the board. If you want a case that needs to be 3D Printed (more on that later).

ReActiveMicro points you over to the Project Wiki for more information, there is a ton there and a link to someone putting the project together. I found this easier to follow along and do than reading the instructions.

The kit was fairly easy and straight forward; I ran into a few small issues around the PS/2 port since the solder points are close together. Getting the few connector ports in can be a bit difficult with a few tiny pins and getting them in the board. As long as you have patience, then you can get through it.

I got it all together, and the board started the first try. I did have the same issue the person who made the video had; I was getting a lot of noise and characters added to the screen. I reflowed a lot of the sockets, and made sure all the socketed chips were fully seated. That cleared up the garbage at startup. The wiki also has some other notes on noise issues the board can show.

I also could not find a PS/2 keyboard in the house, and all the USB keyboards I had didn’t seem to like the USB->PS/2 Adapter. I am not very surprised by this because I didn’t have any very simple, older keyboards.

The USB port that is used for power is also a serial device for a PC/Mac. I plugged into that and got the serial driver working from SparkFun website, they produce the module. Then the output worked well, and I could enter BASIC on the board!

Case

I wanted to put the board in some sort of case, and after searching online I couldn’t find any. I thought I would throw something together quickly that I could put the board in. I took some measurements and threw together a V0 of the case. One small issue was I didn’t account for the RCA jack the video comes out of little let that sticks out. Instead of spending another 7 hours printing a new one, I used a little saw I have to cut a hole out.

Part of my thought of creating a case was to have something I could put the board in, then store it in a cabinet or shelf and not be worried that the board would get damaged. I also made a case that can go over the entire unit to protect it in storage.

Again, looking back small design things could have been changed, like flip the name of the project in the case, so looking at it in the protective cover, the text would be right right way. Getting the scaffolding out of the protective case was not the easiest of things. I designed the protective case with a rail that brings the edge of the mounting board into a locking position when you slide it in. I have to say, that was a nice aspect to the design. It took over 6 hours to print though.

Cisco ISR 4451 Serial Password Recovery

I had to password recover a Cisco ISR 4451, and kept having issues getting into the ROMMON prompt. Every guide mentioned sending a BREAK character during startup, but I could not get that to work. I was using the mini-USB port in the front, and as far as I knew did not have password recovery disabled. It turns out there is a problem with the mini-USB port and the Mac driver, I switched to using a traditional serial cable with a DB-9 connector/RJ45 serial port and suddenly I could get into ROMMON. I wanted to post incase anyone else runs into this.

Below is the startup process, at the end there you should be able to send a BREAK character.

Initializing Hardware ...

System integrity status: 00000610
Rom image verified correctly


System Bootstrap, Version 15.3(3r)S1, RELEASE SOFTWARE
Copyright (c) 1994-2013  by cisco Systems, Inc.

Current image running: Boot ROM0

Last reset cause: PowerOn
Cisco ISR4451-X/K9 platform with 4194304 Kbytes of main memory


Warning: filesystem is not clean
File size is 0x1d482044
Located isr4400-universalk9.03.16.04b.S.155-3.S4b-ext.SPA.bin 
<SEND BREAK HERE>

MisteRdeck MIDI Control Desk

I have been enjoying 3D printing projects recently. I saw a little control board for changing audio levels, and having hotkeys while playing games. The printing took a good long while, and I had to edit some of the parts to work with the parts I found currently on Amazon. I will post the parts list below. The soldering was straight forward, and the project came with a PDF that had good instructions. This also turned into a good opportunity for me to use the new Wiring Pencil, which worked surprisingly well.

For hardware, I am using a Teensy; the Teensy can be a USB keyboard or MIDI device or joystick or serial over the USB connection. The project comes with a premade Arduino file to run it as a MIDI controller. I had not worked before with MIDI input like this, but it seemed the best path forward compared to trying to emulate a keyboard and hitting odd key combinations. Or the alternative of writing something that output serial data then finding, or writing, a daemon for my PC to listen to that device.

For software, I looked at several pieces of software to use the keys and sliders with. I looked at software like VoiceMeeter. While overall that worked, it was very inflexible, and had a giant interface for things I didn’t want to use. Then I found Midi-Mixer, a passion project by a single dev and it is EXACTLY what I needed. The sliders can control single app volume, which is easy to select. And the buttons can be programmed for anything! And easily with a GUI instead of conf files like some other open source projects.

Overall I am enjoying the finished project. It sits next to my keyboard, and allows easy changing of levels while playing games. I added little rubber feet I had laying around so the plastic housing doesn’t slide around on the desk.

To fit the sliders, I needed to modify the knobs, here is the modified versions that work with the sliders I ordered below: MisteRdeck Knob Remix by danberk – Thingiverse.

Make: Makes of MisteRdeck – Arduino-based MIDI Stream Deck by danberk – Thingiverse

Example courtesy of midi-mixer.com

Parts

PartURLPrice
Teensy v3.2Teensy USB Development Board (pjrc.com)$19.80
Gateron Yellow SwitchesGateron Yellow Linear Switches | Kinetic Labs$16.10
Slidershttps://www.amazon.com/gp/product/B079ZP3LS5/$11.99 x 2
Diodeshttps://www.amazon.com/gp/product/B06XB1R2NK/~$6
Key Capshttps://www.amazon.com/gp/product/B01M023NFK/$7.50 x 2

Converting .heic on Windows With Open Source Tools and a Context Menu Shortcut

While taking photos and uploading them places, like this blog, I get the photos in .heic format from the iPhone, then need to convert them into JPEG for WordPress. There are a few paid, and some questionable freeware out there to do it, but I wanted to use open source tools. ImageMagick is an open source tool that can do the conversion, but that requires the command line, so I found the registry keys needed to add a right click context menu to convert the images!

This context menu only shows up when selecting a .heic file as well, which is a nice way to do it. How to install:

  • Install ImageMagick (link), the version I got was “ImageMagick-7.0.11-4-Q16-HDRI-x64-dll.exe”
  • Copy the following lines into a text document
Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\SystemFileAssociations\.heic\Shell\convertojpeg]
@="Convert To JPEG"

[HKEY_CLASSES_ROOT\SystemFileAssociations\.heic\Shell\convertojpeg\command]
@="\"C:\\Windows\\System32\\cmd.exe\" /C magick.exe mogrify -verbose -format jpg \"%1\""

  • Name it install_imagemagick.reg (or really anything.reg)
  • Open that file in file explorer

After install you should be able to right click a .heic photo and “Convert To JPEG”. I did not need to restart/logout/restart explorer. I am calling cmd.exe first instead of the program directly, because this allows you to easily update ImageMagick and not need to directly link to the file.

Using a Custom User-Agent with Google OAuth Client in Java

I have been using the Google OAuth for some of my projects at work for a while. A recent request was to add custom user-agent strings to different apps for the people doing analytics on which apps are using the authentication servers. I have some functions that do custom HTTP Get calls using the Bearer token we get from the OAuth flow, then the library also does its own calls behind the scene. I was able to add a user-agent to my calls easily, but the under the hood ones the library does kept coming up as “Google-HTTP-Java-Client/1.34.2 (gzip)”. I tried a few different ways, and at the same time was searching online, and didn’t see anyone speaking about this. Below is a quick block to put into your app if you want to set the user-agent.

These are the current versions of the OAuth library, and the http client I have been using to do auth.

compile group: 'com.google.oauth-client', name: 'google-oauth-client', version: '1.31.4'
compile group: 'com.google.oauth-client', name: 'google-oauth-client-servlet', version: '1.31.4'
compile group: 'com.google.http-client', name: 'google-http-client', version: '1.39.0'
compile group: 'com.google.http-client', name: 'google-http-client-jackson2', version: '1.39.0'

For my setup, I have the OAuth Servlet that initializes the OAuth flow, then a second servlet which handles the callback; as documented here. I added to the “class OauthCallback extends AbstractAuthorizationCodeCallbackServlet” the following ConnectionFactory under the override for the initializeFlow() function. Replace “myApp-v1.0.1” with your app name. Hope this helps someone!

@Override
protected final AuthorizationCodeFlow initializeFlow() throws IOException {
    ConnectionFactory connectionFactory = url -> {
        HttpURLConnection httpURLConnection = (HttpURLConnection) url.openConnection();
        httpURLConnection.setRequestProperty("user-agent", "myApp-v1.0.1");
        return httpURLConnection;
    };
    return new AuthorizationCodeFlow.Builder(BearerToken.authorizationHeaderAccessMethod(),
            new NetHttpTransport.Builder().setConnectionFactory(connectionFactory).build(),
            new JacksonFactory(),
            .... (code removed);
}

Homelab: NAS 2021

One piece that sits at the heart of my Homelab is the NAS I have. This is actually the same NAS I have written about years ago, looking back on that post brought back memories of the pervious system and Server 2008 that I didn’t recall. In the last year I have added several drives and a new network card to this box, I thought that as well as my experience running FreeNAS, now TrueNAS Core, over 8+ years was worth discussing.

When I built out that box, I had 5x3TB drives, each around $125 dollars. Now those same drives are $40. The rough rule of thumb I was always told is 1GB of RAM for ZFS for every TB of storage you have. So I maxed the mini-itx motherboard out at 16GB of RAM to get as close as I could get. This let met run basic services and I was running a few small VMs/Jails on the box. This did cut into the RAM I had available, but was a nice feature. This allowed me to run the Unifi controller without another system running. Back then, Raspberry Pis came with 256MB of RAM, making it not ideal to run too many services. I later would end up moving all of those to dedicated Raspberry Pis then later VM hosts.

These 5 disks served me well for a while; I every year or two would have a drive die, and it got cheaper and cheaper to replace them. I use this NAS for backups from my Windows desktop, and Macbook. Time Machine backups over the network to Macs works very well with TrueNAS. I ended up getting a smaller version of this box for my parents home, and sister, you can run the OS off a USB with a single or 2 small hard drives in a box like an Intel NUC, then have it always backup their PCs. Reminding people “plug in that USB drive” to backup seems to never stick. TrueNAS offers one click updates, with optional automatic checkin; this makes keeping the system up to date easily.

There have been reports of recent corruption with 12.0, but I have not seen that. Also there was a bug where you could get a banner saying “THIS IS A TESTING RELEASE NOT FOR PRODUCTION” on a production branch, so that is fun. These days those backups, and my Veeam backups are done to the NAS. I tried to use it as a iSCSI and then a NFS target, but the IO was a bit too much for these old spinning drives. Now I use vSAN, as mentioned, which has performed well for VMs, that leaves the NAS just as dumb storage for Veeam. Veeam is a good product that makes it very easy to backup VMs, I will probably write an article on it later. The software has a free 10 VM backup license for Homelabs.

In 2020 I was using a high percentage of the storage for backups and VMs, and was pondering upgrading. I didn’t want to throw down enough money to build a whole new system, and I liked this case a lot, so I started to look at what I could do to add to it. I was using 5 drives, but the case technically supports 7, with 2 being on the bottom. The issue was, I didn’t have enough SATA ports to add to the system. This brings me to one of the scariest, worst, best, cards I have bought. This card, adds 4 ports through a mini PCI-E connection. It actually works really well, with the drives coming up like any other, it gives you 1 PCI E Lane at roughly 2.5Gbp/s for my version. I have 2 drives of the now 7 I have in a RAIDZ2 (RAID 6), and for over a year it has worked well. The one other thing I added to the box was a 10GB networking card, I did a push a bit ago to move most of the Homelab server stuff to 10GB, and this box was part of that. TrueNAS is built on FreeBSD, and has good hardware compatibility, I got an old Intel X520 for compatibility and ease. I have seen it get near 5gbit/s, averaging closer to 2gb/s with writes.

First of all, yes the card is at a slight angle, but it works fine and is secure, so we will ignore that. I also used this time to upgrade the CPU. If you look for 7 year old CPUs on eBay, they are actually not that much money. I went from a Celeron from when I bought the system to a i5-4590. With this new CPU (and breaking a leg on the stock cooler) I ordered a new CPU cooler. That turned into an issue because they sent me the wrong version for an AMD instead of the Intel mount. You can see the very very tiny clearance that the CPU cooler has to the chipset heatsink. I also had this system in the office, since with adding disks to ZFS you need to destroy the pool and rebuild. I had to move all the data off to another system, destroy the array, then move it all back. Dynamically adding disks is always a dream ZFS has had and is always around the corner. Hopefully with OpenZFS 2.0, and the merging of the Linux and Unix code bases, we will get shiny new features like that.

Overall the system has worked well for the last 8 or so years, I have 4TB which is about 30% free still. I could probably clean it out more if I tried. I also have been using OneDrive to backup critical things like family photos, which slightly lowers my need for the system. The homelab AD has all the machines automount a chunk of storage as a shared drive, which makes normal home things and transferring files easier. I will continue to run this, and see how vSAN works for me going forward. I am a bit wary of vSAN running into issues on the consumer level gear I have, so having a whole backup of my VMs on the NAS gives me some peace of mind.

The years of using FreeNAS/TrueNAS were a good jumping off point as we recently got new Netapp Appliances at work, and I was tasked with learning them. Netapp ONTAP uses very similar concepts; instead of zVol you have FlexVol, instead of Datasets you have FlexGroups. Netapp also does some weird things like using Raid-4 or Raid-4 with added protection, instead of a traditional Raid-5/Raid-Z. If you work for a company that has a Netapp and want to learn more about it, I would push you to get the Netapp Simulator. It is a VM image that contains a virtual Netapp to play around with. It’s much better to break a virtual Netapp than a production one.

Mister Project Keyboard Case

Over the holidays I got parts to put together a Mister FPGA system (project home, sub-reddit). This is an open source project which allows to run classic game consoles and classic computers in hardware on the FPGA. Instead of normal emulation, where in software you pretend to be the CPU/GPU/Hardware of what the original code would run on, this projects has a Field Programable Gate Array that can change itself into being that hardware. By doing this, the system can get very very close to 100% accurate running of these old systems. Each system is created into a “core” which is applied to the FPGA to run software. The community around the Mister Project is growing, there are some projects to get systems like N64, and PSX working on this platform; but the Mister Project standardized a while ago on one FPGA, which may not be up to that task once the new cores are done because of their size and complexity.

There are many nice features that have been built out for the projects over the years. Standardizing around the DE10-Nano FPGA, there are many add-on boards you can get for it. From additional RAM, to VGA outputs. The FPGA has a ARM CPU that manages the base system, that supports Wi-Fi cards, Bluetooth, and has automatic updating features. With an IO board that most people who use the project get, you can click a button to reboot the system, or another to go back to the main menu and select the core you want to run. I have a standard IO board, USB Hub, and 256MB of RAM addon. The documentation for the open source project is actually good, with it all centering around the Github Wiki. There are automatic installers for the SD card you need to do the initial ARM side setup.

I was most interested in one of the completed cores, it is a 486DX (project home) with Sound Blaster, and everything you need to run Dos/Windows 3.1/Win 95. Having played many games as a child in that environment, having a 386SX, I was excited to give it a try.

Hardware

When I was thinking of getting the parts for the project, I looked on Thingiverse to see if anyone had put a case up; there are several. The one that caught my eye had an embedded keyboard in it (link, updated case), that one had a note on it that an update to the case was coming soon, and to hold off on printing. The estimate for printing the case was around 24 hours, and I didn’t want to do it twice, so I waited. I reached out to the creator who worked away over the holiday season to get the update out. Myself and another were chatting with him in the comments about printing it, and the creator graciously put up the design, before all the instructions were done so the two of us could start printing.

USB Board, with input against the case

This is the largest thing I have printed on the printer, with my print bed holding up to 220mm, and the case coming in at ~210mm. It printed great. I used PETG instead of PLA plastic to have added resistance to heat. After that, it was screwing parts together, and making a tiny circuit board to support the normal buttons on the top of the case. I ran into a small problem with the updated USB board I have, its input was blocked by the side of the case. The creator had a different revision of the USB board, and thus hadn’t tested with my version. I ordered some cables online and ended up checking the pinouts and making my own header to USB cable, after that it was smooth sailing.

I ordered a collection of M3 screws, to have different sizes. That is the size the case was built around. I also had some screws that do not have heads on them, I was able to use these internal screws to hold some of the boards in. I will put a full list of the parts I ordered below, including the headers for the Mister IO board, which took a bit of research to find.

The USB board, and the Mister FPGA itself need 5V power, the USB board came with a Y cable to breakout a single power brick into the 2 boards, but it was not designed for them to be this far apart. Usually the USB board stacks directly under the FPGA, with this case they sit several inches apart. I ended up getting a 1ft extension cable to be able to make up the difference. While that worked I then got a 2.1×5.5mm barrel connector and socket to put on the back of the case, now it has a nice flush place on the back of the case to plugin the power for the USB board. I am using a SD card right now for all my storage. The 128gb it gives me is fine to get started. I have seen people with setups that have a SATA SSD in the case with a USB adapter. This case supports in in the spot under the FPGA. If you load the system up with a ton of classic games and systems, that may be needed.

Setup and Software

Setup I used the Mister “Mr Fusion” Windows installer. Popped in a 128gb micro SD card, and a few minutes later it was ready to go. It takes about 10 minutes the first time it is setup and has internet access to download all the “updates” which is every core registered with the project. The Wi-Fi and Bluetooth dongles were automatically detected, I just had to enter Wi-Fi credentials.

I think the case came out nicely, and have been having fun installing things on it and playing with it. While the 2GB virtual hard drive I gave Windows 95 is on a SD card and gives decent read/write speeds there, the FGPA 486 at 90mhz still struggles a bit with Windows 95. People are working on getting the perf better. Improvements like recently added L2 caching can help. With the click of a button I can swap it over to Windows 3.1 on a different virtual drive and load up my DOS collection. One of the benefits of the Mister project as mentioned is the ARM management layer, I can add files to a ISO, then SCP it to the system. You can also use any size SD card for all your images, and when you want a new virtual hard drive, its a few clicks away. Then mounting those images is straight forward. Windows 3.1 and 95 are supposed to be able to open a null modem connection to the host and transfer files/browse the internet that way, I have yet to get this working.

After all the posts I have done on here recently I couldn’t just play around with the 486. I also got the Mac Plus side of the house running. You can run with 512kb, 1mb, or 4mb of RAM. It has a 20MB HDD, and 2 floppy drives. There is also a Turbo mode, which we obviously need because turbo! And because classic Macs can be slow…

All together it is a fun project I continue to play with. I like being able to play with classic systems like a Commodore 64 without it using up space in my small apartment. The ease of loading software also makes for a very enjoyable experience. If anyone has experience with this, or has questions feel free to comment below!

Parts List

I tend to get packs of things when working on a project like this. I can use them later and it gives be options with several sizes. I did not include the Mister Board and IO board since there are many sellers of those standard parts, I did include the USB and Bluetooth because they have been proven to work.

Wireless 802.11AC USBhttps://www.amazon.com/gp/product/B01MY7PL10/
Bluetooth Adapterhttps://www.amazon.com/gp/product/B07J5WFPXX/
Keyboardhttps://www.amazon.com/gp/product/B06XGHP35N/
LEDs for breakouthttps://www.amazon.com/gp/product/B07G49PJLG/
Buttons for breakout boardhttps://www.amazon.com/gp/product/B071KX71SV/
Board for breakouthttps://www.amazon.com/gp/product/B078W1F3PC/
USB 3.0 extensionhttps://www.amazon.com/gp/product/B087BK3WRB/
USB 3.0 90 degree connectorhttps://www.amazon.com/gp/product/B0793NQRYZ/
Ethernet 90 degree connectorhttps://www.amazon.com/gp/product/B01MSIE5HO/
USB Board Cable Headerhttps://www.amazon.com/gp/product/B01DP13EDA/
USB Cable header to micro usbhttps://www.amazon.com/gp/product/B07WLM7Q6Z/
1×5 pin IO board cablehttps://www.amazon.com/gp/product/B072VM9HVC/
1×7 pin IO board cablehttps://www.amazon.com/gp/product/B01IZDBXF4/
Power port and cableshttps://www.amazon.com/gp/product/B01NBOP46E/
Power Extension Cablehttps://www.amazon.com/gp/product/B01M6C24XG/
Internal Screwshttps://www.amazon.com/gp/product/B07QFVRS77/

Homelab: Hypervisors – Part 2 – VMware

What I want to say is, after deciding it was time to move to VMware and attempt to use vSAN instead of Storage Spaces Direct (S2D) I wanted to research the hardware I had and see if it would work on ESXi 7.0. But of course I did not thoroughly read all of the changes vSphere 7.0 has brought. The holiday was approaching and I was going to use this time to do my migration. I had read up on vSAN and knew I needed cache drives. I bought a few small (250GB) NVME drives to put into each system. Getting those drives installed took a day because I needed to create a custom 3D printed mount. That would give me a good speed boost for my storage no matter what. Having recently upgraded to 10GB networking, I already had HP and SolarFlare 10gb networking cards. The time came and I copied all of the VMs I had in Microsoft VHDX format to my NAS (which wasn’t getting changed), then unplugged the first Hypervisor, and attempted a ESXi 7.0 Install.

One hardware change I should note, I am using USB 3.0 128GB thumb drives for the ESXi OS. This also allowed me to leave the original Windows drive untouched, allowing for easy rollback if this was a nightmare. I put the ESXi 7.0 disk into the first system AND! Error, no networking card found… I started searching online and quickly found a lot of people pointing to this article. ESXi 7.0 had cut a ton of network driver support, everything from the Realtek motherboard NIC to the 10GB SolarFlare card would not be supported, with no way around it (I tried). It comes down to 6.x had a compatibility layer in it where Linux drivers could be used if there were not native drivers, 7.0 removes this. I then got a ESXi 6.7 installer (VMware doesn’t allow you to just download older versions on a random account, but Dell still hosts their version) and installed that. Everything came online and started working. Now that I knew the one thing blocking me was that, I installed all my systems with 6.7 while I waited for the 3 new Supermicro AOC-STGN-i2S Rev 2.0 Intel 82599 2-Port 10GbE SFP+ cards I ordered. Using the Intel 82599 chipset, they have wide support. 2 Ports is nice; and, the 2.0 revision of the card is compact allowing them to fit into my cases. So far I recommend them, they also are around $50 on eBay, which is not bad.

I played with a few of the systems, but decided to wait till the new network cards were in a few days later to initialize vSAN and copy all of the data back over. I used this guide, from the same author of the other post about ESXi 7.0 changes to configure the disks in the system how I wanted them. At one point I thought I was stuck, but I just had to have VMware rescan the drives. I setup a vSphere appliance on one of the hosts. This gives me all the cluster functionality, and single webpage to manage all the hosts. Here I an also create a “Distributed Switch” which is a virtual switch template which can be applied to each of the hosts. I can set the vlans I have, and how I want them to work in one place, then deploy it to all the systems easily. This works as long as all your hosts have identical network configurations. After watching a YouTube video or two on vSAN setup I went ahead setting that up. The setup was straight forward, the drives reported healthy, and now I was ready to put some data on it.

A small flag about vSAN, it uses a lot of RAM to manage itself and track which system has what. I was seeing about 10-12 gb of ram used on each of my hosts, that has 32gb to begin with. There are guides online for this, and I believe it can be tweaked. It has to do with how large a cache drive you have, and your total storage. Not a big deal, but if you are running a full cluster, something to be aware of.

Migrating the old VMs from their Hyper-V disk images to VMware was not too difficult. I used qemu-img to convert from VHDX to VMDK. The VMDK images that qemu creates are the desktop version of the VMDK format. VMwares desktop products create slightly different disk images than the server versions. I then unloaded these VMDKs onto the vSAN and used the internal vmkfstools on ESXi Shell to convert those images to the server versions. The Windows systems realized the changes, and did a hardware reset, they worked right away. The Linux systems (mostly CentOS 8) would not boot under any of the SCSI controllers VMware had. After reading online, and a bit of guessing, I booted them with the IDE controller which appeared to be the only one dracut had modules for. Once the systems were online I could do updates, and with the new kernel version they had available they made new initrd images. These images being created on the platform with the new virtual hardware, installed the SCSI controller modules and could then be changed from IDE to SCSI mode.

So far other than the hardware changes that needed to happen, moving to VMware has worked out well. I am using a VMware Users Group license, https://www.vmug.com/, which is perfect for homelabs, and doesn’t break the bank. I am starting to experiment with some of the newer or just more advanced VMware features that I have not used before. We spoke of vSAN, I also have setup DRS (Distributed Resource Scheduler, allowing for VMs to move between hosts as resources are needed), and want to setup a key manager server to play with VM encryption and virtual TPMs.

Now that I am off of that… unsupported… Storage Spaces Direct configuration updates are much easier. I can put a host into maintenance mode, which moves any running VMs, then reboot it and once its back online, things re shuffle. This does mean I need enough space on the cluster for 1/3 of it to be off at a time, but that is ok. I am running 32gb of ram, with 2 empty DIMMS in each system, when the time comes I can inexpensively add more RAM.

If you/your work has a NetApp subscription, there is a NetApp Simulator which is a cool OVA you can deploy on VMware to learn NetApp related things. I was using that at work to learn how to do day to day management of NetApps. Another neat VM image that comes in the form of OVA I found recently is Nextcloud’s appliance. They have a single OVA that has a great flow for taking you through configuring their product.

Overall the VMware setup as been as easy as I thought it could be. Coming from a workplace who runs their management systems without a lot of access, it has been nice having vSphere 7.0. It automatically checks in online, and lets me know when there are updates for different parts of the system.

Homelab: Hypervisors – Part 1 – Hyper-V

For the last year I have been running Microsoft Hyper-V on Server 2019. Due to mounting issues I moved over to VMware vSphere, this first post will discuss my Hyper-V setup and feedback about it; then the next post will speak about my migration and new setup. When I started building out my home setup I was studying to take a Windows Server certification for work, with that and about half of the virtual machines I had at home were Windows, Hyper-V was the choice for hypervisor. One feature that stood out to me was Dynamic Memory on Hyper-V because my home setup was not that large; as well as the automatic virtual machine activation (Microsoft Doc). Later, I was attempting to run Storage Spaces Direct (S2D, SSD would be a confusing acronym so Storage Spaces Direct goes by S2D), except my setup was not supported, which made me run a… not recommended configuration… more on that soon; and I kept having issues around the Hyper-V management tools. I decided it was time to migrate from Hyper-V and S2D to VMware vSphere and vSAN.

(Please note for feedback I am discussing Windows Server 2019 here, and VMware vSphere 7.0)

Selecting a Hypervisor

I wanted to briefly go over a bit more of my thought process when selecting a Hypervisor. I already mentioned some of the reasons it made sense, but I also wanted to mention more of my thoughts. I started the search looking for a Type-1 Hypervisor awhile ago, I was going to be running on a Intel NUC with a few Windows and Linux VMs. Being a homelab, I thought I would look at free options.

Having used Proxmox years ago with a ton of issues I wanted to steer clear of it (looking back this probably was not fair, it was several years since I last used it and I believe it has gotten better); I had also used CitrixHypervisor (formerly XenServer) with many issues including a storage array killing itself randomly in one reboot. One of the requirements I gave myself was to have a real management system, I did not want to run KVM on random Linux hosts. That brought me to the 2 big ones, VMware and Microsoft. VMware has a lot of licensing around different features, but was the system I knew better. I could get a VMware Users group membership for homelabs, and that would take care of the licensing. On the other hand, with me studying for Windows Server tests, and the book speaking of different Windows Server and Hyper-V features, I thought I would give it a try. The following are things I liked about it, and then what turned me away from it.

Great Things About Hyper-V

I want to give a fair overview of my year plus running Hyper-V. There are some great features; dynamic memory allows you to run modern OSes with a upper and lower limit on memory, and then most of the time while the VMs are idling your memory footprint is very low. Another great feature is the earlier mentioned automatic activation, as long as your Hyper-V host (Windows Server 2019 Standard or Datacenter not the free Hyper-V Server) is activated, it can pass that activation to your guests and allow you to run Server 2012+. All of the services are running on Windows, out of the box you get all the benefits there; such as creating group policies for your servers and using that to do a lot of your fleet management. I recently have started using Windows Admin Center, which gives you a single view on all your Windows systems and allows you to update them all in one place. Hyper-V works well if you have a single node, and want to do basic things with it; when you move to doing clustering and advanced storage Hyper-V starts to give you a lot of issues.

Hyper-V Manager on Server 2016 (and 2019)

General Hyper-V Issues

To dive more into the woes I was having with Hyper-V, some of it is my own doing, some of it is the tools. Even before I was running S2D, I was running several Hyper-V boxes each with its own storage. I will go into my issues with S2D soon. Hyper-V’s management tools are not good. You have several options on how you will manage the systems, the first and easiest is Hyper-V Manager. This is a simple program that allows you to 1-1 manage a Hyper-V system. I mean 1-1 because if you have VMs that are part of a failover cluster, you can connect to them here to view them but that is it. Hyper-V Manager only allows you to manage VMs that live on one hypervisor with no redundancy; for casual use, it works. I use it for my primary AD host because I don’t want anything fancy going on with that box, when I need to start everything from scratch, I need AD and DNS to come up cleanly.

Maybe you have outgrown the one off server management and want to move your systems into a cluster. Now its time for Failover Cluster Manager. You add all the servers into a Failover Cluster together, and get through the checks you have to pass. Then there is a wizard to migrate your VMs from Hyper-V Manager into Failover Cluster Manager. One requirement to do this is to have storage that every box in the cluster can use, either S2D or iSCSI (you can do things like Fibre Channel, but I was not going to do that). I used the tool and the VM said all its files were moved onto shared iSCSI storage that all the machines could use. Should be good right? Things seem to be working. Then I would move certain VMs to other hosts and it would fail, just some of them. It came down to either a ISO, or one of the HDD hibernation files, or checkpoints (Microsoft version of Snapshots) being on one of the hosts, and the UI NOT mentioning this. Thus, when the VM tried to load on another system, a file it needed was not there and it could not load. Failover Manager is also fairly simplistic and doesn’t not give you a ton of tools. Again, Windows Admin Center adds some nice info on a standard cluster, but it is not fantastic; leaving you to dig through Powershell to try to manage your Failure Cluster.

On occasion the Virtual Machine Manager service that is in charge of managing the VMs, and gives the interface to monitor, modify, and access the VMs would lock up. Hyper-V Manager and Cluster Manager would show no status for the VMs, and I would have to restart the service. These minor issues would stack up over time.

To manage Hyper-V remotely (meaning from any other system) you need to setup Windows Remote Management, winrm. This system by default uses unencrypted HTTP. Encryption can be turned on with a few commands in the command line, but it creates a cert based on your hostname and IP address. If you have more than one IP, OR you are in a Failover Cluster this means you will be spending a lot of time customizing these certificates because it will just get a cert for your host, and when that node because the Failover Cluster manager, it needs that virtual IP and hostname in the cert. I had to create different certs for that virtual interface and put them on the different nodes manually, there are people in the Microsoft support forum talking about this. Here is an example incase it helps anyone of creating a cluster listener after manually creating a cluster cert.

winrm create winrm/config/listener?Address=IP:192.168.3.8+Transport=HTTPS ‘@{Hostname=”home-cluster.home.ntbl.co”;CertificateThumbprint=”BFCDE6C85A0B12426A44BC3F44236313317C63CC”;ListeningOn=”192.168.3.8″}’

There also is a System Center Manager that is another package you can purchase from Microsoft to manage Hyper-V. Having dealt with System Center to manage Windows systems at work, I did not want to touch that at all. Hyper-V has a lot of things going for it, and the underlying code running VMs works well 99% of the time. I wish Microsoft put more time to grow the tools you use for managing it. Parts of the process like setting up networking on different nodes could be much smoother, in comparison with VMware Distributed Switching. I installed one of the systems I had on Windows Server Core (no user GUI) to learn more about that. If your primary interface needs a VLAN for management, this a painful experience. You have to create the Hyper-V virtual switch and attach your management interface to it and assign the VLAN all from within Powershell. If you need to do it, this is a good resource. Thing like this, and the winrm issues, make Hyper-V feel unpolished even after being in the market for years.

S2D Issues

I wanted to put these systems into a failover cluster, allowing them to move VMs between each other as needed, except then I needed shared storage. I attempted to use iSCSI from my FreeNAS box; alas, with 7, old, spinning drives, the speed was not great with more than a few VMs. Then I thought I had some spare SATA SSDs and I could use S2D to do shared storage. For those who have not attempted to setup S2D, your hard drives have to either be NVME, or an internal HDD controller. The system will refuse to work on any configuration that it does not like. With most of my systems being small form factor PCs, and I am just using a few SATA drives I got USB 3.1 4 bay SATA enclosures. Not optimal but decent speed and it allows me to add a good number of drives to each system without a large expense of a full RAID or SAS controller.

S2D refused to work with these drives. I believe it came down to the controller the USB drives was using and it not signaling something the systems wanted. The drives showing up as Removable also made Windows refuse. There are commands you can run like below that will enable more disk types to work, but I could not get my dries to show up.

(Get-Cluster).S2DBusTypes=4294967295
Powershell command to enable all disk types in Storage Spaces Direct from this article

Then I had an idea, an evil and terrible and great idea. I created 3 VMs, one on each Hyper-V box, then gave the 3 disks of each server to the VM in full. Now I had 3 VMs, each with 3 drives (and a separate OS drive) to run S2D. To the VM OS, it looked like they had 3 SCSI HDDs that was happy to use for S2D. I put these three Windows Servers into a failover cluster together, and setup S2D. Overall setup was not too bad. If you have Windows Admin Center configured it is much easier to setup and use Storage Spaces Direct than the GUI in Windows Server. There are a ton of Powershell commands for configuring S2D and you will probably end up using a bunch of them.

This worked! The systems were in a failover cluster, of their own, and my main failover cluster that controlled the VMs could use it as shared storage. If you use Windows Admin Center you can get nice stats from the Storage cluster about the sync status of the disks. Every time one of the storage nodes reboots, the cluster needs to re-sync itself. There are different RAID levels you can set the S2D setup to, I set it to have 2 additional copies of each set of data, this means each node has a full copy of the data; this uses a lot of space but i can have 1 node run everything (which ended up being overkill).

This setup ran for a while decently, other than the small VM overhead, it was fast and worked. The issues arose when the second Tuesday of the month came around and I needed to do patching. The storage network was sitting on top of the Hypervisors, and they didn’t really understand that. I often ran into problems where I would shutdown one of the storage nodes to patch it, and patch the host, then the other 2 nodes would lock up or say all storage was lost. This would occur even when preemptively moving who was the main node, and prepping to restart. With storage dropping out from under all the VMs, they would die and need to manually be rebooted or repaired. This made me start to look for a new setup after a few of these months.

All in all, I ended up running for over a year about 5 Windows VMs and 5 Linux VMs on Hyper-V with good uptime. One benefit of Hyper-V is you get the hardware compatibility of Windows, which is vast. The big downside of Hyper-V is the tools around it. At times they seem unfinished, at other times buggy. My next post will be about the migration, and my experience with vSphere 7.0!

Resources

Good guide for Storage Spaces Direct

http://woshub.com/configure-storage-spaces-direct-s2d-windows-server-2016/