Hardware

Optiplex 5050 Back view

Dell Optiplex 5050 Micro Windows Server Installation

Recently I was able to pick up some Dell Optiplex 5050 Micros for $60 on eBay. These are tiny machines, with an Intel i5-7500T (4 core/4 Thread) CPU, 8GB of ram, and a 256GB SSD. For $60 they needed a power supply, but those are easy to come by. My plan was to replace my aging Intel NUC that is the core domain services for the house (AD, Radius, CA) and perhaps the aging firewall, if I can figure out how to get a second NIC into the system, more on that later.

My philosophy when running a standalone network (even with internet access) is to have at least 1 of your Domain Controllers (DCs) be a physical box at all times. An alternative is a dedicated hypervisor with local disks, but anyone who has tried to start a VM manually on VMWare knows how painful it can be without any interface to the system other than the command line. In addition, these days it’s easy to make all the DCs virtual, but if you ever have to cold boot your environment; then you run into not having DNS. Following not having DNS, things like vCenter and vSAN can’t come up cleanly, and there are more and more chain on effects. Having a physical machine allows you to bring DNS and core services up first, then start all other services that rely on your domain.

The first task I had was to get one of the Optiplex 5050s ready for Windows Server. I started with upgrading the ram to 16GB, because I had it laying around. After that, since this is an eBay purchase, I updated the firmware/BIOS and ran diagnostics before it touched the home network. The seller was nice enough to install Windows 10 Pro on the machine, which has a license in the BIOS; but I formatted the drive before starting that instance. People are generally nice, but who knows what was in that image. After getting Windows Server 2022 installed I hit my first issue. Server 2022 does not have a driver for the Intel i219-V that is in this chassis.

I tried getting the drivers from the Dell site, but Windows refused to use them because they were for Windows 10, and not Server edition. My current fix for this was going to select the driver, telling it to “Browse my computer for drivers”, letting me pick, then manually selecting the “Intel” “Intel(R) Ethernet Connection (2) I219-V” driver. I had a USB ethernet dongle that worked for me to get online and at least be able to see that driver. Now the box is happily online. The main issue with this technique is that I keep getting an “Optional” Windows Update for an updated driver that seems to never install. I think that is because I installed the Dell driver, but it never runs correctly.

Another thing I try to do with most systems, especially the systems in charge of security is get Virtualization Based Security running. This is a newer Windows feature, where core elements that need to maintain secrets are run in tiny Hyper-V containers. The user never sees it, but this gives added protection to the system. If you run “msinfo32”, you can get an output of its status. Most of the time, you need to enable chipset virtualization support; then add the system feature of “Host Guardian Hyper-V Support”. On older systems (Windows Server 2019) and desktops, I think it’s just called “Hyper-V”, then you get these features enabled.

On paper this machine is 78% faster than the Intel i5-3427U, and that makes a world of difference. The old system took a while to boot, and a while to backup, which is what spurred me to upgrade. This system feels amazingly fast for a $60 system. Keep in mind that it cost less than the Raspberry Pi 4, has Intel, and didn’t have to wait the years Raspberry Pis take right now!

I have the main DC run domain services, DNS, Network Policy Service (RADIUS), and certificate services. For the first two, I just had to install Domain Services and DNS and the system started acting in that role. For NPS I exported the config from the old DC, and then installed the service and imported onto the new one. As a reminder, Domain Services has to be installed first, or if you have NPS/Certificate Services installed, then try to do Domain Services, it will tell you it can’t install. Certificate Services, I added a new CA, stopped the old one’s service, and removed it as an enrollment agent in ADSI. My 802.1x and other certs given out by GPO are short lived, around 90 days; I will wait for the old ones to expire and systems to naturally get newer certs.

The second system I got; I thought I would try to do some hardware hacking. My hope was to repurpose it as a firewall for my aging Dell Optiplex 990 from 2011. To do this I would want to add at least 1 more NIC to the system. I ordered a 1gb ethernet NIC that goes where the WLAN chip goes. At first it did not show up in Linux and I was worried. Turns out the system bios had “wlan” disabled, and by enabling that, it turned on that PCIe channel. Then the card would show up. Having mounted the ethernet port in the extra serial blank this system has did make it look very clean and easy. I had to tuck the wire away as it came from the front of the unit to the back and had the sata drive siting on it. After playing with it a good amount, removing the card, reseating, putting electrical tape under it, I was able to get the line up, but not reliably at 1gb/s, it tended to go down to 100mb/s a lot in coming up. While things like loosening the screw holding it down, and putting electrical tape under it helped, the system was not reliable enough for me to feel comfortable using it for homelab-production. I shaved down the connectors at the end of the card, with them being that large, the screw couldn’t easily get between them. That did not help that much.

In the end I am enjoying the one system as a new DC. And eventually will figure out what I want to do with the other one. With having a NVMe slot, and SATA internally, in addition to being able to go up to 32GB of ram on a low power budget they are very capable little machines.

THE µKENBAK-1

Back again with another retro computer kit from the same creator as THE ALTAIR-DUINO, a small quick kit in the µKENBAK-1. The µKENBAK-1 with the µ in front denotes one of the earlier versions of the kit. This is smaller than the original computer kit, compared to the full-sized replica or nanoKENBAK-1 now offered by the creator. This is a small, and simple kit. Running off an Atmel processor (same as Arduino), this little recreation offers a fun, simple front panel, and relatively quick assembly.

Compared to some of the other kits that have been posted here, this one is straight forward to put together. While you have the classic soldering, the kit is all through hole components and is a pleasant hour or so to put together. The most time for me in putting the kit together actually came down to getting the PCB with the stand-offs in the case and lined up with the back holes. This proved to be a difficult, and time-consuming process. You need to pre tighten them on the front panel, which then slides into the case, and line them up with the back holes. Between the standoffs being plastic and wanting to strip, and them wanting to wiggle all over, most of my time went into this instead of soldering. In the end, I got 5/6 in place and called it a day.

Evil Stand-Offs

The creator of this kit shows his experience in creating these kits, in little details, which make the kit a nice experience; one example is the usb extension cable which gives you an easy connection out the back is the perfect length to do the job but not be in your way. Another is the instruction booklet coming with a bunch of examples on how to use the computer, right after the assembly instructions. These instructions come in a nice spiral book included in the box.

The creators website, https://adwaterandstir.com/kenbak/ also goes into detail about the creator of this machine, (the original one in the 1970’s), and its history.

This is one of the easier kits I have done, but enjoyable in its ease to put together. I would recommend this kit to someone who is looking to get started with these kits.

Ruckus ICX 7150-C12P Switch Repair

A while ago I purchased a Ruckus ICX 7150-c12p off eBay to use at home. It gives 14x1gb/s ports, and 2 SFP+ ports. The SFP+ ports are limited to 1gb/s by default, and there is a honor system license for upgrading them to 10gb/s. These switches go for $600 – $1200 depending on where you get them and which license you get with it (1gb/s vs 10gb/s). The switch is also POE, and can do 4 POE+ (30 watt) ports. I had one of these switches and it worked great. I wanted to get a second one to replace the WiFi link I was using across my apartment with a fiber link.

Instead of paying ~$250, which was their going rate on eBay; I saw a forum post about replacing this models power supply, and thought I would give that a shot. I got a broken switch for $45, and then a PSU for $50. The PSU I used was a SL Power LB130S56K 56V 2.32 130W. Armed with someone’s photos of doing this repair it ended up going fine. The hardest part of the whole operation is that the pins going onto the main board are reversed from what the power supply comes with, so you need to flip them. I have been running the unit for almost 2 years now without issue.

This model of switch is great because of its features and is fanless. The fanless-ness part of it is nice for homelabs near your desk, because the switches are silent. Because they are fanless, they cant have anything put on top of them, and need some room to breath. I think a lot of the ones you see online dead are because someone didn’t give it enough air, and the PSU died. Note when looking for a similar dead switch on eBay, you really want the seller saying “when plugged in nothing happens”, not “it periodically blinks” because that could be bad ram and its in a boot loop.

Having run two of these switches for over a year, I can give some feedback. I really like them. I have the two I have in a stack, I login once and manage both. When it comes time for firmware updates you SCP the file to the management IP, and it downloads the file to both, and then flashes and reloads. I came from using Cisco gear usually, or sometimes Arista; the CLI is a bit different, and Ruckus handles VLAN setup a bit weird, but once you get used to it, it makes sense. They are solid switches, with POE, that you can set and forget for a while.

New PC Build 2022

Having built my PC a few years ago, I was thinking about upgrading it, but with chip prices being what they are, and graphics cards costing more than a new car, I thought I would wait. Then a friend of mine happen to have an AMD 5800X that he was willing to give me a deal on… After years of having the custom case, while it was nice, it was HUGE and made it difficult to setup a desk in a tiny New York apartment. I used this opportunity to shrink a bit, and update some of the components. I say some, because some of them (like the graphics card) were going to stay to save cost.

Old case being taken apart, its a bit of a mess

At this point, this post is mostly a standard PC build, with a few hiccups along the way. Looking on the Small Form Factor PC subreddit, and https://caseend.com/ (a website for small pc cases), I settled on the ZZAW C3. It is relatively small, supports Micro-ATX, full size graphics cards, and all-in-one water coolers in a ~22 liter case. I always try to get Micro-ATX over Mini-ITX for my desktop because you get more PCI slots; as well as 4 DIMMs for RAM instead of 2. I also wanted to try one of these all-in-one water coolers, since I never had and thought I could get good results (and a quiet case).

The case came nicely wrapped. There were not a lot of instructions on how to put the case and system together, you kind of just had to know. That took a bit of trial and error. There are a few screws that are very similar sizes, and not labeled. The whole setup went smoothly after that except that getting the cooler to fit in the case was a bit of a challenge, and the motherboard… So, the motherboard… I got a AsRock X570 Pro4 motherboard, it had all the features I wanted. I get the case put together, I installed my previous power supply, I kept the RAM and graphics card, moved my SSD over (I had a PCIe Gen 4 Samsung 980 Pro on order, but it hadn’t come in yet), put thermal paste on the CPU, install the cooler AND… nothing. System will not boot.

There are lights on the motherboard saying CPU, RAM, and DISK failed. I start searching online and trying different things. A bit in, someone says “make sure your motherboard is updated to support 5000 series AMD”. I just got this board, its fairly new, it has to work right? Well turns out you need BIOS version 3.20 to support the CPU I have, AsRock is at 4.20 for this motherboard. After taking my old 2600X out of my old motherboard, taking out the new CPU, cleaning the thermal paste, reseating, re-thermal pasting the 2600X, turns out the board shipped with version 3.10… One revision before what I needed. I updated the BIOS, then swapped the CPU back, doing all those steps again. This time, the system worked.

I later got the Samsung 980 Pro SSD, which was a tiny bit of a headache to migrate over because I had Bitlocker on and trying to relocate the bootloader was not straight forward. I kept getting “Boot Device Not Found” and eventually somehow doing a system restore to before the last Windows Update got Windows 10 to reinstall the bootloader on the drive. I have no idea why that would have fixed it. I had disabled Bitlocker before starting this ordeal, but that just does it one reboot, I really needed to just fully disable it to save me typing in the recovery pin many times.

The system is working well. The only remaining issue, that is very odd, but I am just living with is whenever the system attempts to do a restart, it freezes before coming back. It properly shuts down, and if I hold the power button and then turn it back on everything is fine. But it refuses to gracefully restart. It also will fully shutdown fine. Odd…

Building a Pac Man Battle Royale Table

This is a post I was working off and on for several years. It is something I always was meaning to finish, and got very near the end, the Covid and life happen. Instead of throwing it out I figured I would do some small edits then put out as is.

(2017) Friends and I always found the game Pac-Man Battle Royale to be fun, but when we went to look at the price of a cabinet they were $5,000! Worth it if you are a bar or arcade, but for a few friends playing games a bit over the top. After a evening at Barcade, I started the trek to see if I could build one myself for less.

Original Table

The first step was figuring out where I could legally get the game, a number of sites offered the ROM but that is not what I was looking for. Next I found it was ported and available on Steam! We are in luck! The game is available for less than $10! But once I start it, I am greeted by a “fun” border and changes they made to the screen for the Windows version.

Steam Version
Steam Version
Arcade Version
Arcade Version

On the left you can see the Steam version; there is a border, along with all the player text is facing one direction. On the right is the original version, where the screen goes to the edge, and the 3rd and 4th players face the other direction for when players are standing around the table. This version also has been made to play with Xbox 360 wired controllers, it works with keyboard but a lot of the interface seems to be built around that.

The solution, a brave hero on Github made a modified DirectX9 driver that edits the game screen as you play! https://github.com/vikbez/pacbrcade After installing this file, I was able to get the game to look just like the original! Below are the before and afters from the repo, and I can attest it does a great job! Throw in a script to start the game at startup of a Intel Compute Stick (a full pc on a HDMI dongle), and this was ready to go.

(2021) Now it came time to build the cabinet, I hadn’t built something thing big before and didn’t really have a large plan. The hope was to have this live at the office. Being in NYC it needed to be able to fold up, and then be put somewhere when not in use. Part of my plans were to give it folding legs, and one side of the table should be a rest, so the legs can be folded and it can be put on its side somewhere out of the way.

Most of the construction was actually done in one weekend in 2017, I got some 2x4s and went to the maker space I am a member of fat cat FAB LAB – NYC Hackerspace to cut the boards down to the sizes. Now this was a bit of a rough day because I was in NYC, and I am literally grabbing large 2x4s (some are 6 or 7 feet long) then walking a few blocks to the woodshop as people are drinking on a Saturday around me on the streets, then cutting them and walking several blocks to the office. After all is said and done, I used a normal drill and some wood screws to put it all together. I then used yellow vinyl wrap to make the table a bit more dressed up. Getting bubbles out of the vinyl wrapper, on a wood surface that isn’t completely flat was a bit of a challenge. I had a area for the screen to go in, then 4 sides; 2 with cup holders, and 2 with joysticks.

I used Teensy micro controllers as the joysticks, they emulate joysticks on a computer and you an make any input trigger any signal you want. I wanted to add some more style to the unit so I made Player 1 through 4 acrylic panels to go around the joystick and button. These were laser cut at the same woodshop I used before. I also ended up getting craft beer labels, and putting it on the inside of the buttons; that gave the unit a little more character. I put the rubber molding along the side of the unit, as you would expect from any good arcade system. After installing an old screen I found around I toped it with a sheet of clear plexiglass. This was a learning experience of plexiglass scratches easily and can crack if too much pressure is put in say a screw hole.

The whole thing worked, it booted up on the Compute Stick, auto loaded Steam, and started the game. The main issue that had me put it away for a while was the joystick handling. The joysticks I had were 8 way joysticks; they could go to the 4 sides but also to all the corners, we didn’t want the corners. In a game like Pac-Man, going to the corners of the joystick made the character either not move or go in one of the 2 directions you were facing.

The bottom of the joysticks had a plastic piece which allowed the rod of the joystick to go in certain directions. On the bottom of the units I had it was a empty square. I wanted it to be a diamond, this would have forced the player in going one of 4 directions instead. The plan was to design a piece then 3D print it and attach it to the bottom of all the joysticks.

With that I put the system into a closet, where it sat for several years. There was the issue on top of all this of being in NYC and there was no place to put the thing where it would not be in the way. I worked on this before I had a 3D printer and could have put the piece together quickly.

In the end it was fun to put something together quickly like this. I got to do some bigger wood working and vinyl wrap something. One take away I have from it was the momentum of a project cant be a very motivating thing. I started and put most of the system together in a single weekend. Then the last few percent of the project, getting the joysticks correct, I lost the energy (and didn’t have a place to put the thing) and went onto other projects. I try to use that motivation to push through projects when possible, and use this project as a reminder to do that.

Briel Computers Replica 1 Plus

Kit

I recently ordered the Briel Computers Replica I Plus, a Apple I clone. Instead of the originals big board to do a lot of NTSC generation, it uses a more modern single chip. The shipment came in a small box, and with everything I needed. The creator of the kit did a great job including everything you need, down to including an anti-static strap! The project came with some solder, but not nearly enough for everything, I think it was thicker to go with the structural points. Briel Computers sells the kit through ReActiveMicro.com. At $135 it is one of the less expensive kits I have had, but also comes with just the board. If you want a case that needs to be 3D Printed (more on that later).

ReActiveMicro points you over to the Project Wiki for more information, there is a ton there and a link to someone putting the project together. I found this easier to follow along and do than reading the instructions.

The kit was fairly easy and straight forward; I ran into a few small issues around the PS/2 port since the solder points are close together. Getting the few connector ports in can be a bit difficult with a few tiny pins and getting them in the board. As long as you have patience, then you can get through it.

I got it all together, and the board started the first try. I did have the same issue the person who made the video had; I was getting a lot of noise and characters added to the screen. I reflowed a lot of the sockets, and made sure all the socketed chips were fully seated. That cleared up the garbage at startup. The wiki also has some other notes on noise issues the board can show.

I also could not find a PS/2 keyboard in the house, and all the USB keyboards I had didn’t seem to like the USB->PS/2 Adapter. I am not very surprised by this because I didn’t have any very simple, older keyboards.

The USB port that is used for power is also a serial device for a PC/Mac. I plugged into that and got the serial driver working from SparkFun website, they produce the module. Then the output worked well, and I could enter BASIC on the board!

Case

I wanted to put the board in some sort of case, and after searching online I couldn’t find any. I thought I would throw something together quickly that I could put the board in. I took some measurements and threw together a V0 of the case. One small issue was I didn’t account for the RCA jack the video comes out of little let that sticks out. Instead of spending another 7 hours printing a new one, I used a little saw I have to cut a hole out.

Part of my thought of creating a case was to have something I could put the board in, then store it in a cabinet or shelf and not be worried that the board would get damaged. I also made a case that can go over the entire unit to protect it in storage.

Again, looking back small design things could have been changed, like flip the name of the project in the case, so looking at it in the protective cover, the text would be right right way. Getting the scaffolding out of the protective case was not the easiest of things. I designed the protective case with a rail that brings the edge of the mounting board into a locking position when you slide it in. I have to say, that was a nice aspect to the design. It took over 6 hours to print though.

MisteRdeck MIDI Control Desk

I have been enjoying 3D printing projects recently. I saw a little control board for changing audio levels, and having hotkeys while playing games. The printing took a good long while, and I had to edit some of the parts to work with the parts I found currently on Amazon. I will post the parts list below. The soldering was straight forward, and the project came with a PDF that had good instructions. This also turned into a good opportunity for me to use the new Wiring Pencil, which worked surprisingly well.

For hardware, I am using a Teensy; the Teensy can be a USB keyboard or MIDI device or joystick or serial over the USB connection. The project comes with a premade Arduino file to run it as a MIDI controller. I had not worked before with MIDI input like this, but it seemed the best path forward compared to trying to emulate a keyboard and hitting odd key combinations. Or the alternative of writing something that output serial data then finding, or writing, a daemon for my PC to listen to that device.

For software, I looked at several pieces of software to use the keys and sliders with. I looked at software like VoiceMeeter. While overall that worked, it was very inflexible, and had a giant interface for things I didn’t want to use. Then I found Midi-Mixer, a passion project by a single dev and it is EXACTLY what I needed. The sliders can control single app volume, which is easy to select. And the buttons can be programmed for anything! And easily with a GUI instead of conf files like some other open source projects.

Overall I am enjoying the finished project. It sits next to my keyboard, and allows easy changing of levels while playing games. I added little rubber feet I had laying around so the plastic housing doesn’t slide around on the desk.

To fit the sliders, I needed to modify the knobs, here is the modified versions that work with the sliders I ordered below: MisteRdeck Knob Remix by danberk – Thingiverse.

Make: Makes of MisteRdeck – Arduino-based MIDI Stream Deck by danberk – Thingiverse

Example courtesy of midi-mixer.com

Parts

PartURLPrice
Teensy v3.2Teensy USB Development Board (pjrc.com)$19.80
Gateron Yellow SwitchesGateron Yellow Linear Switches | Kinetic Labs$16.10
Slidershttps://www.amazon.com/gp/product/B079ZP3LS5/$11.99 x 2
Diodeshttps://www.amazon.com/gp/product/B06XB1R2NK/~$6
Key Capshttps://www.amazon.com/gp/product/B01M023NFK/$7.50 x 2

Homelab: NAS 2021

One piece that sits at the heart of my Homelab is the NAS I have. This is actually the same NAS I have written about years ago, looking back on that post brought back memories of the pervious system and Server 2008 that I didn’t recall. In the last year I have added several drives and a new network card to this box, I thought that as well as my experience running FreeNAS, now TrueNAS Core, over 8+ years was worth discussing.

When I built out that box, I had 5x3TB drives, each around $125 dollars. Now those same drives are $40. The rough rule of thumb I was always told is 1GB of RAM for ZFS for every TB of storage you have. So I maxed the mini-itx motherboard out at 16GB of RAM to get as close as I could get. This let met run basic services and I was running a few small VMs/Jails on the box. This did cut into the RAM I had available, but was a nice feature. This allowed me to run the Unifi controller without another system running. Back then, Raspberry Pis came with 256MB of RAM, making it not ideal to run too many services. I later would end up moving all of those to dedicated Raspberry Pis then later VM hosts.

These 5 disks served me well for a while; I every year or two would have a drive die, and it got cheaper and cheaper to replace them. I use this NAS for backups from my Windows desktop, and Macbook. Time Machine backups over the network to Macs works very well with TrueNAS. I ended up getting a smaller version of this box for my parents home, and sister, you can run the OS off a USB with a single or 2 small hard drives in a box like an Intel NUC, then have it always backup their PCs. Reminding people “plug in that USB drive” to backup seems to never stick. TrueNAS offers one click updates, with optional automatic checkin; this makes keeping the system up to date easily.

There have been reports of recent corruption with 12.0, but I have not seen that. Also there was a bug where you could get a banner saying “THIS IS A TESTING RELEASE NOT FOR PRODUCTION” on a production branch, so that is fun. These days those backups, and my Veeam backups are done to the NAS. I tried to use it as a iSCSI and then a NFS target, but the IO was a bit too much for these old spinning drives. Now I use vSAN, as mentioned, which has performed well for VMs, that leaves the NAS just as dumb storage for Veeam. Veeam is a good product that makes it very easy to backup VMs, I will probably write an article on it later. The software has a free 10 VM backup license for Homelabs.

In 2020 I was using a high percentage of the storage for backups and VMs, and was pondering upgrading. I didn’t want to throw down enough money to build a whole new system, and I liked this case a lot, so I started to look at what I could do to add to it. I was using 5 drives, but the case technically supports 7, with 2 being on the bottom. The issue was, I didn’t have enough SATA ports to add to the system. This brings me to one of the scariest, worst, best, cards I have bought. This card, adds 4 ports through a mini PCI-E connection. It actually works really well, with the drives coming up like any other, it gives you 1 PCI E Lane at roughly 2.5Gbp/s for my version. I have 2 drives of the now 7 I have in a RAIDZ2 (RAID 6), and for over a year it has worked well. The one other thing I added to the box was a 10GB networking card, I did a push a bit ago to move most of the Homelab server stuff to 10GB, and this box was part of that. TrueNAS is built on FreeBSD, and has good hardware compatibility, I got an old Intel X520 for compatibility and ease. I have seen it get near 5gbit/s, averaging closer to 2gb/s with writes.

First of all, yes the card is at a slight angle, but it works fine and is secure, so we will ignore that. I also used this time to upgrade the CPU. If you look for 7 year old CPUs on eBay, they are actually not that much money. I went from a Celeron from when I bought the system to a i5-4590. With this new CPU (and breaking a leg on the stock cooler) I ordered a new CPU cooler. That turned into an issue because they sent me the wrong version for an AMD instead of the Intel mount. You can see the very very tiny clearance that the CPU cooler has to the chipset heatsink. I also had this system in the office, since with adding disks to ZFS you need to destroy the pool and rebuild. I had to move all the data off to another system, destroy the array, then move it all back. Dynamically adding disks is always a dream ZFS has had and is always around the corner. Hopefully with OpenZFS 2.0, and the merging of the Linux and Unix code bases, we will get shiny new features like that.

Overall the system has worked well for the last 8 or so years, I have 4TB which is about 30% free still. I could probably clean it out more if I tried. I also have been using OneDrive to backup critical things like family photos, which slightly lowers my need for the system. The homelab AD has all the machines automount a chunk of storage as a shared drive, which makes normal home things and transferring files easier. I will continue to run this, and see how vSAN works for me going forward. I am a bit wary of vSAN running into issues on the consumer level gear I have, so having a whole backup of my VMs on the NAS gives me some peace of mind.

The years of using FreeNAS/TrueNAS were a good jumping off point as we recently got new Netapp Appliances at work, and I was tasked with learning them. Netapp ONTAP uses very similar concepts; instead of zVol you have FlexVol, instead of Datasets you have FlexGroups. Netapp also does some weird things like using Raid-4 or Raid-4 with added protection, instead of a traditional Raid-5/Raid-Z. If you work for a company that has a Netapp and want to learn more about it, I would push you to get the Netapp Simulator. It is a VM image that contains a virtual Netapp to play around with. It’s much better to break a virtual Netapp than a production one.

Mister Project Keyboard Case

Over the holidays I got parts to put together a Mister FPGA system (project home, sub-reddit). This is an open source project which allows to run classic game consoles and classic computers in hardware on the FPGA. Instead of normal emulation, where in software you pretend to be the CPU/GPU/Hardware of what the original code would run on, this projects has a Field Programable Gate Array that can change itself into being that hardware. By doing this, the system can get very very close to 100% accurate running of these old systems. Each system is created into a “core” which is applied to the FPGA to run software. The community around the Mister Project is growing, there are some projects to get systems like N64, and PSX working on this platform; but the Mister Project standardized a while ago on one FPGA, which may not be up to that task once the new cores are done because of their size and complexity.

There are many nice features that have been built out for the projects over the years. Standardizing around the DE10-Nano FPGA, there are many add-on boards you can get for it. From additional RAM, to VGA outputs. The FPGA has a ARM CPU that manages the base system, that supports Wi-Fi cards, Bluetooth, and has automatic updating features. With an IO board that most people who use the project get, you can click a button to reboot the system, or another to go back to the main menu and select the core you want to run. I have a standard IO board, USB Hub, and 256MB of RAM addon. The documentation for the open source project is actually good, with it all centering around the Github Wiki. There are automatic installers for the SD card you need to do the initial ARM side setup.

I was most interested in one of the completed cores, it is a 486DX (project home) with Sound Blaster, and everything you need to run Dos/Windows 3.1/Win 95. Having played many games as a child in that environment, having a 386SX, I was excited to give it a try.

Hardware

When I was thinking of getting the parts for the project, I looked on Thingiverse to see if anyone had put a case up; there are several. The one that caught my eye had an embedded keyboard in it (link, updated case), that one had a note on it that an update to the case was coming soon, and to hold off on printing. The estimate for printing the case was around 24 hours, and I didn’t want to do it twice, so I waited. I reached out to the creator who worked away over the holiday season to get the update out. Myself and another were chatting with him in the comments about printing it, and the creator graciously put up the design, before all the instructions were done so the two of us could start printing.

USB Board, with input against the case

This is the largest thing I have printed on the printer, with my print bed holding up to 220mm, and the case coming in at ~210mm. It printed great. I used PETG instead of PLA plastic to have added resistance to heat. After that, it was screwing parts together, and making a tiny circuit board to support the normal buttons on the top of the case. I ran into a small problem with the updated USB board I have, its input was blocked by the side of the case. The creator had a different revision of the USB board, and thus hadn’t tested with my version. I ordered some cables online and ended up checking the pinouts and making my own header to USB cable, after that it was smooth sailing.

I ordered a collection of M3 screws, to have different sizes. That is the size the case was built around. I also had some screws that do not have heads on them, I was able to use these internal screws to hold some of the boards in. I will put a full list of the parts I ordered below, including the headers for the Mister IO board, which took a bit of research to find.

The USB board, and the Mister FPGA itself need 5V power, the USB board came with a Y cable to breakout a single power brick into the 2 boards, but it was not designed for them to be this far apart. Usually the USB board stacks directly under the FPGA, with this case they sit several inches apart. I ended up getting a 1ft extension cable to be able to make up the difference. While that worked I then got a 2.1×5.5mm barrel connector and socket to put on the back of the case, now it has a nice flush place on the back of the case to plugin the power for the USB board. I am using a SD card right now for all my storage. The 128gb it gives me is fine to get started. I have seen people with setups that have a SATA SSD in the case with a USB adapter. This case supports in in the spot under the FPGA. If you load the system up with a ton of classic games and systems, that may be needed.

Setup and Software

Setup I used the Mister “Mr Fusion” Windows installer. Popped in a 128gb micro SD card, and a few minutes later it was ready to go. It takes about 10 minutes the first time it is setup and has internet access to download all the “updates” which is every core registered with the project. The Wi-Fi and Bluetooth dongles were automatically detected, I just had to enter Wi-Fi credentials.

I think the case came out nicely, and have been having fun installing things on it and playing with it. While the 2GB virtual hard drive I gave Windows 95 is on a SD card and gives decent read/write speeds there, the FGPA 486 at 90mhz still struggles a bit with Windows 95. People are working on getting the perf better. Improvements like recently added L2 caching can help. With the click of a button I can swap it over to Windows 3.1 on a different virtual drive and load up my DOS collection. One of the benefits of the Mister project as mentioned is the ARM management layer, I can add files to a ISO, then SCP it to the system. You can also use any size SD card for all your images, and when you want a new virtual hard drive, its a few clicks away. Then mounting those images is straight forward. Windows 3.1 and 95 are supposed to be able to open a null modem connection to the host and transfer files/browse the internet that way, I have yet to get this working.

After all the posts I have done on here recently I couldn’t just play around with the 486. I also got the Mac Plus side of the house running. You can run with 512kb, 1mb, or 4mb of RAM. It has a 20MB HDD, and 2 floppy drives. There is also a Turbo mode, which we obviously need because turbo! And because classic Macs can be slow…

All together it is a fun project I continue to play with. I like being able to play with classic systems like a Commodore 64 without it using up space in my small apartment. The ease of loading software also makes for a very enjoyable experience. If anyone has experience with this, or has questions feel free to comment below!

Parts List

I tend to get packs of things when working on a project like this. I can use them later and it gives be options with several sizes. I did not include the Mister Board and IO board since there are many sellers of those standard parts, I did include the USB and Bluetooth because they have been proven to work.

Wireless 802.11AC USBhttps://www.amazon.com/gp/product/B01MY7PL10/
Bluetooth Adapterhttps://www.amazon.com/gp/product/B07J5WFPXX/
Keyboardhttps://www.amazon.com/gp/product/B06XGHP35N/
LEDs for breakouthttps://www.amazon.com/gp/product/B07G49PJLG/
Buttons for breakout boardhttps://www.amazon.com/gp/product/B071KX71SV/
Board for breakouthttps://www.amazon.com/gp/product/B078W1F3PC/
USB 3.0 extensionhttps://www.amazon.com/gp/product/B087BK3WRB/
USB 3.0 90 degree connectorhttps://www.amazon.com/gp/product/B0793NQRYZ/
Ethernet 90 degree connectorhttps://www.amazon.com/gp/product/B01MSIE5HO/
USB Board Cable Headerhttps://www.amazon.com/gp/product/B01DP13EDA/
USB Cable header to micro usbhttps://www.amazon.com/gp/product/B07WLM7Q6Z/
1×5 pin IO board cablehttps://www.amazon.com/gp/product/B072VM9HVC/
1×7 pin IO board cablehttps://www.amazon.com/gp/product/B01IZDBXF4/
Power port and cableshttps://www.amazon.com/gp/product/B01NBOP46E/
Power Extension Cablehttps://www.amazon.com/gp/product/B01M6C24XG/
Internal Screwshttps://www.amazon.com/gp/product/B07QFVRS77/

Homelab: Network 2020

As a younger person in my career I got a few Cisco certs, the study material was available to me, and I thought it would be an interesting thing to learn. At this point, I have had a CCNP for almost 10 years and I still enjoy messing with networking even if it is not my day to day job. While I historically have used Cisco a lot, there are many other brands out there these days that have good gear, some even low power enough that I can run at home and not worry about the power bill. Below is my current home setup, it has changed a lot over time, and this is more of a snapshot than a proper design document. That is what homelabs are for right? Messing around with things.

Firewall

The firewall I am running is one I have mentioned on here before. The system itself is an OLD Dell Optiplex 990, released in early 2011 and soon to have its 10th birthday! Idling at ~30 watts, it works well for what I need it to with a gen 2 i7, and 8gb of ram. I added a 4 port Intel gigabit ethernet card to it, which allows for more ports and hardware offloading of a lot of IP tasks.

I looked around at different firewall OS options. Pfsense is the obvious one, but I found its interface lacking. (I use Palo Alto Networks firewalls at work and that interface/flow is more what I am used to) Opnsense is a bit better, but still leaves something to be desired on the UI side. Then I tried a Home License of Sophos XG. It is free as long as you stay at 4 cores or less, with 6 or less GB of ram used; you are given an “evaluation license” until 2099-12-31, if it runs out I will ask for an extension. For more than a year I have been enjoying it, the interface is slick, and you get the enterprise auto patching built in. In the time I have run it, I have had 1 zero-day attack on the product and it was immediately patched without me having to login. I use it as my home firewall between vlans, a DHCP server, and I also have IPSec and SSL VPNs for when I am away from home. The system does DNS for the house (on the less secure vlans, AD does those) and allows for block lists to be used. This is like a pihole but built into the product.

There are a few things it does a little odd, but I enjoy not having to go and write weird config files on the backend of some Linux/BSD to have my firewall work. I have it hooked into AD for auth, and that way I can login with a domain admin, and allow users who have domain accounts to VPN into home. It has been VERY stable, and usually only reboots when I tell it to do an update, or that one time the ~10 year old PC blew a power supply.

Cross Room Link

At the start of the year, I was running a Ubiquiti Wi-Fi mesh at home, it got decent speeds, and allowed me to not run wires over the apartment. The access points used were these models, link. They were only 2×2 802.11AC Wave 1; got decent speeds (around 400mbps), but being in a New York City apartment, I would get interference sometimes, even on 5ghz. The interferences would cause issues when playing games or transferring files. The bigger issue was my desk with a bunch of computers, and the firewall were on different sides of this link, meaning any data that was on a different vlan had to go over and back on this Wi-Fi link. On top of that, I will mention I basically HAVE to use 5ghz, I did a site survey with one of the APs and the LOWEST used 2.4ghz band near me was 79% utilized…

Anyway I started looking around for what I would replace it with, I always thought fiber could be a way to go since its small and if I could get white jackets on it, then it would blend in with the wall. I spend a few weeks emailing and calling different vendors trying to find someone who would do a single cable run of white jacketed fiber. Keep in mind this is early 2020 with Covid starting up. Lots of places could not do orders of 1, or their website would say they could and later they would say they couldn’t and refund me. Finally I found blackbox.com, I have no affiliation with them they just did the job quickly and I appreciate that. I got a 50 meter or so run, and was able to install that with the switches below.

Switches

Now that I had the fiber I needed some small switches I could run at home. After looking at what others have on reddit and www.servethehome.com I found the Ruckus ICX 7150-C12P. A 14 1gb/s ethernet, switch with 2 1/10gb SFP+ ports. The switch is compact, fan-less, and has 150 watts of POE! I can run access points, and cameras off of it without other power supplies. I have learned to look for before buying this sort of gear off eBay to try to get the newest firmware. With Cisco and HPe they love to put it behind a wall that requires an active support contract. Not only does Ruckus NOT do that, they have firmware available for their APs that allows it to run without a controller, more on that later.

I ended up buying 1 of the Ruckus switches “used” but it came sealed in box. Then getting another one broken, after seeing some people online mention they sometimes over heat if it was somewhere without proper ventilation and that can kill their power supply. The unit is fan-less, but the tradeoff there is nothing can sit on top of it, because it needs to vent. I was able to get one for around $40, then a new power supply for $30, all in I spend $70 for a layer 3 switch with 10gb ports! Now I have these 2 units on opposite sides of the room, in a switch stack. This way they act as one and I only need to manage “one switch”.

With the Ubiquiti gear no longer acting as a Wi-Fi link, which I have written about before, I only had one of the APs running. As mentioned before the access point was only 2×2 antennas and 802.11AC Wave 1. I was pondering getting a new Wi-Fi 6 access point, while looking around someone on reddit, again, suggested looking at Ruckus access points. Their antenna design is very good, and with their “Unleashed” firmware you get similar features to running a Ubiquiti controller. After looking at the prices I had to decide if I wanted to go Ubiquiti with Wi-Fi 6, and wait for their access points to come out, or get something equally priced but more enterprise level like a 802.11AC Wave 2 access point (like a Ruckus R510 or R610 off eBay).

I recently had a bad experience with some Ubiquiti firmware, then all of a sudden they killed Ubiquiti Video with very little warning, and some the more advanced functions I would want to do are either minimally or not documented with Ubiquiti. One could argue that I am used to enterprise gear, and Ubiquiti is more “pro-sumer” than enterprise; thus, I should not be upset at the lack of enterprise features. That made me decide to try something new. I ended up getting a Ruckus R610 off eBay and loading the “Unleashed” firmware on it. I can say the speeds and coverage is much better than the older access point. It is 3×3 802.11AC Wave 2, and with most of my devices still being 802.11AC I figured that was a good call.

One feature of the Unleashed firmware is it can manage all your Ruckus hardware. The web management portal has a place to attach your switches as well, and do some management of them there. I have been scared to do this, and coming from a traditional CLI switch management background have yet to do so.

Unleashed Home Screen

I was able to POE boot the AP just like I did with Ubiquiti, converting the firmware was easy, and there are many guides on Youtube for it. The UI does not have the same polish that Ubiquiti does, but the controller is in the AP itself which is very nice. There is a mobile app, but it is fairly simplistic. The web interface allows for auto updating, and can natively connect to Active Directory making it very easy to manage authentication.

There are 3 wireless networks in the home, 1 is the main one for guests, with their 6 year old unpatched android phones, that has a legacy name and meh password, that way I don’t have to reset some smart light switches Wi-Fi settings. This is where all the IoT junk lives. There is one with a better password that connects to the same vlan I am slowly moving things in the house over to, at least the key is more secure. Then there is the X wireless network, this one is not broadcast and has 802.1X on it. When a user authenticates with their domain creds, depending on the user and device I send them to a different vlan. This is mostly used for trusted devices like our laptops, and iPad when I want to do management things. This network for my domain account allows me on the management network.

10gb/s Upgrade!

The latest upgrade I have embarked on was 10gb/s. I moved my active VM storage off of the NAS to Storage Spaces Direct for perf. While the NAS has worked well for years, the 7 – 3 TB disks do not give fantastic IOPS when different VMs are doing a lot of transactions. After lots of thought and trials I went with Storage Spaces Direct and will write about it later. The main concern was that it allows all the hypervisors to have shared storage and keep it in sync, and to do that they need good interconnects. This setup is the definition of, lab-do-not-do-in-prod, with 3 nodes each with 3 SSDs over USB 3. I knew with USB 3 my theoretical bottle neck was 5gb/s, which is much better than the 1gb/s I had, that also had to be shared with all server and other traffic.

First I had to decide how I would layout the 10gb/s network, while the ICX 7150 has 2 – 10gb/s ports, 1 is in use to go between the switches. After looking around and comparing my needs/wants/power/loudness-the-significant-other-would-put-up-with I got a MikroTik CRS309-1G-8S+IN. I wasn’t super excited to use them, since their security history is not fantastic, but I didn’t want to pay a ton or have a loud switch. I run the switch with the layer 2 firmware, and then put its management interface on a cut off vlan, that way it is very limited on what it can do.

After that I got a HP 10gb/s server cards, and tried a Solarflare S7120. Each had their ups and downs, the HPs are long and would not fit into some of the slim desktops I had. But when the would work, like in the Dells, they would work right away without issues. The Solarflare are shorter cards which is nice, but most of them ship with a firmware that will not work on some motherboards or newer operating systems. For these you need to find a system they work in, boot to Windows (perhaps an older version) then flash them with a tool off their website. After that they work great. I upgraded the 3 main hypervisors, and the NAS. I have seen the hypervisors hit 6.1gb/s when syncing Storage Spaces. With memory caching I can get over the rated disk speed.

That is the general layout of the network at this point. I am using direct attached cables for most of the systems. I did order some “genuine” Cisco 10gb/s SFP+ off Amazon for ~$20, I didn’t believe they would be real, but I had someone I know who works at Cisco look them up and they are real. Old stock shipped to Microsoft in 2012 or so, but genuine parts. The Ruckus switches and these NICs do not care which brand the SFPs are, so I figured I would get one I knew. The newer Intel NICs will not work with non Intel SFPs so look out.

To summarize, the everything comes in from my ISP to the Sophos XG box, then that connects to a port on one of the Ruckus switches. Those two Ruckus switches have a fiber link between them. Then one of the SFP+ ports on the Ruckus switch goes to a SFP+ port in the MikroTik switch. All the hypervisors hang off that MikroTik switch with SFP+ DACs. Desktops, video game consoles, and APs all attach to 1gb/s ethernet ports on the Ruckus switches. I have tried my best to label all the ports as best I can to make managing everything easier. I’m sure this will evolve more with time, but for my apartment now 10gb/s networking with a Ruckus R610 AP has been working very well.