Hardware

Ruckus ICX 7150-C12P Switch Repair

A while ago I purchased a Ruckus ICX 7150-c12p off eBay to use at home. It gives 14x1gb/s ports, and 2 SFP+ ports. The SFP+ ports are limited to 1gb/s by default, and there is a honor system license for upgrading them to 10gb/s. These switches go for $600 – $1200 depending on where you get them and which license you get with it (1gb/s vs 10gb/s). The switch is also POE, and can do 4 POE+ (30 watt) ports. I had one of these switches and it worked great. I wanted to get a second one to replace the WiFi link I was using across my apartment with a fiber link.

Instead of paying ~$250, which was their going rate on eBay; I saw a forum post about replacing this models power supply, and thought I would give that a shot. I got a broken switch for $45, and then a PSU for $50. The PSU I used was a SL Power LB130S56K 56V 2.32 130W. Armed with someone’s photos of doing this repair it ended up going fine. The hardest part of the whole operation is that the pins going onto the main board are reversed from what the power supply comes with, so you need to flip them. I have been running the unit for almost 2 years now without issue.

This model of switch is great because of its features and is fanless. The fanless-ness part of it is nice for homelabs near your desk, because the switches are silent. Because they are fanless, they cant have anything put on top of them, and need some room to breath. I think a lot of the ones you see online dead are because someone didn’t give it enough air, and the PSU died. Note when looking for a similar dead switch on eBay, you really want the seller saying “when plugged in nothing happens”, not “it periodically blinks” because that could be bad ram and its in a boot loop.

Having run two of these switches for over a year, I can give some feedback. I really like them. I have the two I have in a stack, I login once and manage both. When it comes time for firmware updates you SCP the file to the management IP, and it downloads the file to both, and then flashes and reloads. I came from using Cisco gear usually, or sometimes Arista; the CLI is a bit different, and Ruckus handles VLAN setup a bit weird, but once you get used to it, it makes sense. They are solid switches, with POE, that you can set and forget for a while.

New PC Build 2022

Having built my PC a few years ago, I was thinking about upgrading it, but with chip prices being what they are, and graphics cards costing more than a new car, I thought I would wait. Then a friend of mine happen to have an AMD 5800X that he was willing to give me a deal on… After years of having the custom case, while it was nice, it was HUGE and made it difficult to setup a desk in a tiny New York apartment. I used this opportunity to shrink a bit, and update some of the components. I say some, because some of them (like the graphics card) were going to stay to save cost.

Old case being taken apart, its a bit of a mess

At this point, this post is mostly a standard PC build, with a few hiccups along the way. Looking on the Small Form Factor PC subreddit, and https://caseend.com/ (a website for small pc cases), I settled on the ZZAW C3. It is relatively small, supports Micro-ATX, full size graphics cards, and all-in-one water coolers in a ~22 liter case. I always try to get Micro-ATX over Mini-ITX for my desktop because you get more PCI slots; as well as 4 DIMMs for RAM instead of 2. I also wanted to try one of these all-in-one water coolers, since I never had and thought I could get good results (and a quiet case).

The case came nicely wrapped. There were not a lot of instructions on how to put the case and system together, you kind of just had to know. That took a bit of trial and error. There are a few screws that are very similar sizes, and not labeled. The whole setup went smoothly after that except that getting the cooler to fit in the case was a bit of a challenge, and the motherboard… So, the motherboard… I got a AsRock X570 Pro4 motherboard, it had all the features I wanted. I get the case put together, I installed my previous power supply, I kept the RAM and graphics card, moved my SSD over (I had a PCIe Gen 4 Samsung 980 Pro on order, but it hadn’t come in yet), put thermal paste on the CPU, install the cooler AND… nothing. System will not boot.

There are lights on the motherboard saying CPU, RAM, and DISK failed. I start searching online and trying different things. A bit in, someone says “make sure your motherboard is updated to support 5000 series AMD”. I just got this board, its fairly new, it has to work right? Well turns out you need BIOS version 3.20 to support the CPU I have, AsRock is at 4.20 for this motherboard. After taking my old 2600X out of my old motherboard, taking out the new CPU, cleaning the thermal paste, reseating, re-thermal pasting the 2600X, turns out the board shipped with version 3.10… One revision before what I needed. I updated the BIOS, then swapped the CPU back, doing all those steps again. This time, the system worked.

I later got the Samsung 980 Pro SSD, which was a tiny bit of a headache to migrate over because I had Bitlocker on and trying to relocate the bootloader was not straight forward. I kept getting “Boot Device Not Found” and eventually somehow doing a system restore to before the last Windows Update got Windows 10 to reinstall the bootloader on the drive. I have no idea why that would have fixed it. I had disabled Bitlocker before starting this ordeal, but that just does it one reboot, I really needed to just fully disable it to save me typing in the recovery pin many times.

The system is working well. The only remaining issue, that is very odd, but I am just living with is whenever the system attempts to do a restart, it freezes before coming back. It properly shuts down, and if I hold the power button and then turn it back on everything is fine. But it refuses to gracefully restart. It also will fully shutdown fine. Odd…

Building a Pac Man Battle Royale Table

This is a post I was working off and on for several years. It is something I always was meaning to finish, and got very near the end, the Covid and life happen. Instead of throwing it out I figured I would do some small edits then put out as is.

(2017) Friends and I always found the game Pac-Man Battle Royale to be fun, but when we went to look at the price of a cabinet they were $5,000! Worth it if you are a bar or arcade, but for a few friends playing games a bit over the top. After a evening at Barcade, I started the trek to see if I could build one myself for less.

Original Table

The first step was figuring out where I could legally get the game, a number of sites offered the ROM but that is not what I was looking for. Next I found it was ported and available on Steam! We are in luck! The game is available for less than $10! But once I start it, I am greeted by a “fun” border and changes they made to the screen for the Windows version.

Steam Version
Steam Version
Arcade Version
Arcade Version

On the left you can see the Steam version; there is a border, along with all the player text is facing one direction. On the right is the original version, where the screen goes to the edge, and the 3rd and 4th players face the other direction for when players are standing around the table. This version also has been made to play with Xbox 360 wired controllers, it works with keyboard but a lot of the interface seems to be built around that.

The solution, a brave hero on Github made a modified DirectX9 driver that edits the game screen as you play! https://github.com/vikbez/pacbrcade After installing this file, I was able to get the game to look just like the original! Below are the before and afters from the repo, and I can attest it does a great job! Throw in a script to start the game at startup of a Intel Compute Stick (a full pc on a HDMI dongle), and this was ready to go.

(2021) Now it came time to build the cabinet, I hadn’t built something thing big before and didn’t really have a large plan. The hope was to have this live at the office. Being in NYC it needed to be able to fold up, and then be put somewhere when not in use. Part of my plans were to give it folding legs, and one side of the table should be a rest, so the legs can be folded and it can be put on its side somewhere out of the way.

Most of the construction was actually done in one weekend in 2017, I got some 2x4s and went to the maker space I am a member of fat cat FAB LAB – NYC Hackerspace to cut the boards down to the sizes. Now this was a bit of a rough day because I was in NYC, and I am literally grabbing large 2x4s (some are 6 or 7 feet long) then walking a few blocks to the woodshop as people are drinking on a Saturday around me on the streets, then cutting them and walking several blocks to the office. After all is said and done, I used a normal drill and some wood screws to put it all together. I then used yellow vinyl wrap to make the table a bit more dressed up. Getting bubbles out of the vinyl wrapper, on a wood surface that isn’t completely flat was a bit of a challenge. I had a area for the screen to go in, then 4 sides; 2 with cup holders, and 2 with joysticks.

I used Teensy micro controllers as the joysticks, they emulate joysticks on a computer and you an make any input trigger any signal you want. I wanted to add some more style to the unit so I made Player 1 through 4 acrylic panels to go around the joystick and button. These were laser cut at the same woodshop I used before. I also ended up getting craft beer labels, and putting it on the inside of the buttons; that gave the unit a little more character. I put the rubber molding along the side of the unit, as you would expect from any good arcade system. After installing an old screen I found around I toped it with a sheet of clear plexiglass. This was a learning experience of plexiglass scratches easily and can crack if too much pressure is put in say a screw hole.

The whole thing worked, it booted up on the Compute Stick, auto loaded Steam, and started the game. The main issue that had me put it away for a while was the joystick handling. The joysticks I had were 8 way joysticks; they could go to the 4 sides but also to all the corners, we didn’t want the corners. In a game like Pac-Man, going to the corners of the joystick made the character either not move or go in one of the 2 directions you were facing.

The bottom of the joysticks had a plastic piece which allowed the rod of the joystick to go in certain directions. On the bottom of the units I had it was a empty square. I wanted it to be a diamond, this would have forced the player in going one of 4 directions instead. The plan was to design a piece then 3D print it and attach it to the bottom of all the joysticks.

With that I put the system into a closet, where it sat for several years. There was the issue on top of all this of being in NYC and there was no place to put the thing where it would not be in the way. I worked on this before I had a 3D printer and could have put the piece together quickly.

In the end it was fun to put something together quickly like this. I got to do some bigger wood working and vinyl wrap something. One take away I have from it was the momentum of a project cant be a very motivating thing. I started and put most of the system together in a single weekend. Then the last few percent of the project, getting the joysticks correct, I lost the energy (and didn’t have a place to put the thing) and went onto other projects. I try to use that motivation to push through projects when possible, and use this project as a reminder to do that.

Briel Computers Replica 1 Plus

Kit

I recently ordered the Briel Computers Replica I Plus, a Apple I clone. Instead of the originals big board to do a lot of NTSC generation, it uses a more modern single chip. The shipment came in a small box, and with everything I needed. The creator of the kit did a great job including everything you need, down to including an anti-static strap! The project came with some solder, but not nearly enough for everything, I think it was thicker to go with the structural points. Briel Computers sells the kit through ReActiveMicro.com. At $135 it is one of the less expensive kits I have had, but also comes with just the board. If you want a case that needs to be 3D Printed (more on that later).

ReActiveMicro points you over to the Project Wiki for more information, there is a ton there and a link to someone putting the project together. I found this easier to follow along and do than reading the instructions.

The kit was fairly easy and straight forward; I ran into a few small issues around the PS/2 port since the solder points are close together. Getting the few connector ports in can be a bit difficult with a few tiny pins and getting them in the board. As long as you have patience, then you can get through it.

I got it all together, and the board started the first try. I did have the same issue the person who made the video had; I was getting a lot of noise and characters added to the screen. I reflowed a lot of the sockets, and made sure all the socketed chips were fully seated. That cleared up the garbage at startup. The wiki also has some other notes on noise issues the board can show.

I also could not find a PS/2 keyboard in the house, and all the USB keyboards I had didn’t seem to like the USB->PS/2 Adapter. I am not very surprised by this because I didn’t have any very simple, older keyboards.

The USB port that is used for power is also a serial device for a PC/Mac. I plugged into that and got the serial driver working from SparkFun website, they produce the module. Then the output worked well, and I could enter BASIC on the board!

Case

I wanted to put the board in some sort of case, and after searching online I couldn’t find any. I thought I would throw something together quickly that I could put the board in. I took some measurements and threw together a V0 of the case. One small issue was I didn’t account for the RCA jack the video comes out of little let that sticks out. Instead of spending another 7 hours printing a new one, I used a little saw I have to cut a hole out.

Part of my thought of creating a case was to have something I could put the board in, then store it in a cabinet or shelf and not be worried that the board would get damaged. I also made a case that can go over the entire unit to protect it in storage.

Again, looking back small design things could have been changed, like flip the name of the project in the case, so looking at it in the protective cover, the text would be right right way. Getting the scaffolding out of the protective case was not the easiest of things. I designed the protective case with a rail that brings the edge of the mounting board into a locking position when you slide it in. I have to say, that was a nice aspect to the design. It took over 6 hours to print though.

MisteRdeck MIDI Control Desk

I have been enjoying 3D printing projects recently. I saw a little control board for changing audio levels, and having hotkeys while playing games. The printing took a good long while, and I had to edit some of the parts to work with the parts I found currently on Amazon. I will post the parts list below. The soldering was straight forward, and the project came with a PDF that had good instructions. This also turned into a good opportunity for me to use the new Wiring Pencil, which worked surprisingly well.

For hardware, I am using a Teensy; the Teensy can be a USB keyboard or MIDI device or joystick or serial over the USB connection. The project comes with a premade Arduino file to run it as a MIDI controller. I had not worked before with MIDI input like this, but it seemed the best path forward compared to trying to emulate a keyboard and hitting odd key combinations. Or the alternative of writing something that output serial data then finding, or writing, a daemon for my PC to listen to that device.

For software, I looked at several pieces of software to use the keys and sliders with. I looked at software like VoiceMeeter. While overall that worked, it was very inflexible, and had a giant interface for things I didn’t want to use. Then I found Midi-Mixer, a passion project by a single dev and it is EXACTLY what I needed. The sliders can control single app volume, which is easy to select. And the buttons can be programmed for anything! And easily with a GUI instead of conf files like some other open source projects.

Overall I am enjoying the finished project. It sits next to my keyboard, and allows easy changing of levels while playing games. I added little rubber feet I had laying around so the plastic housing doesn’t slide around on the desk.

To fit the sliders, I needed to modify the knobs, here is the modified versions that work with the sliders I ordered below: MisteRdeck Knob Remix by danberk – Thingiverse.

Make: Makes of MisteRdeck – Arduino-based MIDI Stream Deck by danberk – Thingiverse

Example courtesy of midi-mixer.com

Parts

PartURLPrice
Teensy v3.2Teensy USB Development Board (pjrc.com)$19.80
Gateron Yellow SwitchesGateron Yellow Linear Switches | Kinetic Labs$16.10
Slidershttps://www.amazon.com/gp/product/B079ZP3LS5/$11.99 x 2
Diodeshttps://www.amazon.com/gp/product/B06XB1R2NK/~$6
Key Capshttps://www.amazon.com/gp/product/B01M023NFK/$7.50 x 2

Homelab: NAS 2021

One piece that sits at the heart of my Homelab is the NAS I have. This is actually the same NAS I have written about years ago, looking back on that post brought back memories of the pervious system and Server 2008 that I didn’t recall. In the last year I have added several drives and a new network card to this box, I thought that as well as my experience running FreeNAS, now TrueNAS Core, over 8+ years was worth discussing.

When I built out that box, I had 5x3TB drives, each around $125 dollars. Now those same drives are $40. The rough rule of thumb I was always told is 1GB of RAM for ZFS for every TB of storage you have. So I maxed the mini-itx motherboard out at 16GB of RAM to get as close as I could get. This let met run basic services and I was running a few small VMs/Jails on the box. This did cut into the RAM I had available, but was a nice feature. This allowed me to run the Unifi controller without another system running. Back then, Raspberry Pis came with 256MB of RAM, making it not ideal to run too many services. I later would end up moving all of those to dedicated Raspberry Pis then later VM hosts.

These 5 disks served me well for a while; I every year or two would have a drive die, and it got cheaper and cheaper to replace them. I use this NAS for backups from my Windows desktop, and Macbook. Time Machine backups over the network to Macs works very well with TrueNAS. I ended up getting a smaller version of this box for my parents home, and sister, you can run the OS off a USB with a single or 2 small hard drives in a box like an Intel NUC, then have it always backup their PCs. Reminding people “plug in that USB drive” to backup seems to never stick. TrueNAS offers one click updates, with optional automatic checkin; this makes keeping the system up to date easily.

There have been reports of recent corruption with 12.0, but I have not seen that. Also there was a bug where you could get a banner saying “THIS IS A TESTING RELEASE NOT FOR PRODUCTION” on a production branch, so that is fun. These days those backups, and my Veeam backups are done to the NAS. I tried to use it as a iSCSI and then a NFS target, but the IO was a bit too much for these old spinning drives. Now I use vSAN, as mentioned, which has performed well for VMs, that leaves the NAS just as dumb storage for Veeam. Veeam is a good product that makes it very easy to backup VMs, I will probably write an article on it later. The software has a free 10 VM backup license for Homelabs.

In 2020 I was using a high percentage of the storage for backups and VMs, and was pondering upgrading. I didn’t want to throw down enough money to build a whole new system, and I liked this case a lot, so I started to look at what I could do to add to it. I was using 5 drives, but the case technically supports 7, with 2 being on the bottom. The issue was, I didn’t have enough SATA ports to add to the system. This brings me to one of the scariest, worst, best, cards I have bought. This card, adds 4 ports through a mini PCI-E connection. It actually works really well, with the drives coming up like any other, it gives you 1 PCI E Lane at roughly 2.5Gbp/s for my version. I have 2 drives of the now 7 I have in a RAIDZ2 (RAID 6), and for over a year it has worked well. The one other thing I added to the box was a 10GB networking card, I did a push a bit ago to move most of the Homelab server stuff to 10GB, and this box was part of that. TrueNAS is built on FreeBSD, and has good hardware compatibility, I got an old Intel X520 for compatibility and ease. I have seen it get near 5gbit/s, averaging closer to 2gb/s with writes.

First of all, yes the card is at a slight angle, but it works fine and is secure, so we will ignore that. I also used this time to upgrade the CPU. If you look for 7 year old CPUs on eBay, they are actually not that much money. I went from a Celeron from when I bought the system to a i5-4590. With this new CPU (and breaking a leg on the stock cooler) I ordered a new CPU cooler. That turned into an issue because they sent me the wrong version for an AMD instead of the Intel mount. You can see the very very tiny clearance that the CPU cooler has to the chipset heatsink. I also had this system in the office, since with adding disks to ZFS you need to destroy the pool and rebuild. I had to move all the data off to another system, destroy the array, then move it all back. Dynamically adding disks is always a dream ZFS has had and is always around the corner. Hopefully with OpenZFS 2.0, and the merging of the Linux and Unix code bases, we will get shiny new features like that.

Overall the system has worked well for the last 8 or so years, I have 4TB which is about 30% free still. I could probably clean it out more if I tried. I also have been using OneDrive to backup critical things like family photos, which slightly lowers my need for the system. The homelab AD has all the machines automount a chunk of storage as a shared drive, which makes normal home things and transferring files easier. I will continue to run this, and see how vSAN works for me going forward. I am a bit wary of vSAN running into issues on the consumer level gear I have, so having a whole backup of my VMs on the NAS gives me some peace of mind.

The years of using FreeNAS/TrueNAS were a good jumping off point as we recently got new Netapp Appliances at work, and I was tasked with learning them. Netapp ONTAP uses very similar concepts; instead of zVol you have FlexVol, instead of Datasets you have FlexGroups. Netapp also does some weird things like using Raid-4 or Raid-4 with added protection, instead of a traditional Raid-5/Raid-Z. If you work for a company that has a Netapp and want to learn more about it, I would push you to get the Netapp Simulator. It is a VM image that contains a virtual Netapp to play around with. It’s much better to break a virtual Netapp than a production one.

Mister Project Keyboard Case

Over the holidays I got parts to put together a Mister FPGA system (project home, sub-reddit). This is an open source project which allows to run classic game consoles and classic computers in hardware on the FPGA. Instead of normal emulation, where in software you pretend to be the CPU/GPU/Hardware of what the original code would run on, this projects has a Field Programable Gate Array that can change itself into being that hardware. By doing this, the system can get very very close to 100% accurate running of these old systems. Each system is created into a “core” which is applied to the FPGA to run software. The community around the Mister Project is growing, there are some projects to get systems like N64, and PSX working on this platform; but the Mister Project standardized a while ago on one FPGA, which may not be up to that task once the new cores are done because of their size and complexity.

There are many nice features that have been built out for the projects over the years. Standardizing around the DE10-Nano FPGA, there are many add-on boards you can get for it. From additional RAM, to VGA outputs. The FPGA has a ARM CPU that manages the base system, that supports Wi-Fi cards, Bluetooth, and has automatic updating features. With an IO board that most people who use the project get, you can click a button to reboot the system, or another to go back to the main menu and select the core you want to run. I have a standard IO board, USB Hub, and 256MB of RAM addon. The documentation for the open source project is actually good, with it all centering around the Github Wiki. There are automatic installers for the SD card you need to do the initial ARM side setup.

I was most interested in one of the completed cores, it is a 486DX (project home) with Sound Blaster, and everything you need to run Dos/Windows 3.1/Win 95. Having played many games as a child in that environment, having a 386SX, I was excited to give it a try.

Hardware

When I was thinking of getting the parts for the project, I looked on Thingiverse to see if anyone had put a case up; there are several. The one that caught my eye had an embedded keyboard in it (link, updated case), that one had a note on it that an update to the case was coming soon, and to hold off on printing. The estimate for printing the case was around 24 hours, and I didn’t want to do it twice, so I waited. I reached out to the creator who worked away over the holiday season to get the update out. Myself and another were chatting with him in the comments about printing it, and the creator graciously put up the design, before all the instructions were done so the two of us could start printing.

USB Board, with input against the case

This is the largest thing I have printed on the printer, with my print bed holding up to 220mm, and the case coming in at ~210mm. It printed great. I used PETG instead of PLA plastic to have added resistance to heat. After that, it was screwing parts together, and making a tiny circuit board to support the normal buttons on the top of the case. I ran into a small problem with the updated USB board I have, its input was blocked by the side of the case. The creator had a different revision of the USB board, and thus hadn’t tested with my version. I ordered some cables online and ended up checking the pinouts and making my own header to USB cable, after that it was smooth sailing.

I ordered a collection of M3 screws, to have different sizes. That is the size the case was built around. I also had some screws that do not have heads on them, I was able to use these internal screws to hold some of the boards in. I will put a full list of the parts I ordered below, including the headers for the Mister IO board, which took a bit of research to find.

The USB board, and the Mister FPGA itself need 5V power, the USB board came with a Y cable to breakout a single power brick into the 2 boards, but it was not designed for them to be this far apart. Usually the USB board stacks directly under the FPGA, with this case they sit several inches apart. I ended up getting a 1ft extension cable to be able to make up the difference. While that worked I then got a 2.1×5.5mm barrel connector and socket to put on the back of the case, now it has a nice flush place on the back of the case to plugin the power for the USB board. I am using a SD card right now for all my storage. The 128gb it gives me is fine to get started. I have seen people with setups that have a SATA SSD in the case with a USB adapter. This case supports in in the spot under the FPGA. If you load the system up with a ton of classic games and systems, that may be needed.

Setup and Software

Setup I used the Mister “Mr Fusion” Windows installer. Popped in a 128gb micro SD card, and a few minutes later it was ready to go. It takes about 10 minutes the first time it is setup and has internet access to download all the “updates” which is every core registered with the project. The Wi-Fi and Bluetooth dongles were automatically detected, I just had to enter Wi-Fi credentials.

I think the case came out nicely, and have been having fun installing things on it and playing with it. While the 2GB virtual hard drive I gave Windows 95 is on a SD card and gives decent read/write speeds there, the FGPA 486 at 90mhz still struggles a bit with Windows 95. People are working on getting the perf better. Improvements like recently added L2 caching can help. With the click of a button I can swap it over to Windows 3.1 on a different virtual drive and load up my DOS collection. One of the benefits of the Mister project as mentioned is the ARM management layer, I can add files to a ISO, then SCP it to the system. You can also use any size SD card for all your images, and when you want a new virtual hard drive, its a few clicks away. Then mounting those images is straight forward. Windows 3.1 and 95 are supposed to be able to open a null modem connection to the host and transfer files/browse the internet that way, I have yet to get this working.

After all the posts I have done on here recently I couldn’t just play around with the 486. I also got the Mac Plus side of the house running. You can run with 512kb, 1mb, or 4mb of RAM. It has a 20MB HDD, and 2 floppy drives. There is also a Turbo mode, which we obviously need because turbo! And because classic Macs can be slow…

All together it is a fun project I continue to play with. I like being able to play with classic systems like a Commodore 64 without it using up space in my small apartment. The ease of loading software also makes for a very enjoyable experience. If anyone has experience with this, or has questions feel free to comment below!

Parts List

I tend to get packs of things when working on a project like this. I can use them later and it gives be options with several sizes. I did not include the Mister Board and IO board since there are many sellers of those standard parts, I did include the USB and Bluetooth because they have been proven to work.

Wireless 802.11AC USBhttps://www.amazon.com/gp/product/B01MY7PL10/
Bluetooth Adapterhttps://www.amazon.com/gp/product/B07J5WFPXX/
Keyboardhttps://www.amazon.com/gp/product/B06XGHP35N/
LEDs for breakouthttps://www.amazon.com/gp/product/B07G49PJLG/
Buttons for breakout boardhttps://www.amazon.com/gp/product/B071KX71SV/
Board for breakouthttps://www.amazon.com/gp/product/B078W1F3PC/
USB 3.0 extensionhttps://www.amazon.com/gp/product/B087BK3WRB/
USB 3.0 90 degree connectorhttps://www.amazon.com/gp/product/B0793NQRYZ/
Ethernet 90 degree connectorhttps://www.amazon.com/gp/product/B01MSIE5HO/
USB Board Cable Headerhttps://www.amazon.com/gp/product/B01DP13EDA/
USB Cable header to micro usbhttps://www.amazon.com/gp/product/B07WLM7Q6Z/
1×5 pin IO board cablehttps://www.amazon.com/gp/product/B072VM9HVC/
1×7 pin IO board cablehttps://www.amazon.com/gp/product/B01IZDBXF4/
Power port and cableshttps://www.amazon.com/gp/product/B01NBOP46E/
Power Extension Cablehttps://www.amazon.com/gp/product/B01M6C24XG/
Internal Screwshttps://www.amazon.com/gp/product/B07QFVRS77/

Homelab: Network 2020

As a younger person in my career I got a few Cisco certs, the study material was available to me, and I thought it would be an interesting thing to learn. At this point, I have had a CCNP for almost 10 years and I still enjoy messing with networking even if it is not my day to day job. While I historically have used Cisco a lot, there are many other brands out there these days that have good gear, some even low power enough that I can run at home and not worry about the power bill. Below is my current home setup, it has changed a lot over time, and this is more of a snapshot than a proper design document. That is what homelabs are for right? Messing around with things.

Firewall

The firewall I am running is one I have mentioned on here before. The system itself is an OLD Dell Optiplex 990, released in early 2011 and soon to have its 10th birthday! Idling at ~30 watts, it works well for what I need it to with a gen 2 i7, and 8gb of ram. I added a 4 port Intel gigabit ethernet card to it, which allows for more ports and hardware offloading of a lot of IP tasks.

I looked around at different firewall OS options. Pfsense is the obvious one, but I found its interface lacking. (I use Palo Alto Networks firewalls at work and that interface/flow is more what I am used to) Opnsense is a bit better, but still leaves something to be desired on the UI side. Then I tried a Home License of Sophos XG. It is free as long as you stay at 4 cores or less, with 6 or less GB of ram used; you are given an “evaluation license” until 2099-12-31, if it runs out I will ask for an extension. For more than a year I have been enjoying it, the interface is slick, and you get the enterprise auto patching built in. In the time I have run it, I have had 1 zero-day attack on the product and it was immediately patched without me having to login. I use it as my home firewall between vlans, a DHCP server, and I also have IPSec and SSL VPNs for when I am away from home. The system does DNS for the house (on the less secure vlans, AD does those) and allows for block lists to be used. This is like a pihole but built into the product.

There are a few things it does a little odd, but I enjoy not having to go and write weird config files on the backend of some Linux/BSD to have my firewall work. I have it hooked into AD for auth, and that way I can login with a domain admin, and allow users who have domain accounts to VPN into home. It has been VERY stable, and usually only reboots when I tell it to do an update, or that one time the ~10 year old PC blew a power supply.

Cross Room Link

At the start of the year, I was running a Ubiquiti Wi-Fi mesh at home, it got decent speeds, and allowed me to not run wires over the apartment. The access points used were these models, link. They were only 2×2 802.11AC Wave 1; got decent speeds (around 400mbps), but being in a New York City apartment, I would get interference sometimes, even on 5ghz. The interferences would cause issues when playing games or transferring files. The bigger issue was my desk with a bunch of computers, and the firewall were on different sides of this link, meaning any data that was on a different vlan had to go over and back on this Wi-Fi link. On top of that, I will mention I basically HAVE to use 5ghz, I did a site survey with one of the APs and the LOWEST used 2.4ghz band near me was 79% utilized…

Anyway I started looking around for what I would replace it with, I always thought fiber could be a way to go since its small and if I could get white jackets on it, then it would blend in with the wall. I spend a few weeks emailing and calling different vendors trying to find someone who would do a single cable run of white jacketed fiber. Keep in mind this is early 2020 with Covid starting up. Lots of places could not do orders of 1, or their website would say they could and later they would say they couldn’t and refund me. Finally I found blackbox.com, I have no affiliation with them they just did the job quickly and I appreciate that. I got a 50 meter or so run, and was able to install that with the switches below.

Switches

Now that I had the fiber I needed some small switches I could run at home. After looking at what others have on reddit and www.servethehome.com I found the Ruckus ICX 7150-C12P. A 14 1gb/s ethernet, switch with 2 1/10gb SFP+ ports. The switch is compact, fan-less, and has 150 watts of POE! I can run access points, and cameras off of it without other power supplies. I have learned to look for before buying this sort of gear off eBay to try to get the newest firmware. With Cisco and HPe they love to put it behind a wall that requires an active support contract. Not only does Ruckus NOT do that, they have firmware available for their APs that allows it to run without a controller, more on that later.

I ended up buying 1 of the Ruckus switches “used” but it came sealed in box. Then getting another one broken, after seeing some people online mention they sometimes over heat if it was somewhere without proper ventilation and that can kill their power supply. The unit is fan-less, but the tradeoff there is nothing can sit on top of it, because it needs to vent. I was able to get one for around $40, then a new power supply for $30, all in I spend $70 for a layer 3 switch with 10gb ports! Now I have these 2 units on opposite sides of the room, in a switch stack. This way they act as one and I only need to manage “one switch”.

With the Ubiquiti gear no longer acting as a Wi-Fi link, which I have written about before, I only had one of the APs running. As mentioned before the access point was only 2×2 antennas and 802.11AC Wave 1. I was pondering getting a new Wi-Fi 6 access point, while looking around someone on reddit, again, suggested looking at Ruckus access points. Their antenna design is very good, and with their “Unleashed” firmware you get similar features to running a Ubiquiti controller. After looking at the prices I had to decide if I wanted to go Ubiquiti with Wi-Fi 6, and wait for their access points to come out, or get something equally priced but more enterprise level like a 802.11AC Wave 2 access point (like a Ruckus R510 or R610 off eBay).

I recently had a bad experience with some Ubiquiti firmware, then all of a sudden they killed Ubiquiti Video with very little warning, and some the more advanced functions I would want to do are either minimally or not documented with Ubiquiti. One could argue that I am used to enterprise gear, and Ubiquiti is more “pro-sumer” than enterprise; thus, I should not be upset at the lack of enterprise features. That made me decide to try something new. I ended up getting a Ruckus R610 off eBay and loading the “Unleashed” firmware on it. I can say the speeds and coverage is much better than the older access point. It is 3×3 802.11AC Wave 2, and with most of my devices still being 802.11AC I figured that was a good call.

One feature of the Unleashed firmware is it can manage all your Ruckus hardware. The web management portal has a place to attach your switches as well, and do some management of them there. I have been scared to do this, and coming from a traditional CLI switch management background have yet to do so.

Unleashed Home Screen

I was able to POE boot the AP just like I did with Ubiquiti, converting the firmware was easy, and there are many guides on Youtube for it. The UI does not have the same polish that Ubiquiti does, but the controller is in the AP itself which is very nice. There is a mobile app, but it is fairly simplistic. The web interface allows for auto updating, and can natively connect to Active Directory making it very easy to manage authentication.

There are 3 wireless networks in the home, 1 is the main one for guests, with their 6 year old unpatched android phones, that has a legacy name and meh password, that way I don’t have to reset some smart light switches Wi-Fi settings. This is where all the IoT junk lives. There is one with a better password that connects to the same vlan I am slowly moving things in the house over to, at least the key is more secure. Then there is the X wireless network, this one is not broadcast and has 802.1X on it. When a user authenticates with their domain creds, depending on the user and device I send them to a different vlan. This is mostly used for trusted devices like our laptops, and iPad when I want to do management things. This network for my domain account allows me on the management network.

10gb/s Upgrade!

The latest upgrade I have embarked on was 10gb/s. I moved my active VM storage off of the NAS to Storage Spaces Direct for perf. While the NAS has worked well for years, the 7 – 3 TB disks do not give fantastic IOPS when different VMs are doing a lot of transactions. After lots of thought and trials I went with Storage Spaces Direct and will write about it later. The main concern was that it allows all the hypervisors to have shared storage and keep it in sync, and to do that they need good interconnects. This setup is the definition of, lab-do-not-do-in-prod, with 3 nodes each with 3 SSDs over USB 3. I knew with USB 3 my theoretical bottle neck was 5gb/s, which is much better than the 1gb/s I had, that also had to be shared with all server and other traffic.

First I had to decide how I would layout the 10gb/s network, while the ICX 7150 has 2 – 10gb/s ports, 1 is in use to go between the switches. After looking around and comparing my needs/wants/power/loudness-the-significant-other-would-put-up-with I got a MikroTik CRS309-1G-8S+IN. I wasn’t super excited to use them, since their security history is not fantastic, but I didn’t want to pay a ton or have a loud switch. I run the switch with the layer 2 firmware, and then put its management interface on a cut off vlan, that way it is very limited on what it can do.

After that I got a HP 10gb/s server cards, and tried a Solarflare S7120. Each had their ups and downs, the HPs are long and would not fit into some of the slim desktops I had. But when the would work, like in the Dells, they would work right away without issues. The Solarflare are shorter cards which is nice, but most of them ship with a firmware that will not work on some motherboards or newer operating systems. For these you need to find a system they work in, boot to Windows (perhaps an older version) then flash them with a tool off their website. After that they work great. I upgraded the 3 main hypervisors, and the NAS. I have seen the hypervisors hit 6.1gb/s when syncing Storage Spaces. With memory caching I can get over the rated disk speed.

That is the general layout of the network at this point. I am using direct attached cables for most of the systems. I did order some “genuine” Cisco 10gb/s SFP+ off Amazon for ~$20, I didn’t believe they would be real, but I had someone I know who works at Cisco look them up and they are real. Old stock shipped to Microsoft in 2012 or so, but genuine parts. The Ruckus switches and these NICs do not care which brand the SFPs are, so I figured I would get one I knew. The newer Intel NICs will not work with non Intel SFPs so look out.

To summarize, the everything comes in from my ISP to the Sophos XG box, then that connects to a port on one of the Ruckus switches. Those two Ruckus switches have a fiber link between them. Then one of the SFP+ ports on the Ruckus switch goes to a SFP+ port in the MikroTik switch. All the hypervisors hang off that MikroTik switch with SFP+ DACs. Desktops, video game consoles, and APs all attach to 1gb/s ethernet ports on the Ruckus switches. I have tried my best to label all the ports as best I can to make managing everything easier. I’m sure this will evolve more with time, but for my apartment now 10gb/s networking with a Ruckus R610 AP has been working very well.

ESXi Migration & Lenovo ThinkCentre M710s

I have started a transition from Hyper-V and Storage Spaces Direct to VMWare vSphere and vSAN. I apologize that these blog posts order is all over the place. Part of the transition is upgrading the hardware on some of the hosts I have, including getting 250GB NVME drives for vSAN cache. I started the migration with one of the desktops that run in the cluster, a Lenovo ThinkCentre M710s. After finding the small slot the NVME drive goes in, I realized there is a manufacture piece of plastic you are supposed to get to install a NVME drive. Since I do not have that, and do not want to pay for it, I spent a good bit more than a hour the first day of the migration creating this bracket and 3D printing it. Then while that was printing, I realized one of the feet on the system had gone missing, so I made a small one of those.

This post is just a quick update and a preview of more to come.

NVME Drive Holder: Lenovo ThinkCentre M710s NVME Bracket by danberk – Thingiverse

Foot: Lenovo ThinkCentre M710s Foot by danberk – Thingiverse

Homelab: Overview

I am starting a series about my homelab and how it is all laid out. I have written this article a few times, with months in between. Each time the setup changes, but we seem to be at a stable-ish point where I will start this series. Since I wrote this whole article and now a while later am editing it, I will mark with italics and underline when present me is filling in. I think it will give a neat split of growth in the last year or so I have been working on this. Or it will make it illegible, we will see. My home setup gives me a good chance to test out different operating systems and configs in a domain environment before using that tech elsewhere like at work.

Hypervisor

Starting off with virtualization technology, I settled a while ago on Microsoft Hyper-V instead of ESXi, the main reason behind it is I already had Windows Server, and Hyper-V allows for Dynamic memory, and allocating a range of memory for a VM. When something like an AD controller is idling, it doesn’t need much memory; when it starts it may, Dynamic memory allows me to take that into account. I will say one place that has bit me later is file storage, but that will be a later post.

The setup is technically “router on a stick”, where the Sophos XG firewall functions as the router, and the rest of the devices hang off of that. The Sophos XG machine is a old Dell Optiplex 990 (almost 10 years old!) with an Intel quad NIC in it. That way it can do hardware offloading for most of the traffic. I intend to do posts for networking, hypervisors, file storage, domain, and more; thus I will not get too in the weeds right now on the particulars.

The file storage is a FreeNAS box recently updated to 7, 3TB HDDS. I have had this box for about over 6 years (I just looked it up in November 2020, one of the drives has 55257 hours or 6.3 years of run time on it); it is older but has worked well for me so far.

The network backbone is a new switch I really like that I was able to get 2 of off eBay; they were broken but I was able to repair them, more on that later as well. They are Brocade, now Ruckus, ICX7150-12P; 12 1GB/s POE ports, 2 additional 1GB/s uplink port, and 2, 1/10GB/s SFP/SFP+ ports. These switches can run at layer 3, but I have the layer 2 firmware on them currently. They have a fiber connection between them, before that I was using 2 Unifi APs in a bridge, that didn’t work fantastic however because A. I am in NY, B. they were only 2×2 802.11AC Wave 1, and C. I am in NY. I custom ordered (so the significant other would not get mad) a white 50m fiber cable to go around the wall of the apartment.

With SSDs in the hypervisor boxes (I call them HV# for short) and iSCSI storage for VMs as well, which VMs are on which host doesn’t particularly matter. Flash forward 6 months or so, since that first sentence was written, I now still use the NAS for backups, but the hypervisors are running Storage Spaces Directed and doing shared storage now. This allows the hypervisors to move move VMs around during patching or pause during a system update if they are less critical. The Intel NUC and small Dell Inspiron are much under powered compared to the mid tower hypervisors, so they run usually only 1 or 2 things. The NUC runs the primary older domain controller, and that is it. It is an older NUC that I got about 7 years ago, so its not that fast. The “servers” in the hypervisor failover cluster are a Lenovo and 2 Dell Optiplex 5050s. I like these Dells because they go for about $200 on eBay, while having a Intel 7600 i5, can support 64GB of ram, and have expansion slots for things like 10gb SFP+ cards. These machines also idle at about 30 watts, which makes the power bill more reasonable.

Some of the services I run include:

  • 2 Domain Controllers (Server 2016, and 2019)
    • Including Routing and Access service for RADIUS and 802.1x on wifi on wired
  • Windows Admin Center Server (Windows Server 2019)
  • Windows Bastion (This box does Windows Management) (Server 2019)
  • Veeam Server (Server 2019)
  • Unifi Controller/Unifi Video for security camera (Ubuntu)
  • 3 Elastic Search boxes for ELK (CentOS 8)
  • Linux Bastion (CentOS 8)
  • Foreman Server (CentOS 8)
  • LibreNMS (This I grew to really like) (CentOS 8)
  • Nessus Server (CentOS 8)
  • Jira Server (CentOS 8)

That is the general overview, I will spend the next while diving into each bit and discussing how it is configured and what I learned in doing that.