Homelab Token Ring

For the LAN Before Time, my retro rack, I wanted to mix the most diverse set of CPU/OS/Networking I could find. There are not a ton of networking standards out there, as Ethernet took over so quickly. One that has always interested me is Token Ring, IEEE 802.5 standard, mostly from IBM as a competitor to Ethernet. Token Ring went through many transitions in its time on the scene, from speed changes to connector changes, lasting from the mid 1980s through the 1990s.

Connectors

The protocol started at 4mb/s (megabits a second), with the computer having a DB9 connector going to a giant 4 pin plug.

Later 16mb/s was added. Most of the cards you will find are 4/16 cards.

The physical connector, and connection speed are independent, you can use either the DB9 or RJ45 connectors to run 4mb/s or 16mb/s.

The cards started in the ISA era and later continued into the PCI era. The connector also evolved to a standard RJ45. There were adapters to go between the older connectors and newer ones. Later cards would include both DB9 and RJ45 connectors. With RJ45, only the middle 4 pins were used, but in a straight through way, allowing normal Ethernet straight through cables to be used.

In the last updates to the protocol, 100mb/s Token Ring was added, but by the time that came out Ethernet had taken much of the market share. And finally in 2001 a 1000mb/s standard was created, but Wikipedia says no devices ever came out for it.

MAUs

Unlike Ethernet, Token Ring cannot connect two computers directly. You need to go through a Media Access Unit, or MAU. These units control ports going in and out of the ring. They can be thought of like an Ethernet hub or switch. The Token Ring itself also needs a terminator on it. Later models contained internal terminators if put into a specific mode. There are MAUs with the old large IBM connector, and there are newer ones with RJ45. There were adapters between any of these connection types for networks in transition.

My MAU Journey

I picked up 2 of the same model MAU. ODS/Motorola 877. These are great units after some hardware tweaks and I would recommend them. While they are the same model, and same firmware revision, Motorola bought the company ODS (Optical Data Systems) which made them. The first one I got has ODS branding and a spot for two switches to control the mode and speed of the MAU. The second one is Motorola branded on the case, but not the board, and is missing the cut out in the case for switches.

From what I can learn with working on it, looking at documentation for other MAUs, and Claude; the device can work in three modes:

RING: Normal Token Ring operation, requires external RI/RO loopback cable to close the ring, use this when daisy-chaining multiple MAUs together, all active lobe ports are part of the ring.
STAR: Each port operates independently (not a true ring), used for certain troubleshooting or special configurations.
LOOP: Internally connects Ring In to Ring Out, self-terminates the ring without external cables, perfect for a single standalone MAU.

The MAUs were designed to have a switch to go between modes. Neither of mine did, both had a physical soldered in jumper setting their mode. The Motorola one didn’t have a hole in the case for a switch to exist, but the PCB is the same. I removed the soldered jumper and replaced it with a standard PC jumper pin, that way I could easily change it when I wanted to. In the end I will leave them both in LOOP mode most of the time, that has internal termination and is used for simple 4 port usage. Bridging the top and middle pin put it into LOOP mode, which is what I needed. Before that it was in RING without termination; each device would join the ring for 10 or so seconds, not hear anything else on the ring, and then disconnect. This MAU appears to be able to automatically go between 4mb/s and 16mb/s mode and I never moved the speed jumper.

The two modifications I made to these devices were the mentioned jumper change; and they come with a FGG 2P power connector onto a RJ45 plug. It says it needs 12V on it, and I wanted to just be able to use a wall plug, I first tried to get that connector, but after finding it tiny and hard to work with, I replaced the port in the device with a standard barrel plug.

Token Ring Drivers

One difficult part of finding Token Ring cards on eBay, you never know if you can find all the drivers. The card I have is a later model PCI card. It’s a Thomas Conrad TC4048. Thomas Conrad seems to have been an interesting company putting out different network cards over the 80s and 90s before ethernet took off. It is easy to find their Token Ring and Arcnet cards online. Finding their drivers on the other hand, proved to be difficult.

Driver Hunting

I found https://archive.org/details/pwork-297 this archive.org ISO, it contains a TON of drivers for devices in the 90s. It lists TC4048 as one of them. I download the image, install the driver AND… Windows 98 says it has the tc4048 files it needs except a “tc4048.dos”. I then found https://www.minuszerodegrees.net/software/Compaq/allfiles.txt this site which has every HP/Compaq driver that used to be on their site. Those are much easier to search. There were several TC4048 items.

I found an archive at https://ftp.zx.net.nz/pub/archive/ftp.compaq.com/pub/softpaq/sp19501-20000/, and downloaded sp19859.exe, which expanded and had “DOSNDIS” and “OS2NDIS”. I knew Compaq rebranded this card, so I yoloed and renamed “DOSNDIS/CPQTRND.DOS” to “tc4048.dos” and put it with the drivers I got from the archive.org image. The Thomas Conrad drivers from different vendors had similar files with different names, but they were the exact same size, and appeared to be the same… I hoped it would just work if I renamed a file from a different vendor to the one I needed. I made progress with error messages now seeing “svrapi.dll” missing in C:\Windows\, and found that file in C:\Windows\System32… and just copied it up one directory…

And magically that worked! I had a 16mb/s connection working between the Cisco 3825 (core) and the Windows 98 PC (edge)! The core of my retro network is a Cisco router. I purchased this Cisco 3825 system a while back because it’s the last one that supports Token Ring, but new enough to have 1gb/s uplink port to my core network. This allows me to host some retro VLANs internally, and firewall them off for security (since none of these systems have gotten patches for decades). I can play with Novell Netware and host a file share of games for the retro systems on this network as well. Using even legacy networks to move files is still a lot easier than a ton of floppy disks. I leave this router off most of the time because it’s a bit power hungry and loud. I have written about it before, and it also hosts my dial up connections.

I now had the Cisco 3825 with a Token Ring card and Windows 98 PC joining a Ring and communicating! I have watched a bunch of clabretro’s videos on Token Ring, and I saw the same issue with the Thomas Conrad drivers that he saw with his cards, Windows joining a Token Ring network and the drivers have an odd interaction. When the computer boots, at that point it tries to join the ring, the system will stay at the Windows startup screen an extra-long amount of time as it tries to enter the ring. The system will also wait at shutdown as it attempts to leave the ring. If the Token Ring card is not plugged in, you get a message about failing to connect after a prolonged startup.

Future Token Ring Plans

I plan to play with Token Ring a bit more both as a standard networking technology alongside the Ethernet network I have. Now that I have two working MAUs I want to experiment with linking them over the ST fiber connectors they have and getting a Token Ring connection over fiber. I am pondering learning FPGAs by building a Token Ring to Ethernet bridge using an FPGA connected to an ISA Token Ring card. I just find it interesting and it would push my FPGA skills; the project would need to translate the headers of Token Ring at layer 2 to Ethernet headers.

Token Ring is the layer 1 and layer 2 technology, after that we use standard TCP/IP on top of it; this has made it easy to get started with Token Ring over another protocol like AppleTalk or IPX. Once the physical connection was up, and devices could enter the ring; I was able to use standard Cisco commands and create a routable DHCP pool for Token Ring.

10″ Homelab Rack

I am working on a project that involves Intel vPro and AMD Dash. I am hoping to automate vPro and DASH systems with ansible the same way you can do iDrac and iLO for OS level development. I needed a few of those Tiny/Mini/Micro PCs to have the actual system API available. After stacking a bunch of them on the shelf next to my desk, I wanted a nicer way to organize them. I tried 3D printing a stand for one, and that worked for a time, then I decided to stack them. First I tried 3D modeling and printing little shelves that could fit them, this didn’t work great, I made the tolerances too tight making them hard to snap together. Then I decided to pivot to a 10″ Homelab rack.

There are many of these out there that come as full kits, but those can go for $100+, and I have a 3D printer… so where is the fun in that…

I found this https://www.printables.com/model/329801-10-ah-rail-based-servernetwork-rack/files model. You buy the metal posts offline, then this is the frame that holds the posts together. That brings the cost to 2 sets of posts (about $10 each), then some filament. I got these posts. That is more my style! There are many 10″ rack 3D models available. Many of them are BIG and engineered with handles so you can carry the systems around. Some are engineered to be 100% 3D printed with no metal posts. I wanted a rack with the strength of the metal posts, but simple.

I printed this, and while it was printing saw someone made a beefed up version with thicker corner pieces: https://www.printables.com/model/666403-10-in-mini-rack. I wish now that I printed that one instead because of the flimsiness, but it’s working as is. The frame is a bit flimsy, instead of the frame holding in the gear, the gear is more holding together the frame. I printed all these parts with 3 walls, and 50% infill. More than likely overkill, but I was just using PLA and wanted extra strength.

I went with 4U rack posts, I don’t need that much more than that now, didn’t want it to get too big, and for $20 investment I can always change the posts out later. Each of the mini pcs is 1U, this would give me 4 “slots”.

I printed this to hold the HP models I have: https://www.printables.com/model/585091-10-inch-rackmount-for-mini-hp-prodesk-elitedesk-g1. They were strong but I wanted to support that back of the mini pcs also, and found someone made a remix for that, short Remix: https://www.printables.com/model/841903-10-forward-or-reverse-rackmount-for-hp-elitedeskpr. The systems airflow is mostly front to back, and there are gaps between the systems for airflow.

I have an Intel NUC I also want to have in the rack. The NUCs are a little taller than 1U it seems, so all the 3D models to mount them are 2U. I didn’t want to use 2U up for it, I would just put it up top and it could stick out. I found this nice shelf: https://www.printables.com/model/1002978-geeekpi-10-in-rack-shelf, and printed that. I used this one specifically because it had adjustable rear supports.

The little rack is working well for me, sitting on a shelf and holding what it needs. The power supplies for these systems go down the shelf the rack is on to a power strip. Since these are development systems, I skipped having a UPS. Once I got the rack posts, the whole rack came together in a day and was easy to put together. This was a better solution than any other home grown one, because I get access to the entire ecosystem of 10″ standard rack parts!

History Phone

Periodically I enjoy doing more artistic electronics projects. I wanted to capture more of the family stories I hear but aren’t recorded or written down. I had the idea to get an old rotary phone, remove the parts inside, and replace them with an ESP32. When this phone is dialed, it looks for a year that is close to the number dialed, and then plays the audio file from a family member telling a story of that year. Along with hearing the story, a companion tablet website would allow you to see photos and get more information about the story. As all my “quick projects” go, this has been going on for a while, partially because I grew the scope some, partially because getting recordings can take a while.

Along with everything below, here is the github repo that contains controller code, 3D models, scripts, and more information. I have had three hardware revisions; the first was a prototype, the second was the first phone that was overly complex, the third was a simple phone with an all in one ESP32 audio board. This project was a great way for me to play with the ESP32 using SD cards, audio, and even Wifi.

General Design

All the different versions of hardware have an ESP32, SD card reader, and an audio chip. The main inputs are on/off hook, and the rotary piece itself. Then for output we have the speaker in the handset. When the handset is picked up, we play a dial tone.

Historically I have used the Arduino IDE to develop embedded code, for this project I moved to PlatformIO. As a more advanced user its MUCH nicer. You can easily add libraries, and do all your development within VSCode.

Outside of the dialing a year, I wanted some fun features that also helped with discovery. Dialing 0 runs an Operator audio file. This file is generated by a python script that scans the SD card contents, then creates an audio file stating which years can be dialed. Dialing 9 plays a random file from the SD card, this helps with random story discovery. Dialing 8 gives the option to change the volume, this prompt is created via another python script.

The rest of the numbers will try to find a year with a fuzzy matching a decade up and down. If no audio is found, it plays a busy signal. If the user doesn’t enter a number within 10 seconds or so, it plays a busy signal.

A quick tangent: the way the rotary dials work on these old phones is interesting. There are two pairs of wires that come off the rotary. Code for it is here. The first opens once the dial moves at all, this pair goes from a closed circuit to open and remains there till it rests at the start again. That’s when you start tracking the rotary. Next the second pair will pulse every time it passes a digit. The flow for reading this is once the movement starts, clear a counter and start counting pulses. It’s shockingly and happily simple.

Pinout

Other than V1 which had all the buttons and front LEDs hooked up, the wiring is fairly simple. There is the SD Card, which is internal on the all the boards; I have done external SD cards before but I prefer to have my ESP32 boards have a SD card reader on them. Audio – headphone amp, internal on V2 board. Then the two wires mentioned for rotary, and 1 for hook on/off. That’s it!

Iterations

Prototype V0

This was my first test, I 3D Printed a holder for a rotary dial I got off eBay for about $10, and wired up all the parts I would need. For the hook I used a simple switch you could toggle on and off. I did not touch things like Wifi in this iteration, this was just to see if I could get the basics working. The rotary was around $10, then a $17 ESP32 with built in SD card reader, then a few extra dollars for a switch, audio amp, and speaker. Total cost was around $30. This was built in a weekend or so.

Phone V1

After I had the prototype working, I moved onto to getting a real phone, and trying to get everything working within that. I got this phone that had several lines, and I thought I would use those buttons for different functions. Over time this seemed like overkill and kept me from making more progress. I also needed more IO ports, and had to start using MUXes to do this, which further discouraged me. I wired up all the front buttons to have a LED behind them, and could read if they were open or closed. All I ended up doing was lighting up line 1 when the handset was picked up. The buttons do have a great tactile feel though.

I did do most of the development with this phone. I got the web interface working, and push many versions; which involved doing a static build, and then coping it to the SD card. I cut a hole in the side of this phone to allow me to externally mount the SD card. When the web pages need updated or I needed to load more content I need the SD card, and that is buried very deep in this version of the phone.

To get the hook signal, I connected to the old phones terminals that close when the phone is on the hook. This worked decently, except I added some code to take several readings and average them. I think the old contacts weren’t the cleanest leading to periodic bad readings.

I tried to do clean wiring on this one. With the ESP32 I used for this version, Freenove CAM Board, it came with headers. I used those below through this mounting plate that swiveled to connect to a MUX and audio amp, a MAX98357.

This version of the phone, was an actual phone… That cost $59 on eBay (ones with all the lines were extra), then same $17 ESP32. I installed a few LEDs, MUXes, and an audio amp. This version probably cost closer to $75, and I used this as my test bench for… Almost a year as I on and off worked on this.

Fancy Phone V2

After how complex (see wiring photos, scream in horror) V1 became, I wanted to go back to the start and make a simpler one with the core phone functions. I also found this board that included an SD card reader, and audio amp all for $20, meaning I could use a phone, and this card for everything.

At this point I am doing my normal eBay scouring, and come across this nice executive desk phone. It’s simple, and the seller accepted my low offer! (I think eBay sellers just want to get rid of these phones) I wired everything up and I had a difficult time recreating V1. The main reason is what several reviews on Amazon said about this board (EC Buying ESP32-Audio-Kit Audio Development Board, ESP32-A1S); it has basically no documentation. Luckily, one person mentioned a repo that had a ton of information on the pinout, and that worked for me.

I probably would use this board again, because now I know how to make it work; but it was not forgiving to get working. There appear to be a bunch of companies that have this same board with different names on Amazon. There are different revisions and you don’t really know which one with which pinout you will get till you get it. The company has a github repo that has SOME information, but not all.

For this phones hook signal, I had to remove the piece that would usually be triggered to fit in the ESP32. I modeled and then 3D printed a bracket that holds the ESP32 in the bottom, and at the same time it has spots for two switches that get contact when the handset is put down.

This version of the phone I got (all these prices are with shipping) for $35, its a nice solid unit. The new ESP32 board was $22. Then I added 2 switches because I needed the hook signal, and some terminals. This version of the phone was closer to $60, and I spent about a month here and there on it.

File Structure

The SD Card has 2 main folders, content and web. Within content, there are some of the stock tones like a dial tone, then there are folders for each year. Within each year: photos, txt files, and mp3 files can live. The system will select at random an MP3 file if there is more than one. The files are supposed to be named starting at 1.mp3 and increment. The idea was you would have 1.mp3 and 1.txt, which would match up and have content matching with the audio. This wouldn’t be hard to make it not that way, but that’s how I was doing it. This is all documented more in the README, on the github page.

The web folder contains a Next.js React app that is built into a static site (a great Next.js feature!).

Web Interface

One feature that was an added on stretch goal was to have a companion website for the phone, allowing for photos and other media while stories were being told. The ESP32 can host a Wifi access point and web app, and I have been learning React, so this was a great place to keep learning! I used Next.js because I have used it before, and I know it can create a static HTML version of your site. I had to overcome some CORS and api issues, then it was smooth sailing. I used the next.config.mjs file to add redirects while doing development.

Adding features like volume control on the web page, and currently playing was helpful to learn how to properly do POST requests with an ESP32. I also added the ability to click a audio file in the web app, and then that will queue up on the phone.

Conclusion

Having my family play with the end product has been fun. One of the hardest parts of the project has been getting audio, cutting it down, and finding accompanying photos.

I had a bunch more ideas for the project. Adding a Bluetooth audio output option, photos overlaid on AI-generated period artwork, family tree generator integration, on-device recording capability, support for phone buttons and indicator lights, and standardized metadata schema. I already have spent far longer than I thought on this project, so those ideas will wait for another day.

In the end this was a fun project that I learned a lot of about web servers and audio processing on an ESP32. These chips continue to amaze me in what they can do for under $20. I will probably use that same audio board again, the documentation is bad but now I know how to get it to work. Here is the repo with even more information, the supporting 3D models, and files. If anyone has questions or recreates this, please leave a comment!

Step-By-Step Setting Up Networking for Virtualization on OpenShift 4.19 for a Homelab

As we continue our Openshift journey to get virtualization working, we have a vanilla node already setup and now we need to get the networking configured. The examples here are from Openshift 4.19.17.

Networking in OpenShift is conceptually two parts that connect. The first part is the host level networking; this is your CoreOS OpenShift host itself. Then there is how do the pods connect into that networking. Usually, the network connects through your network interface card (NIC), to the Container Networking Interface (CNI), then to your pod. Here we will be using a meta plugin that connects between the NIC and the CNI called Multus. Redhat has a good post about it.

Host Level Networking

This part of the networking stack is straight forward if you are used to Linux system networking, and it is setup the same way. Treat the CoreOS node like any other Linux system. The big decision to make in the beginning is how many interfaces you will have.

Networking diagram without sub interface

If you have 1 interface and plan on using virtualization, are you going to use VLANs? If so, then you may want to move the IP of the interface off of the primary interface and onto a VLAN sub interface. This moves the traffic from untagged to tagged traffic for your network infrastructure.

Another reason is there are bugs in the Mellanox firmware, mlx5e, where Mellanox 4 and 5 cards can think you are double VLAN encapsulating, and will start automatically stripping VLAN tags. The solution is to move all traffic to sub interfaces. You will get an error in your dmesg/journalctl of: mlx5e_fs_set_rx_mode_work:843:(pid 146): S-tagged traffic will be dropped while C-tag vlan stripping is enabled

With the interface moved, that frees us up to use it for other VLANs as well. If you deployed network settings via a MachineConfig, you would have to override them there.

The rest of the configuration will be done via the NMState Operator and native Openshift.

NMState VLAN and Linux Bridge Setup

NMState is a Network Manager policy system. It allows you to set policies like you would in Windows Group Policy, or Puppet to tell each host how the network should be configured. You can filter down to specific hosts (I do that for testing, to only apply to 1 host) or deploy rules for your whole fleet assuming nodes are all configured the same way. It’s possible to use tags on your hosts to specify which rules go to which hosts.

NMState can also be used to configure port bonding and other network configurations you may need. After configuration, you get a screen that tells you the state of that policy on all the servers it applies to. Each policy sets one or more Network Manager configurations, if you have multiple NICs and want to configure all of them, you can do them in one policy, but it may be worth breaking the policies apart and having more granularity.

Another way to go about this section, is to SSH into each node, and use a tool such as nmtui to manually set the networking. I like NMState because I get a screen that shows all my networking is set correctly on each node, and updates to make sure it stays that way. I put an example below of setting up port bonding.

Go to the OpenShift web console, if you need to setup OpenShift I suggest checking out either my SNO guide or HA Guide.
Click Operators -> OperatorHub.

Install NMState.
- Worth mentioning you can do all this with OKD, except NMState is very old, and hasn’t been updated in 5 years. Either NMState would need manually installed, or the interfaces would need manually created.

Once installed, you will need to create an “instance” of NMState for it to activate.

Then there will be new options under the Networking section on the left. We want NodeNetworkConfigurationPolicy. Here we create policies of how networking should be configured per host. This is like Group Policy or Puppet configurations.
At the NodeNetworkConfigurationPolicy screen, click “Create” -> “With YAML”.
- I put some additional YAML files below under Additional NodeNetworkConfigurationPolicy YAMLs
We need to create a new sub-interface off of our eno1 main interface for our new vlan, then we need to create a Linux Bridge off that interface for our VMs to attach to.

apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: vlan19-with-bridge           <-- Change This
spec:
  desiredState:
    interfaces:
      - name: eno1.19             <-- Change This
        type: vlan
        state: up
        ipv4:
          enabled: false
        vlan:
          base-iface: eno1
          id: 19                     <-- Change This
      - name: br19                   <-- Change This
        type: linux-bridge
        state: up
        ipv4:
          enabled: false
        bridge:
          options:
            stp:
              enabled: false
          port:
            - name: eno1.19       <-- Change This
              vlan: {}

Important things here:
- Change the 19s to whichever VLAN ID you want to use.
- “ipv4: enabled: false” says we want an interface here, but we are not giving it host level IP networking on our OpenShift node.
- Remove the <– Change This comments
- You MUST leave the “vlan: {}” at the end or it will not work, adding this tells it to leave vlan data how it is because we are processing via the kernel via sub interfaces.

Now we have this configuration, with a secondary interface off of our NIC, and an internal Linux Bridge for the VMs.

The great thing about doing this configuration via NMState, it applies to all your nodes unless you put a filter in, and you get a centralized status about if each node could deploy the config.

Here is an example from my Homelab, with slightly different VLAN IDs than we have been discussing. You can see all three nodes have successfully taken the configuration.

OpenShift VM Network Configuration

Kubernetes and OpenShift use Network Attachment Definitions (NADs) to configure rules of how pods can connect to host level networking or to the CNI. We have created the VLANs and Bridges we need on our host system, now we need to create Network Attachment Definitions to allow our VMs or other pods to attach to the Bridges.

Go to “Networking” -> “NetworkAttachmentDefinitions”.
Click “Create NetworkAttachmentDefinition”
This is easily done, and can be done via the interface or via YAML, first we will do via the UI then YAML.
Before entering the name, make sure you are in the Project / Namespace you want to be in, NADs are Project / Namespace locked. This is nice because you can have different projects for different groups to have VMs and limit which networks they can go to.
Name: This is what the VM Operator will select, make it easy to understand, I do “vlan#-purpose“, example: “vlan2-workstations”.
Network Type: Linux Bridge.
Bridge Name: what was set above, in that example “br19“, no quotes.
VLAN tag number: Leave this blank, we are processing VLAN data at the kernel level not overlay.
MAC spoof check: Do you want the MAC addresses checked on the line. This is a feature which allows the network admin to pin certain MAC addresses and only send traffic out to those allowed. I usually turn this off.
Click “Create“

The alternative way to do a NAD is via YAML, here is an example block:

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: vlan19-data-integration
  namespace: default
spec:
  config: |-
    {
        "cniVersion": "0.3.1",
        "name": "vlan19-data-integration",
        "type": "bridge",
        "bridge": "br19",
        "ipam": {},
        "macspoofchk": false,
        "preserveDefaultVlan": false
    }

You can verify the NAD was created successfully by checking the NetworkAttachmentDefinitions list. Your networking is ready now. Next post, we will discuss getting storage setup.

Additional NodeNetworkConfigurationPolicy YAMLs

NIC Bonding / Teaming

Use mode 4 (802.3ad/LACP) if your switch supports link aggregation; otherwise mode 1 (active-backup) is the safest fallback.

apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: bond0-config
spec:
  desiredState:
    interfaces:
      - name: bond0
        type: bond
        state: up
        ipv4:
          enabled: false
        link-aggregation:
          # mode=1 active-backup
          # mode=2 balance-xor
          # mode=4 802.3ad
          # mode=5 balance-tlb
          # mode=6 balance-alb
          mode: 802.3ad
          options:
            miimon: '140'
          port:
            - eno1
            - eno2

Useful Links

https://github.com/k8snetworkplumbingwg/multus-cni/blob/master/docs/how-to-use.md

https://medium.com/@tcij1013/how-to-configure-bonded-vlan-interfaces-in-openshift-4-18-0bcc22f71200

Step-By-Step Getting Started with High Availability OpenShift 4.19 for a Homelab

Last post, looked at getting started with a SNO (Single Node OpenShift) system. Next we will look at a build with multi-node, or multi-master, OpenShift. This runs the core service of etcd on more than one node, allowing for a single node failure. Some services like the virtual machine services need to run on a master as well, having more than one relieves pressure on that system. With SNO, if your master does not start, the entire cluster cannot start. In addition, SNO upgrades will always introduce downtime with the single master rebooting.

Master nodes do have more services than a simple worker, if you are running a small cluster with 3 nodes, you may want to decide if the extra overhead on the second and third nodes are worth it, or if you want to run leaner and run SNO with extra workers. In my experience of vanilla OpenShift, masters run about 20GB of ram more than worker nodes with no additional services on them.

I have a 3 node cluster that I was migrating from VMware and wanted to run HA. This allows me to do no downtime upgrades, with the three nodes sharing the control role.

My Setup

I am installing onto 3 HP Elitedesk 800 G5s, each with an Intel 9700, and 96GB of RAM (they can go to 128GB when RAM prices aren’t insane). I have a dual 10gb/s NIC in each for networking since I will be running ceph. This is the same Homelab cluster I have had for a bit. These machines aren’t too expensive, they have 8 cores each, can go to 128GB of RAM, and have several PCI slots, and NVMe slots. I have used this guide to install OpenShift 4.17-4.20.

Installation Steps for HA OpenShift

Any line starting with $ is a terminal command to use. The whole process will take about an hour; 30 minutes or so to collect binaries and prep your config files, a minute or two to create the ISO, then 30 minutes of the cluster sitting there and installing.

One important thing to say up front to those who have not used Openshift or Kubernetes before: there is 1 IP that all the applications use, the web server looks at the request coming in and WHICH DNS NAME YOU CONNECTED TO, and then routes your traffic that way. You can have 100% of the things setup right, and when you browser to the IP you get “Application is not available” when trying to access the console. This means the system is working! You just need to connect via the correct DNS name.

Prerequisites: Start by going to the same place as the original post to get a pull secret and binaries you will need for the install. These include openshift-install, and oc.
I am on Fedora 42 and needed to run sudo dnf install nmstate to install nmstate. This is required to transform the configs in the agent-config.yaml into the configs that will be injected into the installation ISO.
Make a folder, called something like “ha-openshift”, and put all the binaries in there.
Config Files: Before we had install-config.yaml, now we will have that AND agent-config.yaml.
Below is an install-config.yaml, I will call out things you will want to change for your setup:
- ```
apiVersion: v1
baseDomain: example.com
compute:
- architecture: amd64
 hyperthreading: Enabled
 name: worker
 platform: {}
 replicas: 0
controlPlane:
 architecture: amd64
 hyperthreading: Enabled
 name: master
 platform: {}
 replicas: 3
metadata:
 name: cluster1
networking:
 clusterNetwork:
 - cidr: 10.131.0.0/16
 hostPrefix: 23
 machineNetwork:
 - cidr: 192.168.4.0/24
 networkType: OVNKubernetes
 serviceNetwork:
 - 172.30.0.0/16
platform:
 baremetal:
   apiVIPs:
   - 192.168.4.5
   ingressVIPs:
   - 192.168.4.7
pullSecret: '{"auths":{"cloud.openshift.com":{"auth":"b3Blbn==","email":"not-my-real-email@gmail.com"}}}'
sshKey: ssh-rsa AAAAB
```
- The “baseDomain” is the main domain to use, your hosts will be master0.<baseDomain>, the cluster name will be <metadata.name>.<baseDomain>. Make sure you put in what you want here because you can’t change it later. This is how users will reference the cluster.
- Under workers and controlPlane, you put how many worker nodes and master nodes you want. This is a big difference between SNO and HA, we are saying 3 instead of 1 master.
- metadata.name is the sub name of this exact cluster. You can have multiple clusters at lets say “example.com”, then setting this will make the cluster apps.cluster1.example.com. (Yes the DNS names get long with OpenShift)
- clusterNetwork and serviceNetwork will be used internally for backend services, only change these if you are worried about the preset ones conflicting with your IP space.
- machineNetwork.cidr is the IP space your nodes will live on, this needs to be set for your DHCP network. This is the range the network will use. Some of the IPs below will need static reservations in your DHCP network, the worker and master nodes can have general pool DHCP addresses. We are assuming DHCP here, you can statically assign IPs but its more work and not something I am going to talk about right here.
- platform.baremetal.apiVIPs is where the API for your cluster will live, this is an additional IP the HA masters will hand back and forth to give the appearance of a single control plane.
- platform.baremetal.ingressVIPs is another IP that will be handed back and forth but will be the HTTPs front door for applications.

agent-config.yaml, I will call out things you will want to change:

apiVersion: v1alpha1
kind: AgentConfig
rendezvousIP: 192.168.4.10
hosts:
  - hostname: hv1
    role: master
    rootDeviceHints:
      serialNumber: "AA22122369"
    interfaces:
      - name: enp1s0f0
        macAddress: 0c:c4:7b:1e:42:14
      - name: enp1s0f1
        macAddress: 0c:c4:7b:1e:42:15
    networkConfig:
      interfaces:
        - name: bond0.4
          type: vlan
          state: up
          vlan:
            base-iface: bond0
            id: 4
          ipv4:
            enabled: true
            address:
              - ip: 192.168.4.10
                prefix-length: 24
            dhcp: false
        - name: bond0
          type: bond
          state: up
          mac-address: 0c:c4:7b:1e:42:14
          ipv4:
            enabled: false
          ipv6:
            enabled: false
          link-aggregation:
            mode: 802.3ad
            options:
              miimon: "150"
            port:
              - enp1s0f0
              - enp1s0f1
      dns-resolver:
        config:
          server:
            - 192.168.3.5
      routes:
        config:
          - destination: 0.0.0.0/0
            next-hop-address: 192.168.4.1
            next-hop-interface: bond0.4
            table-id: 254
  - hostname: hv2
    role: master
    rootDeviceHints:
      serialNumber: "AA22628"
    interfaces:
      - name: enp1s0f0
        macAddress: 0c:c4:7b:1f:06:e2
      - name: enp1s0f1
        macAddress: 0c:c4:7b:1f:06:e3
    networkConfig:
      interfaces:
        - name: bond0.4
          type: vlan
          state: up
          vlan:
            base-iface: bond0
            id: 4
          ipv4:
            enabled: true
            address:
              - ip: 192.168.4.20
                prefix-length: 24
            dhcp: false
        - name: bond0
          type: bond
          state: up
          mac-address: 0c:c4:7b:1f:06:e2
          ipv4:
            enabled: false
          ipv6:
            enabled: false
          link-aggregation:
            mode: 802.3ad
            options:
              miimon: "150"
            port:
              - enp1s0f0
              - enp1s0f1
      dns-resolver:
        config:
          server:
            - 192.168.3.5
      routes:
        config:
          - destination: 0.0.0.0/0
            next-hop-address: 192.168.4.1
            next-hop-interface: bond0.4
            table-id: 254
  - hostname: hv3
    role: master
    rootDeviceHints:
      serialNumber: "203129F9D7"
    interfaces:
      - name: enp1s0f0
        macAddress: 0c:c4:7b:1f:03:c2
      - name: enp1s0f1
        macAddress: 0c:c4:7b:1f:03:c3
    networkConfig:
      interfaces:
        - name: bond0.4
          type: vlan
          state: up
          vlan:
            base-iface: bond0
            id: 4
          ipv4:
            enabled: true
            address:
              - ip: 192.168.4.30
                prefix-length: 24
            dhcp: false
        - name: bond0
          type: bond
          state: up
          mac-address: 0c:c4:7b:1f:03:c2
          ipv4:
            enabled: false
          ipv6:
            enabled: false
          link-aggregation:
            mode: 802.3ad
            options:
              miimon: "150"
            port:
              - enp1s0f0
              - enp1s0f1
      dns-resolver:
        config:
          server:
            - 192.168.3.5
      routes:
        config:
          - destination: 0.0.0.0/0
            next-hop-address: 192.168.4.1
            next-hop-interface: bond0.4
            table-id: 254

rendezvousIP is an IP of a node in charge of the setup. You pick one of them to wait for all other masters/workers to be ready before starting the installation. It will wait for all nodes to be online, check they are ready, install them, then install itself.
The rest of this config is a three times repeated (one per host) setup of each host, things you will want to change:
- hostname, whatever each hostname should be.
- rootDeviceHints.serialNumber, this is the serial number of the disk you want to install the OS onto. There are many different rootHints you can use depending on your setup. https://docs.redhat.com/en/documentation/openshift_container_platform/4.19/html/installing_an_on-premise_cluster_with_the_agent-based_installer/preparing-to-install-with-agent-based-installer#root-device-hints_preparing-to-install-with-agent-based-installer. To find the drive serial numbers, you can boot either a Redhat CoreOS image, or any Linux LiveCD, then run something like $ lsblk -o name,model,size,serial .
- List interfaces with MAC addresses. I did here use port bonding and then subinterfacing the port bond for a vlan. That is a bit advanced and is not required. I do it this way so I can run a trunk port to each node, and then with VMs, run different VMs over different sub interfaces. The website above also has examples of networking without this port bond if you want a simpler config.
- DNS and core routes are the last thing to enter.

DNS Entries: Having created those two files, you know what you want your DNS to be. It’s time to go into your location’s DNS servers and enter addresses just like in the original post. These entries can be made at any time before you start the installation. In the end you should have 1 IP for ingress, 1 for api, then one per node.
- api.cluster1.example.com -> apiVIPs, in my config 192.168.4.5
- api-int.cluster1.example.com -> apiVIPs, in my config 192.168.4.5
- *.apps.cluster1.example.com -> ingressVIPs, in my config 192.168.4.7
- master0.cluster1.example.com -> node1 IP, in my config hv1 so I put 192.168.4.10
- master1.cluster1.example.com -> node2 IP, in my config hv2 so I put 192.168.4.10
- …
Image Creation:
$ mkdir ocp
$ cp *.yaml ocp
$ ./openshift-install –dir ./ocp/ agent create image
This will create a ocp/agent.x86_64.iso
Installation: Boot that iso on all servers. The image will use the hardware you specified in agent-config.yaml and DNS lookups to identify each node. Make sure the systems NTP is working, and their time looks correct, then that each node can curl:
- registry.redhat.io
  quay.io
  cdn01.quay.io
  api.openshift.com
  access.redhat.com
The stack should now install, the main server will show a screen saying the state of the other masters, and when they are all ready, it will proceed with install. This can easily take 30 minutes, and the screen on the rendezvous server can be slow to update.

With any luck you will have all the nodes reboot, and a running stack you can access at your console server location; here that would be console-openshift-console.apps.cluster1.example.com. Each node should show a normal Linux boot up sequence, then will show a login prompt, with that nodes name, and IP address(es). In this learning experience, feel free to restart the installation and the system will wipe the machines again.

In the ha-openshift folder, then the ocp subfolder there will be an auth folder. That will have the kubeadmin and kubeconfig files to authenticate to the cluster. The kubeadmin password can be used to login to oauth at console-openshift-console.apps.cluster1.example.com. The kubeconfig file can be used with the oc command downloaded from Redhat. using $ ./oc --kubeconfig ./ocp/auth/kubeconfig get nodes will show the nodes and their status from your installation machine.

Properly installed cluster example: 
~/homelab_openshift $ ./oc --kubeconfig ./ocp/auth/kubeconfig get nodes
NAME   STATUS   ROLES                         AGE   VERSION
hv1    Ready    control-plane,master,worker   44d   v1.32.9
hv2    Ready    control-plane,master,worker   44d   v1.32.9
hv3    Ready    control-plane,master,worker   44d   v1.32.9

This is an example of a successfully upgraded cluster running, and I am running the standard OpenShift oc get nodes command. Note: the version is the version of Kubernetes being run, not OpenShift.

I will continue this series with posts about Networking, Storage, and VM setup for OpenShift.

Troubleshooting

The install process for OpenShift has a big learning curve. You can make it a bit easier by using Redhats web installer, but that also puts some requirements on the system that a Homelab usually can’t hit, doing the agent based installer bypasses those checks. Once you get your configs dialed in, I have found it easy to reinstall a stack, but getting configs for a stack setup correctly the first few times is tough. The installer also does not do a ton to make it easier on you, if something goes wrong, the biggest indicators I have found are: when SSHed into the installer, the memory usage, the journalctl logs in the installer, and about 8-10 minutes into a good install, you will see the DVD image start to read a lot of data, constant activity on the indicator for a few minutes (that is the CoreOS being written to the disk).

Random things to check in a failing install:

SSH into a node using the SSH key in the install-config.yaml, run $ sudo journalctl and scroll to the bottom to see what’s going on, or just run $ sudo journalctl -f.
- You may see something like:
  - “failing to pull image”: It can’t hit Redhat, or your pull secret expired
  - “ip-10-123-123-132.cluster.local node not recognized”: DNS entries need updated
If the system successfully reboots after an install, but you are not seeing the console start, SSH into a node using the SSH key in the install-config.yaml, run $ top. If your RAM usage is about:
- 1GB, Kubernetes is failing to start, this could be a DNS or image download issue.
- around 8GB, the core systems are attempting to come online, but something is stopping them such as an issue with the api or apps DNS names.
- 12-16+GB of ram used, the system should be online.
Worth repeating for those who haven’t used Openshift before, internal routing is done via DNS names in your request, if you attempt to go to the ingress VIP via the IP you will get “Application is not available”. This is good! Everything is up, you just need to navigate to the correct URL.

Footnotes

Helpful examples: https://gist.github.com/thikade/9210874f322e72fb9d7096851d509e35

Manually Flashing a AsRock BIOS

At the start of 2022 I went about building a new desktop. The system came out well in the end, and I am still using it although it has always had one issue that I mentioned back then… When the system tries to reboot it shuts down Windows, then the fans spin up, and it never resets the CPU. I always assumed this was due to the fact: mistakes were made and when I needed a BIOS update to support the AMD Ryzen 5800X on that motherboard, I flashed the X570 Pro4, NOT X570M Pro4 BIOS. I decided the time had come to fix this and flash the board back to the proper BIOS.

Note: I will use BIOS and UEFI interchangeably here, I know they are technically different.

AsRock gives out the BIOS updates, and you are supposed to load them on a USB drive and select the “Instant Flash” utility in the UEFI menu. The issue is the system thinks it’s a X570 Pro4 (not M), and refuses to flash to the M variant. I saw some people on the internet mention you can use the “AMIBIOS and Aptio AMI Firmware Update Utility” utility to force a BIOS flash on the system and get around this. I did that, and then… the system never booted again… I don’t know what went wrong, or if the utility flashed everything correctly, but I couldn’t get the system to boot.

This led me to another rabbit hole, flashing your bios with an EEPROM writer. There are several to choose from, and of course I got both. The CH341A, and T48 TL866-3G programmers. I tried for a while to get the CH341A to work, and it may have in the end, but I had a better time with the TL866 software of XgPro.

I unplugged everything and took out the BIOS battery. At first, I tried to use some of the tools that came with the flashers to grab the chip on the motherboard. Here was the first hurtle, the chip was a surface mount part that made it very difficult to connect to and every time I tried to read the chip I got an error. Then I found this post, https://www.reddit.com/r/ASRock/comments/1cz1bcz/guide_asrock_spi_tpm_j1_bios_ph_pinout_for_ch341a/, which mentioned the SPI_TPM_J1 header on the motherboard actually hooked directly to all those pins and could be used to flash the chip much easier.

This diagram was the key to getting it working, which pins went to where on the motherboard.

Connecting to the pins proved to be a bit difficult because of how small they were. Turns out they are 2mm pitch instead of 2.54mm (more standard), so I had to order https://www.amazon.com/dp/B07FKLKM7L as one of the commenters suggested to be able to connect easily to the pins. Then I did some janky stuff with jumper wires to get it into the TL866 programmer.

The programmer also uses Windows-only software, and I was on my Linux laptop. I installed a Windows 11 VM and forwarded the USB device to use the software. The Reddit post mentioned that they had a similar motherboard and needing to use the 1.8V adapter for their Winbond W25Q256JW BIOS chip. I assumed the same but when I went to read the bad BIOS back, I was given “ID Check Failed 0xC2 25 39”. Which turns out to be a 3.3V Macronix MX25L256, with Claude assuming it’s a MX25L25673G. I pulled out the 1.8V adapter and hooked right to the chip. I still failed to get an ID with the built in tool, and when I forced the chip type, I was told one pin wasn’t connected right. I adjusted the pin and hit READ again and got a hit, the ID didn’t match the software’s record for that chip type. I ended up disabling ID Check and then doing a read. AND I GOT DATA! This showed my wiring was good enough.

From there I went and got a new BIOS from AsRocks site. Seeing the BIOS file was exactly 32MB, and the BIOS was 256 megabits = 32MB, I felt confident that I could write this whole image and it didn’t have a header or something that would break directly writing this image to the chip.

I hit program and crossed my fingers. After a minute of erasing, then a minute of programming and verifying it said it was done. I unplugged my laptop, and plugged power back in. The system came on when I hit power, but again no POST. Then I waited a minute, and it powered off, this time, when I powered it back on the system came to life with the AsRock UEFI screen! I had to re-enter UEFI settings but we are back!

The last and final irony… on the correct BIOS, up to date, the system still fails to restart…

Step-By-Step Getting started with Single Node OpenShift (SNO) for a Homelab

Preface

I will explain why OpenShift, and will have that blurb after the tutorial for those interested. I have some information for those completely new to OpenShift and Kubernetes (shorthand “K8s”), feel free to jump to “Installation Steps for Single Node OpenShift” for steps. This guide walks you through doing a Single Node OpenShift installation. This should take about 1-2 hours to have a basic system up and running.

In later posts I will go over networking, storage, and the rest of the parts you need to setup. I spoke to some of their engineers, and they were confused when I said this system is not easy to install, and they need to make an easy installation disc like VMware or Microsoft have.

It is worth noting at this point that OKD exists. OKD is the upstream (well moving upstream), open-source version of OpenShift. You are more bleeding edge, but you get MOST of the stack without any licensing. Almost like CentOS was to Redhat Enterprise Linux, except more upstream than in line. There are areas where that is not true, and other hurtles to use it; but I am going to make another post about that.

Single Node OpenShift vs High Availability

There are two main ways to run OpenShift, the first is SNO; Single Node OpenShift. There is no high availability, everything runs with 1 master node, which is also your worker node. You CAN attach more worker servers to a SNO system, but if that main system goes down, then you lose control of the cluster. The other mode to run in is HA, where you have at least 3 nodes in your control plane. For production you would usually want HA, and I will have an article about that in the future, for now I will just install SNO.

Big Changes to Keep in Mind From VMware

A quick note to all the administrators coming from VMware or other solutions, OpenShift runs on top of CoreOS. An immutable OS based on Redhat and ostree. The way OpenShift finds out which config to apply to your node is via DHCP and DNS. These are HARD REQUIREMENTS to have setup for your environment. The installation will fail, and you will have endless problems if you do not have DHCP + DNS setup correctly; trust me, I have been there.

K8s Intro 101

For those who haven’t used Kubernetes before (me a few weeks ago), here are some quick things to learn. A cluster has “master” nodes and “worker” nodes, masters orchestrate, workers run pods. Master nodes can also be worker nodes.

OpenShift by default cannot run VMs. We are installing the Virtualization Operator, operators are like plugins, which will give us the bits we need to run virtualization. OpenShift has OpenShift Virtualization Operator, OKD has KubeVirt. OpenShift Virtualization Operator IS KubeVirt with a little polish on it and supported by Redhat.

Homelab SNO Installation

OpenShift is built to have a minimum of 2 disks. One will be the core OS and the containers that you want to run. The other will be storage for VMs and container data. By default the installer does not support partitioning the disk, forcing you to have 2 disks. I wrote a script that injects partitioning data into the SNO configuration. The current SNO configuration does not seem to have another easy way to add this. The script: Openshift-Scripts/add_parition_rule.sh at main · daberkow/Openshift-Scripts, needs to be run right after “openshift-install”, Step 18. It is run with “$ ./add_parition_rule.sh ./ocp/bootstrap-in-place-for-live-iso.ign ./ocp/bootstrap-in-place-for-live-iso-edited.ign”, then “./ocp/bootstrap-in-place-for-live-iso-edited.ign” is used for Step 20.

I am running on a Hp ProDesk 600 G5 Mini with an Intel 9500T, 64GB of RAM, and a 1TB NVMe drive. You need any computer you can install an OS onto with at least 100GB of storage and probably 32GB of RAM. Redhat CoreOS is a lot more accepting of random hardware than VMware ESXi is.

Installation Steps for Single Node OpenShift

OpenShift has several ways to do an installation, you can use their website and do the Assisted installer or create an ISO with all the details baked in, this time we will go over how to do it with creating a custom ISO with an embedded ignition file.

The following steps will be for a Mac or Linux computer. The main commands you will use interact with your cluster are `kubectl` and `oc`; `oc` is the openshift client, and a superset of the features in the standard `kubectl` command. Those tools work on Windows and have builds. The `openshift-installer` does not, so we can’t install with just Windows. You can try to use WSL to do the install, but it always gave me issues. The Linux system needs to be Rhel 8+/Fedora/Rocky 8+ or Ubuntu 20.10+ because of the requirement for Podman.

As mentioned, DHCP + DNS are very important for OpenShift. We need to plan what our cluster DOMAIN and CLUSTER NAME will be. For this I will use “cluster1” as the cluster, and “example.com” as the domain. Our example IP will be 192.168.2.10 for our node. When I put a $ at the start of a line, that is a terminal command.

First, we will setup DNS, that is a big requirement for OpenShift, to do that you need a static IP address. Give the system a reservation or static IP address for your environment.
Now go and make the following addresses point to that IP, because we are on a single node, these can all point to one IP. Note this is for SNO, for larger clusters you need different hosts and VIPs for these IPs.
1. api.cluster1.example.com -> 192.168.2.10
2. api-int.cluster1.example.com -> 192.168.2.10
3. *.apps.cluster1.example.com -> 192.168.2.10
4. The two api addresses are used for K8s API calls, *.apps is a wildcard where all the sub apps within the cluster will be accessed. These applications use the referrer url of the web request to figure out where the traffic should go, thus everything has to be done via DNS name and not IP.
5. Note: The wildcard for the last entry is needed for some services to work, you can individually add them, but it becomes a lot of work. Wildcards can not be used in hosts file, which means you do need proper DNS. There is a footnote for all the DNS entries you may if you want to run out of a hosts file.
Go to Download Red Hat Openshift | Red Hat Developer.
Sign up for a Redhat Developer account and click “Deploy in your datacenter”.
Click “Run Agent-based Installer locally”.
Download the OpenShift installer, your “pull secret”, and a command line tool.
- You can also download the installer and clients from: https://mirror.openshift.com/pub/openshift-v4/clients/ocp/
Open a terminal and make a “sno” folder wherever you want.
Install Podman on your platform, if that’s Windows that means within WSL2, not on the Windows host.
Copy/extract the openshift-installer, oc, and kubectl commands to that folder.
$ export OCP_VERSION=latest-4.19
$ export ARCH=x86_64
$ export ISO_URL=$(./openshift-install coreos print-stream-json | grep location | grep $ARCH | grep iso | cut -d\” -f4)
$ curl -L $ISO_URL -o rhcos-live-fresh.iso
- I used “rhcos-live-fresh.iso” for the clean ISO, then copied it every time I needed to start over, I found this easier than redownloading.
$ cp rhcos-live-fresh.iso rhcos-live.iso

Create a text file called “install-config.yaml”, copy the following and edit for your setup:

apiVersion: v1

baseDomain: example.com
compute:
- name: worker
  replicas: 0
controlPlane:
  name: master
  replicas: 1
metadata:
  name: openshift
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  machineNetwork:
  - cidr: 192.168.2.0/24
  networkType: OVNKubernetes
  serviceNetwork:
  - 172.30.0.0/16
platform:
  none: {}
bootstrapInPlace:
  installationDisk: /dev/nvme0n1
pullSecret: '{"auths":{"cloud.openshift.com":{"auth":"b3BllBFa…0M4NjNSaEo0RmNXZw==","email":"danisawesome@example.com"}}}'
sshKey: |
  ssh-rsa AAAAB3QQe/… /h3Pss= dan@home

Note: I have removed most of my pull secret, and ssh key

baseDomain: This is your main domain
clusterNetwork: The internal network used by the system, DO NOT TOUCH
machineNetwork: Network your system will have a NIC on, change this to your network
serviceNetwork: Another internally used network, DO NOT TOUCH
installationDisk: The disk to install to
pullSecret: Insert that secret downloaded from Redhat in Step 6
sshKey: The public key to your local accounts ssh key, this will be used for auth later

$ mkdir ocp
$ cp install-config.yaml ocp
$ ./openshift-install –dir=ocp create single-node-ignition-config
- Optional to operate off a single disk
- ./add_parition_rule.sh ./ocp/bootstrap-in-place-for-live-iso.ign ./ocp/bootstrap-in-place-for-live-iso-edited.ign
$ alias coreos-installer=’podman run –privileged –pull always –rm -v /dev:/dev -v /run/udev:/run/udev -v $PWD:/data -w /data quay.io/coreos/coreos-installer:release’
$ coreos-installer iso ignition embed -fi ocp/bootstrap-in-place-for-live-iso.ign rhcos-live.iso
Boot rhcos-live.iso on your computer, it will take 20 or more minutes, then the system should reboot
If everything works, the system will reboot, then after 10 or so minutes of the system loading pods, https://console-openshift-console.apps.cluster1.example.com/ should load from your client computer. The login will be stored on your sno/ocp/auth folder.

Many caveats here: if your install fails to progress, you can ssh in with the SSH key you set in the install-config.yaml file. That is the only way to get in. Check journalctl to see if there are issues. It’s probably DNS. You can put the host names above into the hosts file of the installer and then after reboot the host itself to boot without needing DNS.

You CAN build an x86_64 image using an ARM Mac. You can also create an ARM OpenShift installer to run on a VM on a Mac. The steps are very similar for an ARM Mac except they have aarch64 binaries at: mirror.openshift.com/pub/openshift-v4/aarch64/clients/ocp/latest-4.18/, and you use “export ARCH=aarch64”. Be careful on an ARM Mac about using the x86_64 installer for targeting an x86_64 server, and a aarch64 installer for ARM VMs. Or you will get “ERROR: release image arch amd64 does not match host arch arm64” and have to go to ERROR: release image arch amd64 does not match host arch arm64 – Simon Krenger to find out why.

Hopefully this helps someone, I think OpenShift and OKD could be helpful for a lot of people looking for a hypervisor, but the docs and getting started materials are hard to wrap your head around. I plan to make a series of posts to help people get going. Feel free to drop a comment if this helps, or something isn’t clear.

DNS SNO Troubles

This section is optional, and for those who would like to run without external DNS for a stack. It can lead to the stack being odd, if you dont need this, you may not want to do it. All this was tested on 4.19.17.

The issue you run into here, is the fact that the way DNS works in OpenShift is pods are given CoreDNS entries, and they are given a copy of your hosts resolv.conf. In the event you want to start an OpenShift system completely air-gapped, with no external DNS, you need the entries we stated in other articles, mainly: api.<cluster>.<domain>, api-int.<cluster>.<domain>, *.apps.<cluster>.<domain>, master0.<cluster>.<domain>. Wildcard lookups cannot be in a hosts file. Luckily, because of this, OpenShift ships with dnsmasq installed on all the hosts.

Our flow for DNS will be: the host itself runs dnsmasq, and points to itself for DNS. It has to point to itself on its public IP because that resolv.conf file will be based onto pods; if you put 127.0.0.1 then pods will get that and fail to hit DNS. Then dnsmasq points to your external DNS servers. That way, all lookups hit dnsmasq first, then can be filtered to the outside.

When installing OpenShift: there is the install environment itself, then the OS after reboot, we need these entries to be in both environments.

I have created a script, it is used like the partition script I used in the SNO post. To use it, create your ignition files with openshift-install, then $ ./add_dns_settings.sh ./ocp/bootstrap-in-place-for-live-iso.ign ./ocp/bootstrap-in-place-for-live-iso-edited.ign and install with that edited ignition file.

This allows you to set all the settings you need, and a static IP setting for the host that will run single node. When installing this way, you will need to add some hosts file entries to your client because outside the cluster the DNS entries dont exist. The new SNO system is not in external DNS and that is how OpenShift routes traffic internally. Adding the below line to your clients hosts file with cluster and domain changed should be enough to connect:

192.168.1.10 console-openshift-console.apps.<cluster>.<domain> oauth-openshift.apps.<cluster>.<domain>

Backstory About Why OpenShift

After all the recent price hikes by Broadcom for VMware, my work – like many – have been looking for alternatives. Not only do high hypervisor costs make it expensive for your existing clusters, it makes it hard to grow clusters with that high cost. We already run a lot of Kubernetes and wanted a new system that we could slot in, allowing for K8s and VMs to run side by side (without paying thousands and thousands per node that Broadcom wants). I was tasked with looking at alternatives out there, we were already planning on going with OpenShift as our dev team had started using it, but it doesn’t hurt to see what else is out there. The requirements were: had to be on-prem, be able to segment data by vlan, run VMs with no outside connectivity (more on that later), and have shared storage. There were more but those were the general guidelines. For testing the first thing I installed was Single Node OpenShift (SNO), and that’s what I will start going over here. It does do the job decently well enough, but the ramp up is rough. Gone are the VMware nice installers, and welcome to writing YAML files.

The big other players were systems like Hyper-V, Nutanix, Proxmox, Xen Orchestra, KVM. We are not a big Microsoft shop and a lot of our devs had a bad experience with Hyper-V, so we scratched that one. Also, Hyper-V doesn’t seemed all that loved by Microsoft for on-prem, so that turned us away. I investigated Nutanix but they have a specific group of hardware they want to work with, and a very specific disk configuration where each server needs 3 + SSDs to run the base install. I did not want to deal with that, so we moved on before even piloting it. Proxmox is a community favorite, but we didn’t want to use that for production networks, and thought getting it passed security teams at our customers would be difficult. Xen Orchestra is getting better but in testing had some rough spots and getting the cluster manager going gave some difficulty. This left raw KVM, and that was a non-starter because we want users to easily be able to manage the cluster.

Without finding a great alternative, and the company already wanting to push forward on Redhat OpenShift, I started diving into what it would take to get VMs to where we needed them to be. What I generally found is there is a working solution here, that Redhat is quickly iterating on. It is NOT 1:1 with VMware. You are running VMs within Pods in a K8s cluster. That means you get the flexibility of K8s and the ability to set things up how you want; along with the troubles and difficulties of it. Like Linux, the great thing about K8s is there are 1000 ways to do anything, that also is its greatest weakness.

Footnotes / Reading Materials

DNS Entries needed for normal use:

Chapter 2. Installing OpenShift on a single node | Installing on a single node | OpenShift Container Platform | 4.18 | Red Hat Documentation

SNO on OCP-V – OpenShift Examples

Red Hat OpenShift Single Node – Assisted Installer – vMattroman

Fedora CoreOS VMware Install and Basic Ignition File Example – Virtualization Howto

https://docs.redhat.com/en/documentation/openshift_container_platform/4.18/html/installing_on_bare_metal/user-provisioned-infrastructure#installation-user-infra-machines-advanced_vardisk_installing-restricted-networks-bare-metal

butane/docs/config-openshift-v4_18.md at main · coreos/butane

Some useful information for networking: Deploying Single Node Openshift (SNO) on Bare Metal — Detailed Cookbook | by Reishit Kosef | Medium

Offline installs

https://hackmd.io/@johnsimcall/Sk1gG5G6o

Mellanox Driver Ignoring Vlans

Recently I spend more hours than I want to talk about fixing a server that had a Mellanox ConnectX6-lx card, where I could not get Openshift to get traffic to VMs. I was creating bridges just like I normally do, and traffic was working to the main interface. After a lot of trial and error I wanted to make a quick post in case anyone runs into this.

All of this assumes a trunk port to an interface on Linux (or a bonded interface). If you have an interface in Linux on the native vlan, and that is a standard interface (example eno1). Then you add a sub interface for tagged traffic, eno1.10, the Mellanox mlx5 driver will – in hardware -ignore your vlan tag and just send traffic to the main interface.

One way to see if your card is doing this is search dmesg for “mlx5”: dmesg | grep mlx5, you may see the following:

mlx5_core 0000:0b:00.1: mlx5e_fs_set_rx_mode_work:843:(pid 156): S-tagged traffic will be dropped while C-tag vlan stripping is enabled

(https://github.com/oracle/linux-uek/issues/20)

The Mellanox card is worried about double tagged packets and will drop tags on data coming in. It does this in hardware. You can see the settings for 8021Q kernel module being loaded, and vlan filtering is disabled, but this wont matter. If you change settings like ethtool -K <interface> rx-vlan-offload off it will say the setting is off, but the underlying driver loaded this at init time, and then the settings you set will be ignored. The only way I found to fix it is to move all the IPed interfaces off the main interface.

Once you move the IPed interface under its own sub interface and reboot, data will start flowing to your VMs.

Homelab Dial-Up Networking for Retro Computing

I started mentioning about my LAN Before Time project before. The idea is to have a rack of the most diverse CPU, OS, and Networking technologies I can find. Each computer and piece of equipment bringing something new and unique to the collection. One part of this collection was the network. Coming from a networking background, I also wanted to have the most diverse set of networking technologies that I could. To do this I would need to find a core router that could talk all the different protocols I wanted. Having worked at Cisco in the past, I knew they had gear that went back in time, and could add in some additional fun like AppleTalk. That is where I started my search.

Why the Cisco 3825

I knew I needed a Router and not a switch, because we would be going between several protocols. I could time box the router a bit because I wanted to support AppleTalk and IPX. Support from these no longer exists, and thus would give a cutoff for devices I could get.

Years ago, I was the Cisco 7200 VXR guy at work. They were great routers, that could go up to gigabit and go in and out of a ton of different connections, including some voice ones. The issue was they could do lines like T1, but they couldn’t host an analog modem, and that was one thing I very much wanted.

Now we are getting down to a select group of Cisco devices. I started making a table of the pros and cons of each. I also was hoping for something slightly newer to have 1gb/s links available, and hoping that it wouldn’t use a 1000 watts at idle.

This is a table I made of all the different options for routers and the features they supported:

Model (U)	Small Bays	Large Bays	Token Ring	IPX / Appletalk	Dial-Up	Base Speed	EOL Date	Flash
2513 (1)	0	0	Y		?	10mb	Before time	Internal?
3640 (2)	0	4	Y	Y / Y	N	100mb	2008-12-31	PCMCIA
3725 (2)	3	3	Y	Y / Y	Y	100mb	2012-03-31	CF
3745 (3)	3	4	Y	Y / Y	Y	100mb	2012-03-31	CF
3825 (2)	4	3	Y	Y / Y	Y	1000mb	2016-11-1	CF + USB
3845 (3)	4	4	Y	Y / Y	Y	1000mb	2016-11-1	CF + USB
3945 (3)	3	4	N	N / N	Y	1000mb	2022-12-31
7206XVR (3)	6	0	Y	Y / Y	N	1000mb	2017-02-28?	CF/PCMICA

The Cisco 1800/2800/3800 line all came out around the same time, they were considered the G1 generation routers. Things before them were legacy, and the systems after them like the 3945 were the G2 devices. G2 was a step to get rid of legacy, this means dropping AppleTalk, IPX and some of the things I was interested in. By this point I had realistically narrowed down to a Cisco 3725 or 3825. The 3825 had gigabit on the main controller, allowing my to get a little closer to modern systems; that put me over the edge for it. I also was not interested in its giant cousin, the 3845; the 3825 had enough bays and should be quiet enough for me to run routinely.

In searching for the voice features I wanted, I found I would need the Advanced Services image loaded, and that required a minimum of 512MB of memory. I ordered a system with Advanced Services and 1GB of memory, the max! When it came it had 256MB! I emailed the seller and he mailed me the proper 1GB of RAM. Once I installed it, the system would boot loop, I found one of the 2 sticks was bad. I ended up putting in one of the 512MB sticks he sent, and the 256MB one it came with for a total of 768MB of RAM.

Upgrading the Cisco 3825

Once I had the device on order, I started learning about the world of voice. I had not done much with voice before, there was a lot to learn. First, I needed to know which cards I would need. The 3825 can run older WIC cards, and the newer high speed HWIC cards. I needed analog modems, and to host those, I needed FX-S/FX-O cards. In learning about them: FX-S Foreign Exchange Station, FX-O Office. The office is for a branch office dialing OUT. You usually want FX-S because it provides a dial tone to dial into. S is to get a dial tone and you can call INTO. O is it receives a dial tone.

I got 4 FX-S Ports. One port is used to dial into, another to route the call out. I need more ports than you would think because of this. I got a CISCO VIC3-4FXS/DID, this gives 4 ports in/out. 2 CISCO WIC-1AM-V2, giving 2 modems I can dial into. There are cards that are 2 modems in one card, but they are rare and expensive. (I would get one later)

The system now has 768MB of memory, but I would also need PVDMs; Packet Voice Digital Signal Processor (DSP) Module (PVDM2). I had not worked with these before. They are ASICs that come process voice channels. They come in different numbers of channels, PVDM2-8, PVDM2-16, PVDM2-32, PVDM2-48, PVDM2-64. They were cheap, obviously I got 4 PVDM2-64 to go to 256 voice channels for $20 USD. The math is odd here, there are different complexity channels and some knock you down to 1/2 or less of the listed available channels. A “high complexity codec” can make an 8-channel card only handle 4 or less channels. Like I said, these were cheap, so I got 4 256-Channel cards. The cards go into slots that look like DIMMs.

I bought a T1 card because I thought it would be fun to eventually play with. Then I bought a CISCO NME-16ES-1G-P. This is a Cisco Switch as a card module. They are odd. It runs its own OS. It has an internal port to the main system, then to configure any of the 16 x 10/100mb/s or 1 x 1gb/s ports you run ‘service-module gigabitEthernet 2/0 session’ and it opens a serial connection into the controller of this switch. It has a separate login, configuration, and IOS version. At first, I had to password bypass it because someone had left a password on it. I have used it some, but not much. It needs a firmware update. This is my main source of ethernet ports for devices.

The last card I got is one of the main ones I wanted, a CISCO NM-1FE-1R-2W. This is a 1 FastEthernet (10/100), 1 Token Ring (DB9 or RJ45), and 2 WIC slots. I will do another post on Token Ring later.

Its worth mentioning, for the configuration and tests below, ports 0/0/0 – 3 are our 4FXS ports in HWIC slot 0. In HWIC1 slot (0/1/0 – 1) we have our dual modem card now, a WIC-2AM-v2.

Configuration

As mentioned, you do need a router with the Advanced Services image on it to do voice related features. Once that is loaded, and our hardware was in place, we started configuring the router to move voice data.

We need a pool of IPs that will be given out to clients as they dial in. To do this enter configuration mode and use:

ip local pool dialin-pool 192.168.9.10 192.168.9.20 recycle delay 3600

You can tell from reading this blog that I have a way too complicated home network. I have the router itself on 192.168.7.0/24, then I made 192.168.9.0/24 for dial in addresses. To be able to send data to them and have them route on my core network, I have RIP running between my home firewall and this Cisco router. When I turn the router on, the routes pop up; when I stop using it for a while, the routes go away. You can set the start and stop of your pool to whatever you like.

Next we have to declare when you dial a certain number, where that routes. This is created by declaring a route, saying its a POTS line, and then saying which “Dial pattern” (phone number) goes to which port on the router. The dial-peer name can be anything, I use the number to make it easy. Port 0/0/3 is the 4th port on the FXS that I am CONNECTING OUT OF to the modem. The data flow is 0/0/0 where my Windows 95 client is, through the FXS to 0/0/3, to a 6 inch RJ11 cable to 0/1/0 modem slot. Some parts like FXS routing happen within the cards, then there are times you need to have tiny patch cables to route the stream.

dial-peer voice 3 pots 
 destination-pattern 3 
 port 0/0/3

We need to say how the connection is handled once it comes in the modem. These connections are handled as “line” interfaces from there. We need to tell it this is a dial in line, what its max supported speed is (I just do the max, it will communicate less to the end modem), how flow control will work, and how auth will work.

The line configuration says it beeds ppp, since I do not specify the the auth system, any user is currently allowed… which is great!

line 0/1/0 
 modem Dialin 
 modem autoconfigure discovery 
 transport input all 
 autoselect ppp 
 stopbits 1 
 speed 115200 
 flowcontrol hardware

Last, we need to configure the IP interface of this line, we do that by configuring the async line assigned to the modem port. This also is where we set which pool will be used for dial in users. I added the ppp timeout command because some of the older systems I have were taking a while to respond.

interface Async0/1/0 
 ip address 192.168.9.1 255.255.255.0 
 encapsulation ppp 
 peer default ip address pool dialin-pool
 async mode interactive 
 no keepalive 
 ppp timeout authentication 30

That is the key configuration needed to get dial up working! Below I will put my full config (minus password) in case any of it helps someone. Leave a comment if this helps you, or you need extra help!

Full Configuration

hostname router 
! 
boot-start-marker 
boot system flash:c3825-adventerprisek9-mz.151-4.M10.bin 
boot-end-marker 
! 
enable secret 0 test
! 
aaa new-model 
aaa authentication login default local line 
!
aaa session-id common 
no network-clock-participate slot 1 
dot11 syslog 
ip source-route 
ip cef 
! 
ip dhcp pool TOKEN 
 network 192.168.8.0 255.255.255.0 
 default-router 192.168.8.1 
 dns-server 192.168.7.1 
!
ip domain name lbt.home.ntbl.co 
no ipv6 cef 
!
multilink bundle-name authenticated 
voice-card 0 
crypto pki token default removal timeout 0 
username admin privilege 15 secret 0 admin
username test privilege 0 password 0 test 
!
redundancy 
!
ip ssh version 2 
!
interface GigabitEthernet0/0 
 ip address 192.168.7.10 255.255.255.0 
 duplex auto 
 speed auto 
 media-type rj45 
! 
interface GigabitEthernet0/1 
 no ip address 
 shutdown 
 duplex auto 
 speed auto 
 media-type rj45 
!
interface Serial0/3/0 
 no ip address 
 shutdown 
 clock rate 2000000 
! 
interface FastEthernet1/0 
 no ip address 
 shutdown 
 duplex auto 
 speed auto 
! 
interface TokenRing1/0 
 ip address 192.168.8.1 255.255.255.0 
 ring-speed 4 
! 
interface GigabitEthernet2/0 
 ip address 100.64.0.1 255.255.255.0 
!
interface Async0/1/0 
 ip address 192.168.9.1 255.255.255.0 
 encapsulation ppp 
 peer default ip address pool dialin-pool 
 async mode interactive 
 no keepalive 
 ppp timeout authentication 30 
!
interface Async0/1/1 
 no ip address 
 encapsulation slip 
! 
interface Async0/2/0 
 no ip address 
 encapsulation slip 
! 
interface Async1/0/0 
 no ip address 
 encapsulation slip 
! 
router rip
 network 100.0.0.0 
 network 192.168.7.0 
 network 192.168.9.0 
 neighbor 100.64.0.2 
 neighbor 192.168.7.1 
!
ip local pool dialin-pool 192.168.9.10 192.168.9.20 recycle delay 3600 
ip forward-protocol nd 
no ip http server 
no ip http secure-server 
! 
ip route 0.0.0.0 0.0.0.0 192.168.7.1 
!
control-plane
!
voice-port 0/0/0
!
voice-port 0/0/1
!
voice-port 0/0/2
! 
voice-port 0/0/3 
!
mgcp profile default 
! 
dial-peer voice 3 pots 
 destination-pattern 3 
 port 0/0/3 
!
telephony-service 
 max-conferences 12 gain -6 
 transfer-system full-consult 
!
line con 0 
line aux 0 
line 0/1/0 
 modem Dialin 
 modem autoconfigure discovery 
 transport input all 
 autoselect ppp 
 stopbits 1 
 speed 115200 
 flowcontrol hardware 
line 0/1/1 
 stopbits 1 
 speed 115200 
 flowcontrol hardware 
line 0/2/0 
 stopbits 1 
 speed 115200 
 flowcontrol hardware 
line 1/0/0 
 stopbits 1 
 speed 115200 
 flowcontrol hardware 
line 130 
 no activation-character 
 no exec 
 transport preferred none 
 transport input all 
 transport output lat pad telnet rlogin lapb-ta mop udptn v120 ssh 
line vty 0 4 
 transport input ssh 
!
scheduler allocate 20000 1000 
end

The High Nibble Cromemco Z-1 Replica

The High Nibble is back with another fun retro computer kit. This one of the Cromemco Z-1. I reached out to the creator of the kit when it first went up, and he asked if I was sure I was interested. The kit is 99% the same as the IMSAI 8080, including most of the components except the acrylic panel and new firmware. The kits are so similar the PCB for the main board is the IMSAI 8080. I enjoyed the last one so much and wanted to support the creator (not to mention a new kit means more blinky lights on the shelf); thus, I ordered the new kit.

The kits are not cheap at $300, but come with everything you need, including a ready to go controller. The packaging is done well with individually slotted spots for each switch and IC in cardboard. The metal case is a nice touch. And just like before, the firmware is fantastic. You get a full web interface, external serial ports, the gorgeous front panel, and the system can update over the air update via Wi-Fi.

I won’t go too in-depth about the kit because it is so similar to the other one. The soldering is not too bad, the process starts with one small IC that has to be soldered in the front of the pcb, after that things like sockets and LEDs are easy, large, and straight forward.

I got a new soldering iron, the Pinecil from Pine64. This was my first project using it and I found it delightful. It’s smaller than my old iron, making it easier to handle. It’s advertised as a “smart” soldering iron which means when you put it down in a holder it quickly cools when not in use. A very nice safety feature.

The parts of the last build that were difficult are here too, having to put the system together as a sandwich of acrylic and circuit boards it’s a bit tricky, but once you get it in the right spot tightens up easily. Tape helped keep the build aligned during assembly.

The website for the project has a lot of good information, https://thehighnibble.com/cromemcoZ1/. There’s also a helpful YouTube video, same one as the IMSAI kit, that walks through the construction step-by-step.

The main step that gave me an issue was when you get to Testing, the LS2 light would not come on. After looking in the forums, https://github.com/orgs/thehighnibble/discussions/120, turns out that is a difference in the firmware, and it is not supposed to come on. With the video being for the other kit, it is not mentioned.

Overall, another great kit, and another fun system to add to the collection!