Projects

Homelab: Overview

I am starting a series about my homelab and how it is all laid out. I have written this article a few times, with months in between. Each time the setup changes, but we seem to be at a stable-ish point where I will start this series. Since I wrote this whole article and now a while later am editing it, I will mark with italics and underline when present me is filling in. I think it will give a neat split of growth in the last year or so I have been working on this. Or it will make it illegible, we will see. My home setup gives me a good chance to test out different operating systems and configs in a domain environment before using that tech elsewhere like at work.

Hypervisor

Starting off with virtualization technology, I settled a while ago on Microsoft Hyper-V instead of ESXi, the main reason behind it is I already had Windows Server, and Hyper-V allows for Dynamic memory, and allocating a range of memory for a VM. When something like an AD controller is idling, it doesn’t need much memory; when it starts it may, Dynamic memory allows me to take that into account. I will say one place that has bit me later is file storage, but that will be a later post.

The setup is technically “router on a stick”, where the Sophos XG firewall functions as the router, and the rest of the devices hang off of that. The Sophos XG machine is a old Dell Optiplex 990 (almost 10 years old!) with an Intel quad NIC in it. That way it can do hardware offloading for most of the traffic. I intend to do posts for networking, hypervisors, file storage, domain, and more; thus I will not get too in the weeds right now on the particulars.

The file storage is a FreeNAS box recently updated to 7, 3TB HDDS. I have had this box for about over 6 years (I just looked it up in November 2020, one of the drives has 55257 hours or 6.3 years of run time on it); it is older but has worked well for me so far.

The network backbone is a new switch I really like that I was able to get 2 of off eBay; they were broken but I was able to repair them, more on that later as well. They are Brocade, now Ruckus, ICX7150-12P; 12 1GB/s POE ports, 2 additional 1GB/s uplink port, and 2, 1/10GB/s SFP/SFP+ ports. These switches can run at layer 3, but I have the layer 2 firmware on them currently. They have a fiber connection between them, before that I was using 2 Unifi APs in a bridge, that didn’t work fantastic however because A. I am in NY, B. they were only 2×2 802.11AC Wave 1, and C. I am in NY. I custom ordered (so the significant other would not get mad) a white 50m fiber cable to go around the wall of the apartment.

With SSDs in the hypervisor boxes (I call them HV# for short) and iSCSI storage for VMs as well, which VMs are on which host doesn’t particularly matter. Flash forward 6 months or so, since that first sentence was written, I now still use the NAS for backups, but the hypervisors are running Storage Spaces Directed and doing shared storage now. This allows the hypervisors to move move VMs around during patching or pause during a system update if they are less critical. The Intel NUC and small Dell Inspiron are much under powered compared to the mid tower hypervisors, so they run usually only 1 or 2 things. The NUC runs the primary older domain controller, and that is it. It is an older NUC that I got about 7 years ago, so its not that fast. The “servers” in the hypervisor failover cluster are a Lenovo and 2 Dell Optiplex 5050s. I like these Dells because they go for about $200 on eBay, while having a Intel 7600 i5, can support 64GB of ram, and have expansion slots for things like 10gb SFP+ cards. These machines also idle at about 30 watts, which makes the power bill more reasonable.

Some of the services I run include:

  • 2 Domain Controllers (Server 2016, and 2019)
    • Including Routing and Access service for RADIUS and 802.1x on wifi on wired
  • Windows Admin Center Server (Windows Server 2019)
  • Windows Bastion (This box does Windows Management) (Server 2019)
  • Veeam Server (Server 2019)
  • Unifi Controller/Unifi Video for security camera (Ubuntu)
  • 3 Elastic Search boxes for ELK (CentOS 8)
  • Linux Bastion (CentOS 8)
  • Foreman Server (CentOS 8)
  • LibreNMS (This I grew to really like) (CentOS 8)
  • Nessus Server (CentOS 8)
  • Jira Server (CentOS 8)

That is the general overview, I will spend the next while diving into each bit and discussing how it is configured and what I learned in doing that.

Redhat/CentOS 7-8 PKI/CAC/Smart Card SSH Login with Active Directory and SSSD

I was experimenting with integrating CentOS with my home Active Directory (AD) cluster. I wanted centralized user management, and for a stretch goal, get PKI login working for Smart Card auth. I have used winbind before to connect CentOS 6 to Active Directory, that configuration before was a bit annoying. These days with CentOS/RHEL 7 and 8 we have SSSD, which is more straight forward. For all the following tests I used Putty-CAC (link), a Windows app that allows GSSAPI, and Smart Card auth.

SSSD Config

I will start off with my experience, then follow up with a how to; for this article I already have AD configured to support Smart Card auth, and have stored the Smart Card public key for my user. I will follow up with an article about that configuration. Active Directory integration is straight forward and easy. One setting you can enable is: hiding the domain names from the username, this allows the users to feel native to the system. Using users and groups are easy; I made a group to which I gave sudo access. When using Smart Cards you will need to put NOPASSWD in the sudo entry for that group, because the Smart Card users usually do not have passwords, usually… You can use Smart Card auth with Active Directory AND a password as long as you do not set “Smart card is required for interactive logon”. If you do check that box, AD sets a random password on the backend for that user.

After setup, with this config we store the authorized_keys in AD under the attribute altSecurityIdentities. The main tool to debug Smart Card auth is the tool sss_ssh_authorizedkeys, this allows you to have the system attempt to pull their ssh key on demand. A big warning about SSSD, it loves to cache information. If you attempt to run that command, and then make changes to your sssd.conf or AD, and re-run sss_ssh_authorizedkeys, it will fail because it is caching the failed lookup from before. My recommended command as root between tests where it may be caching is:

systemctl stop sssd && rm -rf /var/lib/sss/db/* && rm -rf /var/lib/sss/mc/* && systemctl start sssd

SSSD Config

1. Setup hostnamectl (make sure your host knows what its name is supposed to be) and dns, for SSSD to work well you need the system to be able to find itself in DNS, you can set up SSSD to auto register with dynamic DNS (more on that later)
2. Install Packages
     - Ubuntu
       apt -y install realmd sssd sssd-tools libnss-sss libpam-sss adcli samba-common-bin oddjob oddjob-mkhomedir packagekit    
     - CentOS
       sudo yum install realmd sssd oddjob oddjob-mkhomedir adcli samba-common samba-common-tools krb5-workstation       

At this point running “# realm discover your_domain_fqdn” will list out services your domain needs for users to login. Usually the main program you need to enable is oddjobd which will create home directories when users login. Note, for these examples I find it easier to have a domain in them than the subsistute it, I will use my home test domain “home.ntbl.co” here.

3. systemctl enable oddjobd
4. systemctl start oddjobd
5. realm join -U admin_user_on_domain home.ntbl.co
6. vim /etc/sudoers.d/winadmins
Add the line “%domain\ admins@home.ntbl.co ALL=(ALL) ALL“, where “domain admins” is a group I have in AD, and “home.ntbl.co” is my domain. This setup does not support Smart Card login with sudo, since you need NOPASSWD for that sudo login. Example "%domain\ admins@home.ntbl.co ALL=(ALL) NOPASSWD:ALL". You can create a sub sudo file like I did here, or visudo to edit sudo and have it syntax checked.


7. Below is my /etc/sssd/sssd.conf without Smart Card auth setup.

 [sssd]
 domains = home.ntbl.co
 config_file_version = 2
 services = nss, pam
  
 [domain/home.ntbl.co]
 ad_domain = home.ntbl.co
 krb5_realm = HOME.NTBL.CO
 realmd_tags = manages-system joined-with-adcli
 cache_credentials = True
 id_provider = ad
 krb5_store_password_if_offline = True
 default_shell = /bin/bash
 ldap_id_mapping = True
 use_fully_qualified_names = false
 fallback_homedir = /home/%u@%d
 access_provider = ad
  
 dyndns_update = true
 dyndns_refresh_interval = 43200
 dyndns_update_ptr = true
 dyndns_ttl = 3600 

Adding “use_fully_qualified_names” changes your username from “dan@home.ntbl.co” to “dan”. Not a requirement, but a nice, quality of life setting. The bottom adds dynamic dns, which will push your IP to AD DNS. Windows does dynamic DNS updates by default, and unless the systems are statically assigned, or even if they are, this can be a nice feature. Now "systemctl stop sssd" and “systemctl start sssd”, then you should be able to login with your AD account.

GSSAPI

Before getting into Smart Card auth, I wanted to briefly mention GSSAPI. This is a method to do auth between systems. It allows Windows clients to one click login to SSH by passing an auth token from your Windows session right to SSH. If you setup SSSD, enable GSSAPIAuthentication in /etc/ssh/sshd_config then you can use an app like Putty-CAC to SSH with GSSAPI. I have found this usually works with SSSD by just setting GSSAPI to yes. If you just want to admin Linux from AD, and have no other requirements I would suggest you look into this for your environment because it is so easy. If yo are going to follow the rest of the guide, make sure to turn GSSAPI back off, or it will log you in automatically and you may think its Smart Card auth working; that fooled me for a few minutes.

Smart Card Auth

For all of my tests, I used the following Smart Card, Amazon link. I think these other cards would work as well, and they are cheaper; but I have not personally tried them. Amazon link. I may write an article later about setting up these cards, if you are interested write a comment below.

Add Certs to AD

You need the Smart Card’s public key data in SSH authorized_keys format. This guide will show you how to get that string from Putty CAC. You have to enjoy when a .gov site tells you to go to user NoMoreFood and get security software, the open source world is great.

In Active Directory, go to Active Directory Users and Computers, turn on Advanced Features, by going to the View menu, and enabling Advanced Features. Then select the user you want to add ssh keys for, and select the “Attribute Editor” tab. You will find an entry at the top called “altSecurityIdentities”, add the line that would usually be in ~/.ssh/authorized_keys there, it should look like “ssh-rsa key_stuff”.

Configuring SSSD for Cert Auth

To add Smart Card auth to SSSD, just add the following to your sssd.conf, merge the sections with the ones from above.

[sssd]
services = nss, pam, ssh, sudo

[pam]
pam_cert_auth = True

[domain/home.ntbl.co]
enumerate = True
ldap_user_extra_attrs = altSecurityIdentities:altSecurityIdentities
ldap_user_ssh_public_key = altSecurityIdentities
ldap_use_tokengroups = True

Now restart sssd. If you run "sss_ssh_authorizedkeys dan" with dan replaced with your name, then you SHOULD get a key back if everything is setup correctly. If you do not get a key back, use the command below to reset sssd and reload. If you still do not get a key then you will need to edit settings in sssd.conf, and continue to tweak:

systemctl stop sssd && rm -rf /var/lib/sss/db/* && rm -rf /var/lib/sss/mc/* && systemctl start sssd

I will say this does seem to take some trial and error. /var/log/sssd/ has some good logs that can help point you in the correct direction if you are running into issues. One quick note I will make, you may see people online say “use the command ‘sss_ssh_authorizedkeys -debug 4 home.ntblc.o’ to debug the command.” This command does not have a debug throw, that that does is uses the -d argument which is domain, then tries to parse the rest. You end up with key lookup attempts on domain “ebug” for user 4. Sadly sss_ssh_authorizedkeys is not very verbose, debugging it is a bit of a pain; do not listen to people who mention the above debug command, at least on CentOS/Rhel 7 and 8 it does not work.

As long as you are getting a key back from the above command, then you can wire it into SSH. Edit /etc/ssh/sshd_config with the following, note some sites say AuthorizedKeysCommandUser should be root, some say it should be nobody. I error on the side of lesser permissions and set it to:

 AuthorizedKeysCommand /usr/bin/sss_ssh_authorizedkeys
 AuthorizedKeysCommandUser nobody

Hope something here has helped someone, feel free to drop a comment.

Finished Front

IMSAI 8080 Replica Kit

I emailed the creator of the IMSAI 8080 kit over at https://thehighnibble.com/imsai8080/#overview when I saw the kit he had, but didn’t know if any were available; then one day out of the blue get an email saying a kit is ready for order! They come out in batches, this makes it take a bit to get them sometimes. To start off I can say this is one of the more well put together kits I have gotten. I was a little worried when I looked at the instructions and it mentioned a lot of part numbers and no photos of the setup; then I realized I had missed a link to a video that walks through the whole process, making it much easier.

The Kit

The kit is one of the more professionally put together kits I have done. From the metal case the system sits in, to the cut out cardboard all the parts come in. The creator, Dave, put a lot of attention to detail into everything. This kit includes everything from serial console lines on the back, to the micro controller having a WiFi antenna, which you can access over WiFi. The shipping box is also the correct size for the unit, you can use it as storage after you are done with the project. This kit comes with what you need ready to go and be put on the board (except solder and flux); some of the kits I have had in the past have required painting. You do pay a bit of a premium for it, but I think it is worth it. The kit also allows RS-232 connectivity or USB directly into the ESP32 controller.

Assembly

I will start off saying the hardest part of putting the kit together is at the beginning. There is a tiny chip for memory and that needs to be surface mounted to the board. This involves getting a small amount of solder onto the pads and then heating them with the chip on it. I accidentally bridged 2 pads and spent a chunk of time getting the tiny amount of solder off. After that part is over, you move onto the micro SD card reader, this is a similar surface mount; yet either I got better after doing the first one or got lucky, that was put down relatively quickly.

There is a diagnostic program you can run at that point to check if what you have soldered so far is working (which is fantastic), after that it is time for you to press on to soldering all the LEDs and their corresponding pieces. This part of the process is much like other kits of this type you may have done. It takes a while, but is a straight forward process.

In the end you sandwich the board together, and get it screwed into the case. There are a few parts where its gets a little tricky if you don’t want to have to pull the whole thing apart, but the video guide easily walks you through it.

Software

The software for the unit is great; the projects website, thehighnibble.com has instructions on how to use it. Similar to other projects there are certain switches you can flip to send the system different commands. But then, my favorite part is you can connect to the device over WiFi (either adhoc or with a AP) and then see the terminal. This allows you to leave the system on a shelf or a desk, and play with it at the same time. This interface also allows you to edit which “disks” are inserted into the system, to easily change images.

Overall it was a very enjoyable kit, other than the first little bit with the surface mounts. I think any beginner could do it once it is past that point. The resistor and LED soldering is straight forward. The metal case, and back clear acrylic make it stand out and look great. The WiFi and web interface put it over the top, as a it you can play with and connect to easily.

Powering up! It’s very shiny

3D Printing and Thingiverse Login Fix

I recently got a new 3D printer (Ender 3 Pro), and thought I would put up some of the small things I have recently printed. In trying to print things from Thingiverse, I couldn’t login even after making an account. I would get a spinning “Logging in” and it would never end. After looking at the network log, I saw it trying to reach out to https://accounts.thingiverse.com/unverified?username=danberk If you run into this issue, go to that URL with your username and it will send you an email to verify your account. Then the site will allow you to login.

Ruckus ICX 7150 sideways shelf mount

I have been using Ruckus ICX 7150-12P switches at home recently, I wanted to have it more out of the way; so I designed and printed a mount that would mount the switch to the side. It came out well and looks good! I also printed a network cable comb to hold all the cables nicely together.

SD card and USB holder!

This is a nice little SD card, micro SD, and USB holder. I 3M stuck it to the shelf I have, next to the printer.

Mac SE with battery mount

I have already posted about the Mac SE Battery mount I made. I put the design up if anyone is interested.

iOS/macOS On-Demand IPsec VPN with Sophos XG

Having a small home lab I wanted to be able to setup internal services, and then on the go be able to access them. While I could setup a L2TP or SSL VPN and connect whenever I wanted to use these services, I thought I would give On-Demand VPN via a iOS/macOS configuration a try. Little did I know the world of hurt I was entering. I will start with the settings you need to get it working, since a lot of people just want that. Then I will talk about the crazy and painful road I went down before finding 1, just 1, set of settings that seem to work. If you have any questions, thoughts, or success stories please comment below!

Fun fact: I will be calling the protocol IPsec here. That is what the original RFC called it, what the original working group was called, and the capitalization they used. Sophos agrees and uses that capitalization, while Cisco and depending on which web page you are on for Microsoft may call it IPSEC or IPSec or IPsec.

On-Demand VPN gives you the ability to set certain websites or IPs, and when your phone or laptop attempts to connect, the machine silently brings a IPsec tunnel online and uses it for that traffic. This allows you to run services at home, and to users (your mom or cat or whomever) it looks like just another website. Apple has 1 big requirement for them, you have to use certificate based auth. You can not use a pre-shared key/password. Also up front, to save you a few days of trying things. iOS and macOS will NOT check your certificate store for your VPN endpoint (Sophos XG) certificate, it HAS to ship with the firmware or you will get the fantastic and descriptive “Could not validate the server certificate.” Also believe it or not, that is one of the most descriptive errors you will get here. There are some posts on the Apple support forums from Apple engineers saying the root CA has to be in already on the device. If anyone gets it to work with your own let me know.

Sophos XG Setup

I am using Sophos XG v18 with a Home license, backed by AD running on a Dell Optiplex for this guide (dont worry it as a cool Intel Nic in it). To setup the IPsec server in Sophos XG first we need to make 2 certificates. Login to the admin portal, then on the bottom left select “Certificates”. You need 2 certificates; 1 is our “local certificate” (we will call it Cert-A) this is a cert that is used for the server (Sophos) end. As previously mentioned, this has to be a real signed cert. I ended up forwarding a subdomain on my site to the firewall, and then using Let’s Encrypt to create a cert for that URL. I used this site, https://hometechhacker.com/letsencrypt-certificate-dns-verification-noip/ to guide me in creating the cert on my laptop, then I uploaded that to the Sophos firewall. This will require you to have access to your domains DNS settings or be able to host a web file.

The second cert (Cert-B) is for the client, Sophos will call it “The Remote Cert”; this is to auth to the firewall, that can just be a locally generated cert. All devices will share this cert. The devices will use their username and password combination to identify the user. I used email as the cert ID, note this email does not have to exist, I just made one up on my domain so I will know what this cert is. Once created, go back to the main Certificates page and download the client/remote certificate, I suggest putting an encryption password on it since the Apple tools seem to freak out if that is missing. But ALSO the password for this cert will be in clear text in your config, so don’t make it a password you care about. These certs all need to be rotated at least once a year, with the newer requirements; Let’s Encrypt is every 90 days and I intend on automating that on one of the Linux machines I have.

Self-signed client end cert

Now that we have our 2 certificates, lets go over to “VPN” on the left hand navigation. I have tried many settings in the main “IPsec Connections,” and none of them have worked for me. I get fun and generic errors from the Mac of “received IKE message with invalid SPI (759004) from other side” or “PeerInvalidSyntax: Failed to process IKE SA Init packet (connect)”.

Click the “Sophos Connect Client” tab, the back end of this client is just a well setup IPsec connection. Fill in the form, from the external interface you want to use, to selecting “Digital certificate” as your auth method, followed by the “Local certificate” which is the Let’s Encrypt one (Cert-A). “Remote certificate” is the one we will load on your device (Cert-B).

Now you select which users you want to have access to use this. I have Active Directory backing my system, so I can select the AD users who have logged in before to the User Portal. This is a trick to Sophos XG you may need, if you use AD and a user doesn’t show up, that means they need to login to the User Portal first.

Select an IP range to give these clients, I suggest something outside any of your normal ranges, then you can set the firewall rules and know no other systems are getting caught in them. Once you are happy, or fill in other settings you want like DNS servers, click “Apply”. After a second it will activate, you can download the Windows and Mac client here, or follow along to make a profile.

Apple Configuration

To create a configuration file you need to download Apple Configurator 2, https://apps.apple.com/us/app/apple-configurator-2/id1037126344 onto a Mac. I know what you are thinking, 2.1 Stars, Apple must love enterprises. Download that from the store and open it up. If you do not have a Mac I attached a template that you can edit as a text document down below. This profile needs a Name, as well as an identifier. The identifier is used to track this config uniquely, if you update the profile, then your device will override old configs instead of merging. You will see on the left there are LOTS of options you can set, the only 2 week need are “Certificates” and “VPN”.

Starting with Certificates, click into that section, then hit the Plus in the top right. Upload the cert we exported from Sophos (Cert-B) earlier for the end device, and enter the password for it. Again note, this password is in plain text in the config file.

Now for the VPN Section. Click the Plus in the top right again to make a new profile, name the connection anything. Set the Connection Type to “IPsec”. IKEv2 is IPsec but a newer version, I will get into some of this later after our config is done and I can rant. Server is your Sophos XG URL. Account and password can be entered here to ease setup, or you can leave one or both blank to make the user enter it when they import the config. You can leave the user/password fields blank (it will give you a yellow triangle but that is fine) and then give it out widely and not have your creds in it… For “Machine Authentication” you want “Certificate”; you will see in selecting “Certificate” all of a sudden the On-Demand area appears. For “Identity Certificate” select the one we uploaded before. Finally we can enable “Enable VPN On Demand” and select the IPs or URLs you want to trigger the VPN.

Once that is done, save the profile and open it on a Mac or you can use this configuration tool to upload it to an iOS device. That should be it! Your devices should be able to start the connection if you ask it, and if you go to the website should auto vpn. Make sure you have firewall rules in Sophos XG for this new IP range, or that can block you from being able to access things.

A small note, from my tinkerings with the On Demand profile if you go to Safari on a iOS device, it will connect when you visit a website that is in the configuration. If you use a random app, such as an SSH application, I didn’t find it always bringing the tunnel up, and at times it had to manually be started. Something to lookout for, a nice part of the the IPsec tunnel is that it starts quickly.

Now that the config is done, I want to mention some of the other things I have learned in tinkering with this for several days. The only way I got it to work is using that Sophos Connect area, and the other big not documented thing is you have to use a publicly trusted cert for the Sophos end. I found 1 Apple engineer mention this on their forum, and a TON of people talking about how they couldn’t get the tunnel to work with their private CA. I have tried uploading a CA, and injecting it different places with different privileges for the Mac and never could get it to work. The Let’s Encrypt cert imminently worked.

For IPsec v1, aka IKEv1, Apple uses the BSD program racoon on the backend to manage the connection. Using the “Console” app you can find the logs of this. For IKEv2 it seems Apple wrote their own client around 2016-2018, there are a lot of reports online that it just doesnt work at all with cert based auth. All the guides about it working stop around 2016. You can find earlier ones, or people using pre-shared keys, but selecting pre-shared keys doesnt allow us to do a On Demand VPN. The bug has been reported for a while, https://github.com/lionheart/openradar-mirror/issues/6082. If you try to do this, you can expect A LOT of “An unexpected error has occurred” from the VPN client. Even looking at the Wireshark traffic didn’t lend any help on tuning Sophos to give the IKEv2 client something it would accept. If someone figures out how to get that to work in this setup please let me now.

Now that everything is setup you can host things yourself. I give the auto connecting VPN less rights than when I do a full tunnel on my laptop, but it allows for things like Jira to be hosted, then mobile clients to easily connect.

Template

For your cert to work in the template it needs to be converted. Sophos will give you a .p12 file for your cert, use the following command to get the version that needs to be in the .mobileconfig file. You’ll at minimum want to edit the cert area and put yours in there, set the password for the cert, and any URLs you need.

openssl enc -a -in user.p12 -out user.enc

Transferring Files To The Macintosh SE over Serial/Kermit

The Mac journey continues with me searching for a way to transfer files from my modern PC/Mac onto the old Macintosh SE I recently was restoring; a way without constantly removing the SD card from the SCSI2SD adapter and mounting it in an emulator. After reading a lot of different pages, and hitting different dead ends, or methods that involved a lot of hardware, time, or monetary investment I found an old reliable way to transfer files.

One of the methods I looked at was an ethernet LAN adapter for the Mac SE; the issue I saw was some of them were expensive and a lot of them required more RAM than the 1MB my SE had. I then turned to the serial ports available in the back of the machine. The Mac does not come with a lot of software to help in this endeavor, which made me use the SCSI2SD adapter to load the initial setup on, then I could use the software to transfer after that.

I ended up using the Kermit protocol, the same protocol used to transfer software to the Compaq Portable II. The project was run by Colombia University for many years. While they have since transferred it to be an open source project, the original project files are still on their FTP server, and this offers everything from DOS to Mac to C64 binaries. ftp://columbia.edu/kermit hosts all the files, for archival purposes I also uploaded a clone of that folder to archive.org; https://archive.org/details/kermit_202008 . Kermit is not fast, being serial and the Mac can’t support anything over 57600 baud; but it offers compatibility with almost every OS at this point. Get ready to experience what dialup was like all over again.

Required Hardware:

  • Serial Adapter for the modern computer if your system doesnt have one on it
  • RS-232 to mini DIN 8 cable (I used this one)

To start the connection, I will be using a modern Mac as the server (a modern Mac being a 2012 Macbook Air), and a USB Serial cable to connect to the Mac SE as client. Using homebrew on the Mac, you can install “c-kermit”. Once that is installed search for your serial device under /dev/, mine is /dev/tty.usbserial1420. Please note wherever you start kermit, will be the home folder for file transfers, I suggest making a folder somewhere that you will drop files to transfer.

Server

$ kermit

> set port /dev/tty.usbserial1420

> set carrier-watch off   # Assume there is no carrier signal

> set speed 57600          # Or whatever the speed has to be

> connect

Get ftp://columbia.edu/kermit/mac/mackermit.hqx and get it onto your Mac SE, through some means. I transferred the whole “mac” folder from Colombia’s FTP server onto my Mac SE. I would suggest a SCSI2SD adapter for this initial transfer. You may be able to use a floppy, but you may hit issues depending on your model of SE. Mine has a 800kb floppy drive, so results of writing floppies from a modern PC usually end with it not reading them. Modern floppy drives are cheap working at 1.44mb, and the tracks wont align. Once you have the Kermit app on the Mac open it up.

Select “Settings”, at the top, then “Communications”. Here you can set the speed to the max speed supported of 57600 over the default 9600 baud. Both of these are terribly slow… but there is nothing we can do about that. Make sure to select the Phone or Serial port based on which you are using; I used the Phone port.

Sorry for the odd quality, capturing a CRT isn’t the easiest

Afterwards, click the “File-Transfer” menu at the top, then “Set Directory” to set where the files transferred should end up. Then open the same “File-Transfer” menu again and “Get file from server”; here you can type in a filename that exists in the folder you opened Kermit on the Server.

Taking photos of CRTs is not the easiest…

Now be prepared to wait for a while… Eventually the files will be in the folder you selected and you are good to go!

A few things to look out for, if you have a older Mac SE like the one here and it only has 1MB of RAM, that means you can only run Mac OS 6. (https://www.lowendmac.com/oldmac/compact3.html) I may upgrade this system in the future to its max which I believe is 4MB, but for now I am stuck with 6. This also means I can only use DiskCopy 4.2, and some good amount of classic apps will not work on Mac OS 6. The biggest issue is there are a lot of archives that are in DiskCopy 6 format, which I can’t load on the system.

The first thing I thought I would do is extract the archive on an old Mac VM on my modern computer, then transfer the files onto the Mac Se. Here I ran into a lot of issues with the file types that exist. If you want to go down a weird rabbit hole, the classic Macs used an odd 4 letter system for the file type, and 4 letter for which program created it, http://livecode.byu.edu/helps/file-creatorcodes.php . The Mac mostly ignores file extensions. There are programs such as ResEdit (that comes on the provided SCSI2SD disk image I used in restoration) where you can edit these attributes, but it usually leads to weird outcomes. Kermit tends to bring files over as “text”. StuffIt seems to do a decent job of just looking at the file extension and allowing you to expand it, then those files are the correct type. This whole issue is something to look out for, doubly so on a System 6 machine and can not run DiskCopy 6.

Otherwise stick to websites that say they backup with DiskCopy 4, or get more RAM… Then have fun with the system! Write that novel you have always wanted to write without distraction.

Mac SE Restoration

Years ago someone gave me a Macintosh SE, 20MB SCSI HDD, with 1MB of RAM. I had it sitting in storage and decided it could use some new life; this involved what I found out to be repairing, upgrading, and getting parts for the little machine. Then I was able to come up with a modern way to transfer files to it, so I can get software off the web, then get it onto the system without too much hassle, but that is getting ahead of myself.

Cleaning/Repairs

Last time I used the machine I remember it working, but then when I went to turn it on the system gave a sad mac with an error. In looking it up, http://www.midiguy.com/MGuy/MacQs/SadMac.html#anchorSE&II, I was told it was a RAM error. Power cycling the machine would periodically change the error, and once in a while get the machine to power up.

Getting the case off needs a special long screw driver, which I happen to have. The back only has 4 screws and then lifts off. Any repairs to these systems have to be done with a lot of care since there is a high voltage CRT. Very carefully I removed the cables from the motherboard, and then removed the motherboard itself.

After I removed the case and looked at the RAM, it was fairly oxidized. I happened to have a can of deoxite, I removed and cleaned all the ram and then the DIMMs. After, what I will say was jankily setting up the motherboard, it booted the first time. I did notice one of the legs on the the little slots didnt look at good as the rest, but it seems to work fine now.

Luckily for me the 20MB, 3.5″ SCSI drive still works fine. I ran diagnostics on it and them came back clean. I wanted to be able to download files from a more modern system by I will do a different post about that.

There were 2 more upgrades I wanted for this machine; first the original 1987 PRAM battery was still on the board. Fortunately it had not leaked at all, but I still want to remove it. I purchased a new 1/2AA, 3.6V battery holder and thought I could use the expansion slot in the back to hold it. I am not using the slot, and that way when the system goes into storage I can pop the battery out easily. I had recently gotten a new 3D printer (Ender v3 Pro), and made a mounting bracket. It needed to be mounted on the inside of the bracket because of the high of the battery holder, but it works well!

The last upgrade I wanted was some sort of mass storage. (Mass storage being anything over a floppy with a few MB) I do have a second Macintosh, I think its a Classic but I need to go get it. Someone gave me a Zip 100MB external SCSI drive, but to get that working you need at least Mac OS 6, with the driver installed. The Mac also only has a 800KB floppy drive, making it hard to transfer files to. I have a USB floppy drive, but these newer USB drives are fairly locked to 1.44MB floppies, as well as I couldnt easily read the file format for it.

Enter the SCSI2SD (v5.5 Pocket Edition)! I got it on ebay from https://www.ebay.com/itm/SCSI2SD-V5-5-Pocket-Edition/193496539667, I don’t know the seller, but the item is great. It allows you to write a disk image you make with Mini vMac or Basilisk II onto a micro SD card, then boot the Macintosh from it! Boom solid state drive for your Macintosh. This also allows you to kickstart the process of getting an OS and software you need to hook the Macintosh up to something more modern. There are different models of these SCSI2SD adapters and different versions. Apparently v6 is faster for some systems version of SCSI. My main feature I wanted was a DB-25 connector directly on it, since a lot of these adapters come with an internal header, and I wanted this to be able to go between Macs.

In researching I found this blog, https://www.savagetaylor.com/2018/01/05/setting-up-your-vintage-classic-68k-macintosh-using-a-scsi2sd-adapter/ it has a great guide on how to setup the device and even images to get you going! (I backed up a lot of the files related to the adapter on archive.org if years later anyone needs them) I’ll skip over that since that blog covers it so well. The device allows multiple SCSI device emulation. Note, if you have a Macintosh like mine that has an internal HDD, that is SCSI ID 0, so make your device 1 or later. When booting the Macintosh you can hold Command-Option-Shift-Delete-# to boot to that device. With this setup I was able to transfer an OS install onto the ZIP disk (at 100MB plenty of space), and update the internal system.

I installed Mac OS 6.0.8, later editions need more than 1MB of RAM. For anyone with a similar system I would suggest running in Finder mode, and not Multifinder. Multifinder kept running low on RAM when trying to run applications. At this point the system is up and running, reliably, and I was able to put some games on it. I will have another article about using our old reliable Kermit to transfer files to the Mac!

Email Alerts on Different Platforms

Different network gear I have has had many problems trying to get email alerts working. I thought I would document them. All of these systems use a service gmail address I made on free/public gmail to send alerts to me.

Sophos, and LibreNMS gave me no problems; if you have issues with them drop a comment below and I can post my settings.

Ruckus AP

The trick to getting Ruckus Unleashed, I used “smtp.gmail.com” and port 587. The issue I ran into is the service email I use to send emails had a long password. Ruckus Unleashed v200.8 supports a maximum of 32 character passwords. I would also mention it dumps the password raw into the logs, so make an account you dont care much about.

Unifi Controller

After digging through logs and getting lots of “There was an error sending the test email to x@gmail.com. Failed to send email for unknown reasons.”, I found one post that mentioned a fix for the console log of “fail to send email: api.err.SmtpSendFailed”. You need to once again use smtp.gmail.com, and port 587, but since its TLS, you need to counter intuitively UNCHECK “Enable SSL”.

Altair 8800 Kit

This post will be a bit more brief than some of the others, I was relaxing around Thanksgiving and put this together. Only afterwards did I realized that I was having such a good time, that hadn’t taken too many photos.

The kit comes from Chris over at https://www.adwaterandstir.com/altair/. The version I have is The Altair-Duino v1.4, which came in a bamboo box. There are now other versions, some with acrylic cases! This post will be about version 1.4.

The Kit

The kit comes with all the parts you need inside the box. The main controller is an Arduino, hence the name The Altair-Duino. There is an SD card that you bend the prongs on (more on that later) which holds the disk images. This is a fun straight forward kit, that comes with everything you need minus solder. The Arduino came with the firmware it needed, and the SD card came with disk images preloaded onto it.

Assembly

The kit comes with a spiral notebook of instructions on how to put it together. These are great, color photos of step by step what to do. You can see them here, https://www.adwaterandstir.com/instructions-14/ , keep in mind this is for my specific version. Like many of the other kits, the longest part of this kit is soldering all the LEDs and resistors onto the board. There are a few ribbon cables that go into place, and you are set. Be slightly careful when putting the switches in, they can be a tighter fit into the holes which is great for stability, but they are at the center of the board and it can flex. Once you get it all in the case and screwed down, clearance is a bit low, so make sure the board is ready to go in, when you put it in.

The one part of the setup that is a bit scary, the system comes with a SD card reader that sits flush with the board; if you want it to be accessible from the back of the case you need to bend the 4 legs on it. I used my trusty Radioshack wire stripper/pliers for that!

Software

I connected over USB, the kit also supports Bluetooth on Windows, to get the serial line out and console in. The system supports loading a bunch of programs that are included. The creators website, https://www.adwaterandstir.com/operation/ includes a bunch of guides on things to do. I loaded up CP/M and for fun, of course Zork!

A easy kit to put together, and a fun little project. I now am amassing a wall of these projects, and will have to get a new shelf for this one. Then I will just wonder where Chris found 256mb micro SD cards!

Windows Server DNSSEC Error 9110

TL;DR; Check that your Domain Controllers are in the correct OU and that Microsoft Key Distribution Service is running

I ran into an issue recently when DNSSEC signing a dns zone where Windows Server 2019 gave a very vague error, and would only display that error after 10 minutes of timeout. This made iterating on it very slow since every change I made was a 10 minute wait. Every guide to setup DNSSEC mentioned right clicking the zone, then clicking sign and as long as you select the default it should just work. On another domain, that happened for me and it just worked; except the one original one that kept timing out.

In setting a custom DNSSEC signing policy I noticed that there were different keystores each of which gave a different error. This made me think it was something to do with the specific one I was using. It was time to troubleshoot the service itself not DNSSEC.

I got a list of the services from a known good, and signing, domain controller; then compared that to the bad one to see what was different. Part way down the list I noticed that Microsoft Key Distribution Service was failing to start, and if I tried to start it, there was an error.

Group Key Distribution Service cannot connect to the domain controller on local host Status 0x80070020.

Checking the Event Log showed an issue in finding the Domain Controllers on the network (error above), which was weird because it is a Domain Controller… In looking at where this system was placed in the domain tree, I saw it had been moved from the original OU for domain controllers to another place. I dragged it back, after applying all the GPOs that were on that other folder to the original Domain Controller folder. Then held my breath, hit start on the Key Distribution Service and it started right away.

After that DNSSEC signed with no issues. Long story short, dont move your DCs it’ll only end in pain. And to the one other person on the internet who has seen this problem and never solved it, 5+ years ago https://www.reddit.com/r/sysadmin/comments/3dedwm/dnssec_will_not_sign/ there is your answer!