Software

Writing a Cybiko Emulator

Background

I have always loved devices. Growing up, my parents would have a Blackberry or before that Palm Pilot, and I thought they were amazing. Seeing my interest, my parents got me a Cybiko Xtreme. A PDA designed for teens and college kids. With games, books, apps, and even wireless communications, all of this in 2001! The device and company have an interesting history:

To quote Rameriez in 2002 (https://everything2.com/title/Cybiko):

With Yang’s surname, the made in Taiwan sticker on the back, and Japanese name (Cybiko means Cyber-Girl in Japanese), you’d be forgiven for mistaking this as an asian product. But, the Cybiko was designed, programmed and tested in Russia.

“They don’t exactly hide the fact that Cybiko’s Russian, but let’s say, they don’t highlight it either.”

(Vaidyanathan Sivakumar, Fund Manager)

There were three iterations of the Cybiko. V1, V2, and Xtreme. The V1 and V2 look very similar. V2 saved some money by cutting the RAM in half. The Xtreme had a whole different look and upgraded the memory and CPU. All of them had a radio operating at ~900 MHz. Some features were really ahead of their time with wireless chat and being able to have one device act as a base station giving other devices E-Mail, or WAP web browsing. You could download E-Mail over serial for V1/V2 and mini-USB for Xtreme to later manually sync back. All of this for around $100 in 2001 or $185 today. There was also an MP3 player add-on for the devices, allowing earlier MP3 usage.

I had an Xtreme with the MP3 player add-on. Being able to play music, games, and go on primitive WAP web pages in 2001 was awesome. There was also a client for AOL Instant Messenger. I could sit in bed and chat with friends from school. In the era of the giant family desktops in the living room, this was a big upgrade!

Goal

Having one of the devices from my childhood and wanting to play around with it more, I found that MAME has an emulator for it created by Tim Schuerewegen around 2001-2007. He did a fantastic job keeping the device alive for people. He’s also active in the Cybiko Discord and wrote a newer web version that I encourage anyone wanting to play around with it to try!

I have been playing with the ESP32, and then found the LilyGo T-Deck, I wanted to see if I could port an emulator onto the device and how close to a full Cybiko it could be. That started me down a path of researching the MAME code and seeing if I could extract enough out for a functioning emulator.

Planning

MAME is an amazing system with many layers to its backend. It has many different CPUs, screens, chipsets emulated, and you can create a new device connecting these different pieces. The issue is that all of that is overhead and on our tiny ESP32, I knew I would need to strip it down as far as I could, and at that point I was basically making a new emulator. I’m also not the best at C, and figured if I was going to make a new emulator I would want to start in Java, then if/when there were issues, I could much more easily diagnose them. There is the downside that Java is slower than some other languages. The Cybiko Xtreme ran at 18 MHz and with its slow black and white screen I figured it would be ok for now. I ended up adding some code where the console outputs how close to real time the frame was delivered, and on my old laptop, frames were ready well before they were needed.

And that is where this project stalled for a while. Creating a new emulator is a giant task, and I didn’t have the time to spend, which I guessed would be weeks, if not months. Enter Claude Code… I figured I would give it a try, and with MAME as its reference, see if we could get the emulator going.

I kept research notes here: https://github.com/daberkow/cybiko-java-emulator/tree/main/docs/research on many parts of the Cybiko system.

Development

This was my first example of how quickly Claude Code works; the original emulator took 10 hours to get Cybiko Xtreme working… then I had too much faith and didn’t save and Claude broke the emulator trying to add a feature and we had 7 hours of fixing (too much trust). That’s when I learned to frequently commit changes to git as things started working. After getting back up and running (and committing code) it took 2 hours to add original Cybiko V1 support. I tried to add V2 but it handled the OS image differently and something was weird. After 12 hours of trying Claude gave up. We later went back and gave it another go and got V2 working. I created a roadmap of features to add sound, try to get communications working, real hardware connection via SDR!

Some features gave more difficulty than others, getting the real time clock right took a while. It kept reporting the date incorrectly, but luckily that is the first thing the firmware asks you and this made it quick to iterate on.

After having a basic emulator working, I turned my sights on seeing if I could get two emulators to communicate. MAME and the web emulator didn’t support this, so it would take some additional research, but a big part of this device’s charm and soul was its ability to communicate between devices.

Again, leaning on Opus 4.6 here, Claude wrote its own disassembler to look at the ROM and how it communicated in the real device to a co-processor and then radio. My idea was to use a multicast network to simulate the radio channels, this treats your local Wi-Fi as a shared medium. This took a lot of trying. The beacons kept not being right and the devices wouldn’t show themselves to each other, but after much trying, success! The original devices have a co-processor that handles the wireless communications. The main CPU hands off comms to this other chip, which we are trying to emulate. Also wireless is half-duplex, and we are emulating that on a much faster and full duplex medium.

With the emulator working, I thought it would be worth putting together a manager application for all the apps I got off the Cybiko Archive. The manager allows you to load them into different NVRAM images that the emulator can then use. For the Cybiko Manager initial version I spent 10 minutes working on a mock UI and creating a design doc. Then in 24 minutes Claude Code built the whole thing… We have been iterating on the manager ever since. Some recent features include the ability for the manager to decode the icons for all the apps and display them.

In getting the code working, daberkow/cybiko-java-emulator: Cybiko Emulator and Manager written in Java, I then had Claude do a port to C. If I wanted to run the emulator on embedded devices, then I would need a C version, daberkow/cybiko-c-emulator. For that one I just targeted the Cybiko Xtreme, ditching the if statements for different versions of devices.

For anyone interested, I had Claude create new research documents as we went about, covering different parts of the emulator: https://github.com/daberkow/cybiko-java-emulator/blob/main/docs/research/wireless-protocol.md, https://github.com/daberkow/cybiko-java-emulator/blob/main/docs/research/rf2915-research.md cover different parts of the radio system for example.

Future Plans

I have a long list of things I would like to eventually get working here. There are a few paths. First, real device interactions. Both via a software-defined radio to an emulator: SDR for having a modern PC talk to a Cybiko. But also using serial comms to hopefully get the manager able to push apps to devices. That way a modern computer can hook to a Cybiko and copy over games and books, maybe even act as an internet relay! The original apps were designed for Windows 9x and are not easy to get working. One difficulty is that radios are half-duplex, at any time you are either listening OR broadcasting. And with software-defined radios, they are not always the quickest to switch back and forth. I may need two, one for transmit and one for receive, which the connection code will need to handle.

I have been working on some documents I may put up as another article about the real Cybiko V1 and its serial connection. It uses a proprietary thin serial cable. A lot of these have been lost to time, and now just the serial cable can go for $50 on eBay. I hope to find a way to recreate them easily. You can get a V1 Cybiko off eBay for $20-$30, but then you have no cable and way to send data to it.

Another path could be resurrecting some of the original tools and making it easier to program for the emulator or a real device.

Last, my one-day moonshot is to create a little Linux device that is close to the original hardware’s size, running the emulator and an SDR to be able to simulate a real device.

This article is already long; I will post another article talking about my journey to get the emulator on the ESP32 / LilyGo T-Deck. I did get it to work, daberkow/cybiko-lilygo, but the ESP32-S3 with its limited internal RAM only managed about 10 FPS, which made it not very usable.

Right now, we have a working Java emulator that covers a good amount of the hardware and can communicate between emulators. There’s also a port of that emulator for just the Xtreme to C, and then a specific build for the LilyGo T-Deck. The Java version is the most documented, please give it a go and let me know how it works for you!

Appendix

Example of Claude Code working through a problem

● Good, the RAM dump is captured. Now let me also capture the I2C trace output. Let me run again briefly and focus on the I2C logs.
● Bash(java -cp build/classes/java/main org.example.cybiko.CybikoXtreme src/main/resources/cybikoxt/cyrom150.bin src/main/resources/cybikoxt/cyos_v1508.bin --headless…)
  ⎿  [I2C-TRACE] PC=0x4ADE8A DR write 0x0A → SCL=HIGH
     [I2C-TRACE] PC=0x4A456E DDR write 0xCB → SDA=LOW
     [I2C] START
     … +56 lines (ctrl+o to expand)
  ⎿  (timeout 30s)
● Excellent! Now I can see the full I2C sequence. Let me analyze what's happening:
  1. CyOS reads all 16 RTC registers (pos 0-15) - gets correct time: sec=0x35, min=0x14, hour=0x01, day=0x99, month=0x02
  2. CyOS writes ALL 16 registers back with defaults: control=0x84 (stop), time=Jan 1 00:00:00, alarm regs with some values
  3. CyOS writes control=0x04 (counting enabled, mode bit set)
  4. CyOS periodically reads registers 1-7 getting the Jan 1 defaults back, with seconds advancing
  So CyOS intentionally resets the RTC to Jan 1 every boot. Now let me disassemble the actual CyOS code to understand why. The I2C code is around PC=0x4A4500. Let me extract and disassemble it from the RAM dump.
● Read 1 file (ctrl+o to expand)
● Now let me write a simple H8S disassembler and analyze the I2C code region. The I2C bit-bang code is at PC 0x4A4500-0x4A4700. In the RAM dump, external RAM starts at 0x400000, so that's offset 0xA4500.
● Task(H8S disassembler for CyOS I2C)
  ⎿  Backgrounded agent (shift+↑ to manage · ctrl+o to expand)
● While the disassembler is being built, let me examine the raw bytes around the I2C code directly. The I2C bit-bang routines are at PC 0x4A4500-0x4A4700, which is RAM offset 0xA4500.
● Bash(xxd -s 0xA4500 -l 0x300 /tmp/cyos_ram.bin | head -60)
  ⎿  000a4500: 1b00 1770 1a91 8918 5e4b 4838 6e4a 0003  ...p....^KH8nJ..
     000a4510: f200 1772 0aa0 1a91 893c 5e4b 4838 6e4a  ...r.....<^KH8nJ
     000a4520: 0004 f200 1772 0aa0 1a91 893c 5e4b 4838  .....r.....<^KH8
     … +35 lines (ctrl+o to expand)
Bash(javac -cp build/classes/java/main /tmp/V2Debug30.java -d /tmp && java -cp build/classes/java/main:/tmp V2Debug30 2>/dev/null)
  ⎿  Loaded boot ROM: src/main/resources/cybikoxt/cyrom117.bin (32768 bytes)
     Loaded flash ROM: src/main/resources/cybikoxt/cyos_v1358.bin (262144 bytes)
     Loaded SPI dataflash: src/main/resources/cybikoxt/flash_v1358.bin (540672 bytes)
     … +58 lines (ctrl+o to expand)
● VRAM never changes after boot ROM - CyOS never writes to the LCD. Serial output stops at "Checking cartridge..." and the string "Initializing extern file devices" exists in RAM but never gets printed. Let me find what
  happens after the cartridge check.
● Searched for 1 pattern (ctrl+o to expand)
● Good - I can see the init sequence in the ROM. After "Checking cartridge..." (at offset 0x403c) the next messages should be "Initializing standard file devices" (0x417f), then "Initializing .tors..." (0x41a4), then
  "Initializing extern file devices" (0x41e3). None get printed. Let me find the code that references the "Checking cartridge" string.
● Read 3 files (ctrl+o to expand)
● The strings aren't loaded via simple MOV.L #imm. Let me trace the actual function call chain during the critical transition from scan completion to idle state.
 Critical discovery! The reschedule() function at 0x1068E4 CALLS 0x106ABC with the current task. And prepare_task() at 0x106874 copies base_priority (offset +85) to current_priority (offset +84) and inserts the task into the
   ready_queue at 0x200230. The context switch compares ready task vs current task priorities.

  The key question: what's the base_priority of the timer entry tasks? And does the context switch actually happen? Let me create a targeted debug.

ONB-Classic

Repo: https://github.com/palantir/onb-classic Docs: https://palantir.github.io/onb-classic/

Background

I have been on a slow mission to open source some of the projects I have worked on over the years. The next project after the IsoFileReader actually predates that other project. ONB-Classic has the ability to serve files directly from an ISO without extracting the ISO, to do this we utilize IsoFileReader. That feature required me to open source IsoFileReader first. ONB-Classic is a fork of the OpenNetBoot project I started around 12 years ago. The project originated with having to PXE (Preboot Execution Environment) boot systems via the network, and I wanted to be able to use Proxy-DHCP (more on that in a second) to boot anywhere no matter the infrastructure. This allowed any sysadmin to pop up a PXE server, quickly image systems, and then tear it down.

When I started this project we were imaging Windows 7 machines with Symantec Ghost (what a time to be alive). It wasn’t new at that time, but it had a built in utility from 3Com which allowed us to do Proxy-DHCP, and PXE boot systems. Proxy-DHCP is when you have an authoritative DHCP server giving out addresses on a layer 2 segment, but then have a secondary DHCP server which jumps in after the Server -> Client offer broadcast and offers up boot information (PXE data) which the client then combines to boot. This has its own RFC as part of DHCP, and works on all (with some difficulty) PXE roms.

Concept

The main concept of OpenNetBoot was to bring PXE, TFTP, and HTTP services together into a simple app, allowing someone to stand up an imaging system or Linux installation system anywhere. I also always hoped to open source it, hence the name. The program started as what you see in the repo. It became classic when I had the idea to make it into a web app, and that web app became more of a platform with plugins that could allow different types of system installation. That greater platform was more work specific and I may write about it later, but was very mission focused for building servers at work. It hooked into different systems like server ordering and customer delivery.

This project was where I really started loving lower level programming, and the things you could do if you owned the whole stack. This allows you to have greater insight into the boot process, in addition to doing some fun tricks with the protocols because we control them. I can send the client to different images depending on data I get at different stages in the pipeline, watching the client progress through the boot stages.

Development / Boot Flow

There were bumps along the way. The system would be able to boot one system, and then not work on the next. Some of this came down to vendors **cough** Realtek **cough**, not following the RFC and requiring extra bytes where there are not supposed to be any. We later moved from shipping all BIOS systems to UEFI, which proved to be a new generation of PXE roms that were more picky. The project was also written in Java, this allowed me to run it on any operating system, but this also led to issues where different systems would treat sending to a broadcast address (255.255.255.255) differently.

At this point it may be worth going through the boot flow, and how I always used ONB. A server boots, asks for DHCP, your local network gives you an IP and ONB comes in and gives boot information based on the headers in your original DHCP request. Are you x86? ARM? Are you a BIOS system, or UEFI? Then we return the address to a boot server (ourselves) and a file to load, usually iPXE.

I have been using iPXE for the whole life of this project, it’s a great boot rom except it never was SecureBoot signed, forcing us to disable SecureBoot for PXE operations. That is until recently! iPXE project after a decade got their rom signed by Microsoft! I have been very excited.

Now that iPXE’s address is given, the client reaches out over TFTP to the server to pull the rom. TFTP is very slow; the client requests bytes of a file over UDP, we send bytes, the client acknowledges and requests next bytes. There is no windowing, and if the link is full because let’s say 100 servers are booting, some UDP packets are dropped, forcing a restart. That makes our goal to leave TFTP as quickly as we can. Once iPXE is loaded, it does a new DHCP request, and gets Proxy DHCP information again, but this time with an iPXE system id. Now we serve them a menu file instead of a boot rom. From now on, we can send them data in HTTP format, which is much, much faster and unlocks things like loading large kernel roms before the heat death of the universe. Loading 100MB Linux kernels at 500kb/s is not feasible for a production environment.

As mentioned, the application is written in Java, this allows it to load anywhere, and the JavaFX UI to work on any of them. Over the years things have changed; JavaFX used to be included in Oracle Java, and as we all moved to open source Java it became its own package. The application back end became heavily multi-threaded with threads dispatched when clients connect. Multi-threading dispatch allows the DHCP and TFTP servers to handle 100+ clients at a time. When a client reaches out, we get their request, and pass it to a new worker thread to respond. Then we can immediately free up the original server process to handle the next client. There are several core threads and the goal is always to get work off them as quickly as possible and hand it to a sub-worker.

I have gone on to update this application, and write other ones using JavaFX. It’s another one of those – devil you know – situations, where I do not love programming in it, but I know how it works. The SceneBuilder allows you to create the XML GUI templates fairly easily. One of the more complicated parts of the application is actually the logging system. It has to be able to pass log messages to the GUI, or the CLI; and then pass some of them to a text log file. This system also has to take logs from different threads as they fire, and try not to block. It naturally grew over time, and has shown to work well.

While I was deep in the protocols, I went off on a weird tangent: I added a ‘virtual NIC’ option to the command line of the app. It allows you to simulate a client on the network. It generates a new MAC address and reaches out to see how the authoritative DHCP server responds. That was fun because it was the first time I acted as the PXE client instead of server, simulating a full network card.

That is all ONB-Classic does. Brings those different parts together to help iPXE get through the process. You load your own menu and images to boot whatever systems you have. The application supports running as a console app, a daemon, or a full GUI app with a tray icon. It works on Windows, Mac, and Linux; over the years has been in production on all three.

UI Design

The app went through several designs and mock-ups. Not altering too much until it became a web platform.

Original Design Mac
Original Design Windows
Original Mock Up
Updated UI Mock Up

I also tried making different logos. This is before generative AI, I had to sit there in Gimp or Inkscape myself and draw ideas. Here are a few for fun.

Final Logo
Attempt 1
Attempt 2
ONB-Classic Settings Page
ONB-Classic Final App Settings page and look

Wrapping Up

I have maintained this app for over a decade now. It ran the heart of our server shipments for years, shipping thousands of servers. It helped launch my career. And gave me a love of lower level programming down to RFC. I am excited to share it with the world, and I hope it helps a sysadmin out there to boot systems. The system is Apache 2.0 licensed, and I am always happy to get pull requests or feedback!

Hamster holding a CD with Java in the background

IsoFileReader

A while ago I was working on a system to handle network boot operations. The main server is written in Java, and I needed to be able to read contents out of ISO files. The easy solution would have been to extract the ISO to the hard drive. I was trying to avoid that to save space; and with all the different images, not have thousands of tiny files littering the drive.

Then I found stephenc repo (java-iso-tools) for reading ISO files in Java. This library worked great! It had examples which helped me get started, and was fast to dive though a file. It supported the traditional ISO-9660 formatted files, which I needed, and I was good to go. Years later, the people over at CentOS and Redhat Linux had the idea to start putting giant SHA hashes as file names. Suddenly the disc images I was getting contained filenames that were 128 characters in length; and sadly java-iso-tools was failing to parse these names. To explain why, we need a bit of a dive into how the ISO-9660 standard works.

ISO-9660 is Developed by ECMA (European Computer Manufacturers Association) as ECMA-119, and then was adopted into ISO-9660. Thus, technically I was able to get the standards documents and investigate how ECMA-119 worked. Images start with a header; pointing to several tables, and the root folder file. The information about files on the disc span out from that root file. The root file, is the root directory on the image. From there every file is either a directory (with/without children) or a file which can be read.

The standard has had many changes to it over the years. While the original ECMA-119/ISO-9660 standard dates back to the start of the CD-ROM, over time people added to the standard. With PC’s at the time running MS-DOS and being able to save files to a FAT file system as 8 letter then 3 letter for extensions, the formatted needed added onto so one day CentOS could have 128 character file names. Some early additions to the format were Rock Ridge, and the Enhanced tables. When reading the first bytes from an image, there are several byte blocks which state which version of the standard they work with; this was forward thinking in this way. The basic tables help simple devices easily be able to read the discs. They can offer short file names, and point to the same binary data other tables later do. Then the enhanced tables can offer more information, and be able to add additional features to the disc. Some of these features can include things like file permissions.

At this point I had decided I needed to fix the problem and was going to write my own library to do it. While it sounds crazy, I enjoy writing these low level libraries. I started with the ECMA-119 standard, and going through the flow, like I was a CD-ROM device reading an image. I would later add on code for Rock Ridge, and reading all the enhanced tables, and even adding on a UDF parser.

I don’t want to spend too much time going through the standard. If you are interested: ECMA-119/ISO 9660 Standard, ECMA-167/ISO_IEC 13346/Original UDF Standard, Rock Ridge, UDF 2.60, there is a collection of the standard documents in depth. This post is more to talk about the project in general, and how I enjoyed working on it. A few constraints I set upon myself were I wanted it to be 100% in Java 8. That way it could be natively compiled if someone wanted to do that, wouldn’t just be connecting to some native binary tool, and would work with older Java code bases. The project currently targets Java 11 being the LTS out at the time I was working on it. I know there are many code bases out there which are Java 8, and I actually dont think there is any code except some tests using Java 9+ features. If someone had a Java 8 project, they could remove the tests and compile to 8. We live in a little bit of an odd time now, where a project like this targets more enterprise users who tend to be back on older versions. And at the same time Java 24 is coming out. I wanted to give high level classes that a user needing a simple tool could use; but at the same time have deeper level objects publicly available.

I was using this in the earlier mentioned network booting environment.There I can be building 100+ servers at a time; speed, small, and fast code were important. I ended up adding as test some performance benchmarks. I test the old library as my control, then I do normal file lookups as well as pre-indexed. I developed a system where certain heuristics of the image are taken and can be stored. Then you can feed in this initial “vector” I called it, of the image and a file vector. If the image matched the initial vector for a few characteristics, we could reasonably assume its the same image originally scanned, then instead of reading all the header tables, we jump to the location of the file vector with trust. This does leave it up to the developer to make sure they are matching pre-indexed images with vectors; but if you do, you can much faster serve files.

This project was fairly straight forward to test, I had many and there are many ISO images out on the internet. And plenty of them are Linux Images! I also had the older library which I could use as a control to test against. I ended up writing many tests which help when people send Pull Requests to make sure nothing has broken. This project I needed done to support what I was working on. There were a few places where I didn’t fully flush out the metadata, but left it to the end user to, if they cared about that data type. I spent a lot of time in Hex Fiend hex editor marking segments and trying to understand where code I had was breaking down.

Over the years of working in Open Source, and going to a technical college, I have seen many strong technical projects that are very impressive code, and can do a ton of interesting things. And then the developer focuses on interesting things they can make their code do, and spends no time putting documentation together. At the same time there are many project that get the job done, but aren’t anything special; these projects put a few documents together and maybe an example, and then get all the usage. The area developers hate to spend time, but can be the most valuable is documentation. That pushed me to spend a lot of time commenting the code, and writing a large README file showing how to work with the library.

I hope you will take a look at the project, maybe use it, and feel free to drop issues as they arise! I have been using the library in production for years now. It doesn’t get a ton of updates, because there hasn’t been a lot I need to add to it. When a PR or Issue arise I take care of it. And with the project being published under my work, I get a lot of automated PRs to help upgrade the library.

Take a look! https://github.com/palantir/isofilereader

Solidworks Connected 2025 Failed to Install Troubleshooting

I use Solidworks Connected Makers edition to do a lot of my 3D Modeling. I had used Fusion 360 in the past, but they kept changing the license and what file types you could export, so I moved to Solidworks. As many posts from the community say, sometimes it’s like the company is trying to make you mad. Every year or so we get the next update, that you have to do because it’s a web-connected (for no reason) platform; and somehow the install breaks and won’t work. I spent a sometime trying to find all the different bits to delete to get it to install properly and wanted to document it. Once you get Solidworks in working order, it works well. It’s getting it there that is difficult.

I had a failed install, the Solidworks site thought the application was installed, but when I clicked run, I got “failed to launch application, not found”. I uninstalled anything related to Solidworks or Dassault Systemes. Then I found and deleted the following.

Files:

  • C:\Program Files\Dassault Systemes\SOLIDWORKS 3DEXPERIENCE
  • %appdata%\SOLIDWORKS
  • %appdata%\DassaultSystemes
  • C:\ProgramData\DassaultSystemes
  • C:\ProgramData\SOLIDWORKS

Registry:

  • Go to HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Installer\UserData
    • Search for “Solidworks”, I found items like \S-1-5-18\Products\0911033B9E8C8E647ABE3D57D2083CB1\InstallProperties, where DisplayName was “Solidworks 2020”, delete anything related to Solidworks at the Products level.
  • Delete “HKEY_LOCAL_MACHINE\SOFTWARE\Dassault Systemes”
  • Delete “HKEY_LOCAL_MACHINE\SOFTWARE\SolidWorks”
  • Delete “HKEY_CURRENT_USER\SOFTWARE\Dassault Systemes”
  • Delete “HKEY_CURRENT_USER\SOFTWARE\SolidWorks”

Even in the best conditions, on a fast machine it will take a LONG time to install, 3+ hours, looking like nothing is happening, I left it overnight. The x MB/y MB installed will not always progress for a while.

Logs in: C:\Program Files\Dassault Systemes\SOLIDWORKS 3DEXPERIENCE\InstallData\log

The install seems to install 4.5 billion MSI files, and then after each runs “.NET Optimization Service”; if your install is progressing, that service should periodically jump in CPU usage. My install halted at 76% 6122MB/14942MB installed for a long time; again, I left it overnight and it managed to finish.

I hope this helps someone. If others have issues or fixes, please leave a comment; maybe together we can get this program to work. In the end, mine worked after being left overnight, and now everything is functioning well with the 2025 release.

Update:

I went to reinstall and got the following errors:

Failed to get msi version for UpgradeCode [{B54313C8-7B46-297F-3AC1-85D9EFD5ECB7}].

Technical details:
The property is unrecognized
Error code: 1608
Invalid data in HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Installer\UserData\S-1-5-18\Products\BE85C2B02A76B522062B1D99E055DD33
Action CheckInstalledMsiVersionAction from feature CODE\win_b64\SWXDesktopInsPreqWPT failed.
Action ID: SWXDesktopInsWPTInstalledAction

Failed to get msi version for UpgradeCode [{00F50064-7000-11D3-8CFE-1050048383C9}].

Technical details:
The property is unrecognized
Error code: 1608
Invalid data in HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Installer\UserData\S-1-5-18\Products\E5A00A437891E38418011307471668D7
Action CheckInstalledMsiVersionAction from feature CODE\win_b64\SWXDesktopInsPreqVBA71 failed.
Action ID: SWXDesktopInsVBA71InstalledAction

I ended up searching for those hashes in the registry, “BE85C2B02A76B522062B1D99E055DD33“, and “E5A00A437891E38418011307471668D7“. Then deleting registry keys where a sub part mentioned Dassault Systems. After that I could progress with the install. Again, this seems to happen if you had Solidworks 2020 installed at any point.

Bypassing “Press Enter for network boot service”

Recently I have been working on some Microsoft SCCM automations. One issue I ran into was when UEFI booting into SCCM I got “Press Enter for network boot service”. Some people said you cant bypass it, others said maybe make it required update. Microsoft’s forums said its part of the firmware, they couldn’t control it; we know that’s wrong, because Linux network boots don’t have this issue and the prompt comes from their efi file.

I wanted to give a quick fix for people who run into this: upgrade your Windows ADK and WinPE to Windows 11. From my readings, Windows 8 ADK didn’t do this button press requirement, and I thought, I would try the newer ADK for Windows 11 instead of Windows 10 version 2004 (or earlier). The second I installed that Windows ADK and restarted, all of a sudden that prompt was gone. Happy booting!

VMWare EAM Failing, and not Allowing Upgrades

I was attempting to upgrade my homelab which I pushed to VMWare vSphere 8.0 because of… YOLO… and after a recent 8.0.1 update I was no longer able to upgrade individual ESXi hosts. I had already updated vCenter to the latest version, now I wanted to upgrade the hosts. That is my normal course of action, vCenter, then hosts; as recommended. When I went to upgrade the hosts I was told:

"Health check fails to retrieve data about service 'vSphere ESX Agent Manager' on '3 Node And Friends'. Verify that the service 'vSphere ESX Agent Manager' is running and try again."

This had me SSH into the appliance and looking at logs. (To quickly mention EAM = “vSphere ESX Agent Manager“) Here are some of the fun errors I was getting in “/var/vmware/eam/eam.log”:

  • “Re-login to vCenter because method: currentTime of managed object: null::ServiceInstance:ServiceInstance failed due to expired client session: null”
  • “failed to authenticate extension com.vmware.vim.eam to vCenter”

Some older guides mentioned unregistering EAM and then re-registering it. This broke my install even worse, and I ended up reverting to a snapshot. (Always snapshot before upgrades…) When I reverted back to before the vCenter upgrade, I realized that EAM was actually failing before the vCenter upgrade; except now I had EAM back in my extension list both on https://vcenter/mob/?moid=ExtensionManager and in vCenter, which was missing after I followed the guide saying to un-register it.

Now that I had the plugin registered, again, I found this KB, and this persons blog very helpful. I ran the recommended commands:

mkdir /certificate

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.crt

/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.key

python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.vim.eam -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s vcenter.my.domain -u Administrator@vsphere.local

And then EAM suddenly showed happy, and the log started showing useful things:

2023-06-06T16:53:37.573Z |  INFO | vim-monitor | ExtensionSessionRenewer.java | 190 | [Retry:Login:com.vmware.vim.eam:f86509907b4cb7c6] Re-login to vCenter b
ecause method: currentTime of managed object: null::ServiceInstance:ServiceInstance failed due to expired client session: null
2023-06-06T16:53:37.573Z |  INFO | vim-monitor | OpId.java | 37 | [vim:loginExtensionByCertificate:443bbd7c03dce9c6] created from [Retry:Login:com.vmware.vim
.eam:f86509907b4cb7c6]
2023-06-06T16:53:37.947Z |  INFO | vim-async-2 | OpIdLogger.java | 35 | [vim:loginExtensionByCertificate:443bbd7c03dce9c6] Completed.

Thats it! Now I can run updates again! If anyone has the same issue, drop a line in the comments. I hope this isn’t a big new vSphere 8.0 issue. I had upgraded this appliance from 7.0, and perhaps that or a cert issue caused issues.

Below is some of my eam.log to help people:

2023-06-06T02:20:29.728Z | ERROR | vlsi | DispatcherImpl.java | 468 | Internal server error during dispatch
com.vmware.vim.binding.eam.fault.EamServiceNotInitialized: EAM is still loading from database. Please try again later.
        at com.vmware.eam.vmomi.EAMInitRequestFilter.handleBody(EAMInitRequestFilter.java:57) ~[eam-server.jar:?]
        at com.vmware.vim.vmomi.server.impl.DispatcherImpl$SingleRequestDispatcher.handleBody(DispatcherImpl.java:373) [vlsi-server.jar:?]
        at com.vmware.vim.vmomi.server.impl.DispatcherImpl$SingleRequestDispatcher.dispatch(DispatcherImpl.java:290) [vlsi-server.jar:?]
        at com.vmware.vim.vmomi.server.impl.DispatcherImpl.dispatch(DispatcherImpl.java:246) [vlsi-server.jar:?]
        at com.vmware.vim.vmomi.server.http.impl.CorrelationDispatcherTask.run(CorrelationDispatcherTask.java:58) [vlsi-server.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_362]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_362]
        at java.lang.Thread.run(Thread.java:750) [?:1.8.0_362]
2023-06-06T02:20:31.769Z |  INFO | vim-monitor | ExtensionSessionRenewer.java | 190 | [Retry:Login:com.vmware.vim.eam:9ae94019eb8cb9a2] Re-login to vCenter b
ecause method: currentTime of managed object: null::ServiceInstance:ServiceInstance failed due to expired client session: null
2023-06-06T02:20:31.769Z |  INFO | vim-monitor | OpId.java | 37 | [vim:loginExtensionByCertificate:b63ca4cf0b995a54] created from [Retry:Login:com.vmware.vim
.eam:9ae94019eb8cb9a2]
2023-06-06T02:20:34.775Z |  INFO | vim-async-2 | OpIdLogger.java | 43 | [vim:loginExtensionByCertificate:b63ca4cf0b995a54] Failed.
2023-06-06T02:20:34.775Z |  WARN | vim-async-2 | ExtensionSessionRenewer.java | 227 | [Retry:Login:com.vmware.vim.eam:9ae94019eb8cb9a2] Re-login failed, due
to:
com.vmware.eam.security.NotAuthenticated: Failed to authenticate extension com.vmware.vim.eam to vCenter.
        at com.vmware.eam.vim.security.impl.SessionManager.convertLoginException(SessionManager.java:329) ~[eam-server.jar:?]
        at com.vmware.eam.vim.security.impl.SessionManager.lambda$loginExtension$4(SessionManager.java:154) ~[eam-server.jar:?]
        at com.vmware.eam.async.remote.Completion.onError(Completion.java:86) [eam-server.jar:?]
        at com.vmware.eam.vmomi.async.FutureAdapter.setException(FutureAdapter.java:81) [eam-server.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$ClientFutureAdapter.setException(MethodInvocationHandlerImpl.java:731) [vlsi-c
lient.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$RetryingFuture.fail(MethodInvocationHandlerImpl.java:578) [vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$RetryingFuture$RetryActionImpl.proceed(MethodInvocationHandlerImpl.java:625) [
vlsi-client.jar:?]
        at com.vmware.eam.vim.security.impl.ExtensionSessionRenewer.retry(ExtensionSessionRenewer.java:149) [eam-server.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$RetryingFuture.setException(MethodInvocationHandlerImpl.java:541) [vlsi-client
.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setResponse(ResponseImpl.java:239) [vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:286) [vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.HttpExchange.invokeWithinScope(HttpExchange.java:54) [vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.TracingScopedRunnable.run(TracingScopedRunnable.java:24) [vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.run(HttpExchangeBase.java:60) [vlsi-client.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_362]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_362]
        at java.lang.Thread.run(Thread.java:750) [?:1.8.0_362]
Caused by: com.vmware.vim.binding.vim.fault.InvalidLogin: Cannot complete login due to an incorrect user name or password.
        at sun.reflect.GeneratedConstructorAccessor58.newInstance(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_362]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_362]
        at java.lang.Class.newInstance(Class.java:442) ~[?:1.8.0_362]
        at com.vmware.vim.vmomi.core.types.impl.ComplexTypeImpl.newInstance(ComplexTypeImpl.java:174) ~[vlsi-core.jar:?]
        at com.vmware.vim.vmomi.core.types.impl.DefaultDataObjectFactory.newDataObject(DefaultDataObjectFactory.java:25) ~[vlsi-core.jar:?]
        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.ComplexStackContext.<init>(ComplexStackContext.java:30) ~[vlsi-core.jar:?]
        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.parse(UnmarshallerImpl.java:167) ~[vlsi-core.jar:?]
        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.unmarshall(UnmarshallerImpl.java:105) ~[vlsi-core.jar
:?]
        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:92) ~[vlsi-core.jar:?]
        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:86) ~[vlsi-core.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.SoapFaultStackContext.setValue(SoapFaultStackContext.java:41) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.processNextElement(ResponseUnmarshaller.java:127) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.unmarshal(ResponseUnmarshaller.java:70) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.unmarshalResponse(ResponseImpl.java:284) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setResponse(ResponseImpl.java:241) ~[vlsi-client.jar:?]
        ... 7 more
2023-06-06T02:20:34.777Z | ERROR | vim-monitor | VcListener.java | 124 | An unexpected error in the changes polling loop
com.vmware.eam.EamRemoteSystemException: Unexpected error communicating with the vCenter server.
        at com.vmware.eam.vim.server.impl.VimRoot.rootOperation(VimRoot.java:106) ~[eam-server.jar:?]
        at com.vmware.eam.vim.server.impl.VimRoot.currentTime(VimRoot.java:78) ~[eam-server.jar:?]
        at com.vmware.eam.vc.VcListener.main(VcListener.java:140) ~[eam-server.jar:?]
        at com.vmware.eam.vc.VcListener.call(VcListener.java:118) [eam-server.jar:?]
        at com.vmware.eam.vc.VcListener.call(VcListener.java:58) [eam-server.jar:?]
        at com.vmware.eam.async.impl.AuditedJob.call(AuditedJob.java:58) [eam-server.jar:?]
        at com.vmware.eam.async.impl.FutureRunnable.run(FutureRunnable.java:55) [eam-server.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_362]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_362]
        at java.lang.Thread.run(Thread.java:750) [?:1.8.0_362]
Caused by: com.vmware.vim.binding.vim.fault.NotAuthenticated: The session is not authenticated.
        at sun.reflect.GeneratedConstructorAccessor57.newInstance(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_362]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_362]
        at java.lang.Class.newInstance(Class.java:442) ~[?:1.8.0_362]
        at com.vmware.vim.vmomi.core.types.impl.ComplexTypeImpl.newInstance(ComplexTypeImpl.java:174) ~[vlsi-core.jar:?]
        at com.vmware.vim.vmomi.core.types.impl.DefaultDataObjectFactory.newDataObject(DefaultDataObjectFactory.java:25) ~[vlsi-core.jar:?]
        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.ComplexStackContext.<init>(ComplexStackContext.java:30) ~[vlsi-core.jar:?]
        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.parse(UnmarshallerImpl.java:167) ~[vlsi-core.jar:?]
        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.unmarshall(UnmarshallerImpl.java:105) ~[vlsi-core.jar
:?]
        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:92) ~[vlsi-core.jar:?]
        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:86) ~[vlsi-core.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.SoapFaultStackContext.setValue(SoapFaultStackContext.java:41) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.processNextElement(ResponseUnmarshaller.java:127) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.unmarshal(ResponseUnmarshaller.java:70) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.unmarshalResponse(ResponseImpl.java:284) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setResponse(ResponseImpl.java:241) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:286) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.HttpExchange.invokeWithinScope(HttpExchange.java:54) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.TracingScopedRunnable.run(TracingScopedRunnable.java:24) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.run(HttpExchangeBase.java:60) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:229) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:114) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:693) ~[vlsi-client.jar:
?]
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:674) ~[vlsi-client.j
ar:?]
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:371) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:322) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:195) ~[vlsi-client.jar:?]
        at com.sun.proxy.$Proxy51.currentTime(Unknown Source) ~[?:?]
        at com.vmware.eam.vim.server.impl.VimRoot.rootOperation(VimRoot.java:101) ~[eam-server.jar:?]
        ... 9 more
2023-06-06T02:20:34.778Z |  INFO | vim-monitor | VcListener.java | 125 | Full stack trace: com.vmware.eam.EamRemoteSystemException: Unexpected error communic
ating with the vCenter server.
        at com.vmware.eam.vim.server.impl.VimRoot.rootOperation(VimRoot.java:106)
        at com.vmware.eam.vim.server.impl.VimRoot.currentTime(VimRoot.java:78)
        at com.vmware.eam.vc.VcListener.main(VcListener.java:140)
        at com.vmware.eam.vc.VcListener.call(VcListener.java:118)
        at com.vmware.eam.vc.VcListener.call(VcListener.java:58)
        at com.vmware.eam.async.impl.AuditedJob.call(AuditedJob.java:58)
        at com.vmware.eam.async.impl.FutureRunnable.run(FutureRunnable.java:55)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
Caused by: (vim.fault.NotAuthenticated) {
   faultCause = null,
   faultMessage = null,
   object = ManagedObjectReference: type = ServiceInstance, value = ServiceInstance, serverGuid = f0ee8343-1721-4676-9069-1a837625c60b,
   privilegeId = ,
   missingPrivileges = null
}
        at sun.reflect.GeneratedConstructorAccessor57.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.lang.Class.newInstance(Class.java:442)
        at com.vmware.vim.vmomi.core.types.impl.ComplexTypeImpl.newInstance(ComplexTypeImpl.java:174)
        at com.vmware.vim.vmomi.core.types.impl.DefaultDataObjectFactory.newDataObject(DefaultDataObjectFactory.java:25)
        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.ComplexStackContext.<init>(ComplexStackContext.java:30)
        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.parse(UnmarshallerImpl.java:167)
        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.unmarshall(UnmarshallerImpl.java:105)
        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:92)
        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:86)
        at com.vmware.vim.vmomi.client.common.impl.SoapFaultStackContext.setValue(SoapFaultStackContext.java:41)
        at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.processNextElement(ResponseUnmarshaller.java:127)
        at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.unmarshal(ResponseUnmarshaller.java:70)
        at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.unmarshalResponse(ResponseImpl.java:284)
        at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setResponse(ResponseImpl.java:241)
        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:286)
        at com.vmware.vim.vmomi.client.http.impl.HttpExchange.invokeWithinScope(HttpExchange.java:54)
        at com.vmware.vim.vmomi.client.http.impl.TracingScopedRunnable.run(TracingScopedRunnable.java:24)
        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.run(HttpExchangeBase.java:60)
        at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:229)
        at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:114)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:693)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:674)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:371)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:322)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:195)
        at com.sun.proxy.$Proxy51.currentTime(Unknown Source)
        at com.vmware.eam.vim.server.impl.VimRoot.rootOperation(VimRoot.java:101)
        ... 9 more

2023-06-06T02:20:34.778Z |  INFO | vim-monitor | VcListener.java | 131 | Retrying in 10 sec.

Moving From Dropwizard to Spring Boot

Where I work historically, we have used Dropwizard, a Java framework for creating web apps a lot. This framework went head-to-head with Spring Boot, in the last few years and has seem to drop out of favor with the community. Nonetheless, I thought for a new project I would dive into using that to get more acquainted with it. After trying to get two of the basic things I needed done, I ended up giving up on Dropwizard and pivoting to Spring Boot instead.

The first big library I tend to use with Java projects, especially web apps, is jOOQ. This framework creates simple objects and gives many ways to interact with your database. The best feature for me is you can have a Gradle plugin scan your database, then create all the objects automatically in Java. Not only does this save you from handwriting a bunch of SQL queries, but it also means when you update the database (probably using something like Flyway), your objects automatically get updated. Now when you compile your program, if you forgot to add that new field somewhere when editing an object, you get a compilation error instead of the application silently failing SQL queries in production.

Dropwizard does not natively support jOOQ, I went looking for a library to add the support I needed. I found benjamin-bader/droptools library. It seemed to do what I needed. I got it wired in, and soon everything was working! I could make objects and with one or two lines edit objects web requests. Wonderful. Then Dropwizard did a major update; version 3.0 was created to keep the javax namespace, and 4.0 was created to move to the jakarta namespace. These versions also moved a bunch of the internals of the Dropwizard libraries around, meaning supporting libraries like droptools would need updated.

That’s when I saw droptools had not been updated for 3 years… I decided I would open a GitHub issue. With not hearing anything for a few days I started tinkering with it. I got a updated build working for Dropwizard 3.0 and did a pull request back to the main repo. In doing this I realized with the Dropwizard 3.0 and 4.0 split, we would need at least 2 versions of the library created at one time. Then on top of that, jOOQ 3.16 was the last to support Java 11, and jOOQ 3.18 was out as the main community supported branch. This means we need to make 4 versions; 2 with Dropwizard 3.0, and 2 with 4.0, then each one having jOOQ 3.16 and 3.18. I rewrite the build pipeline from the Travis CI the repo had to Github actions, and got all 4 versions compiling with some regex to do the edits in the code that were needed. I then used my earlier article, to publish these 4 assets to Maven Central.

This allowed me to update to Dropwizard 4.0, and the Jakarta namespace.

Next, I need to get basic authentication working. My plan is to use Google OAuth as the login mechanism. I do not feel like writing my own for a side project, and out of the ones out there (Google, Facebook, Twitter, Github) I thought it had the most coverage of people, with the least surveillance factor. It is easy enough to get setup with a developer account and get the client-id and secret I needed for OAuth.

Now I had to wire up the OAuth on the application side, this is not too hard I have done this many times with applications at work, but usually there I am using internal libraries. Heading over to the Dropwizard docs didn’t give me exactly what I wanted. They are pretty sparce, and when it comes to setting it up, they mention how to do OAuth but then mention you need to write your own Authenticator and Authorizer for it. I don’t want to do that. I have done that before for servlet-based apps, but this is supposed to be a fun thing, and on the general internet I want a supported auth library. I went searching for an example of how to use the OAuth system. I could not find anything that got me what I wanted.

Then I remembered using Pac4J before with other Java frameworks, this is a security library that has support for many login methods, and many web frameworks. Dropwizard is listed as supported! But the last time it was touched by a human, and not a bot, was over a year ago, and that was just for a small CI fix… I’ll try to get it working anyway!

The dropwizard-pac4j library is what I need, and there is a dropwizard-pac4j-demo which walks you through setting everything up! I get the demo working, I added in Google login support, which wasn’t there by default. Then I spent a day… Where I wanted to get this auth working on Dropwizard 3.0 or 4.0. I don’t want to start working in the older 2.x framework to get stuck later. I downloaded dropwizard-pac4j and the demo locally and started editing them to get the dependencies updated and try to get everything onto the jakarta namespace.

This is where the dependency hell came in. dropwizard-pac4j-demo depends on dropwizard-pac4j, which makes sense. dropwizard-pac4j sets a lot of your project versions based on what it has in it. After updating a ton of dependencies to try to get it to compile it came down to DropwizardTestSupport.java failing to run because it relies on jax-rs-pac4j. jax-rs-pac4j is still in the javax namespace and hasn’t been touched by a human in 6 months or more. This library would need to be updated, because it links directly to the main Jetty Server project which has a dependency on jakarta.servlet.SingleThreadModel in ServletHolder.java, which has been deprecated and removed (discussion), then and I could not get the demo project to load with any combination of dependencies. They all wanted this Jetty 11 file, which should have jakarta.servlet.SingleThreadModel removed, but doesn’t.

I went back and tried to move to Dropwizard 3.0, going back to the javax namespace, but that opened up a bunch of similar issues and a ton of conflicting dependencies in different versions of code dropwizard-pac4j needed. I have my code on GitHub if anyone wanted to continue this journey, or in the future things are in a better place.

With all of that, I thought I would go and check the documentation for Spring Boot. There is a giant page, with in-depth, step by step instructions on how to get Google or GitHub auth working in your app. There is a night and day difference between the support and thoroughness of Dropwizard docs and Spring Boot. With seeing that, I had to decide to change my plans away from Dropwizard. Many on the Java subreddit will debate Spring Boot vs Quarkus; for me, who has only used servlets in the past with embedded Tomcat, I think starting with the popular Spring Boot makes the most sense.

A cup of java with audio waves behind it, from Bing image creator

PodcastFeedHandler and Java-LAME

As I have mentioned in previous posts, I am working on a side project where I work with podcast feeds. Part of the idea is being able to act as a sort of middleware for podcasts. Give the application your podcasts, and you will be able to do things like search transcripts; and, if a podcast has a long intro on each episode, remove that. Searching for where someone mentioned a thing in an episode can be hard to research by just scrolling through an episode; if computer transcripts are 95% effective, that is better than nothing. For trimming podcasts, I find when you go through a big backlog of podcasts, hearing the same 1-minute intro every 30 minutes takes a good chunk of time.

Being that I use Java a lot for work, and am comfortable there, I wanted to make this project with Dropwizard and React. The first bit of the project has been working on the audio recognition engine, which will be a whole post in and of itself. After that I needed to start getting the supporting libraries I wanted. I tend to try to make as much of my code native to the language I am using as possible. That means we want to do as much in Java itself as possible. There are a ton of libraries that call out to FFMPEG or a command line app to handle the feeds; I don’t want to do that. If a side effect of this project is helping the community and writing some additional libraries, that is a win in my book.

PodcastFeedHandler

The first library I needed was a library to read AND write podcast feeds. With this app being middleware, we need to be able to do both. I found MarkusLewis‘s Podcast-Feed-Library, this works great for reading feeds in, but does not support writing. I took a look at his library and architecture a library similarly, except added the ability to get your feed object and then write it out again. In the end I made https://github.com/daberkow/PodcastFeedHandler. This library is written entirely in Java with no dependencies. Using Java 11, I can have all the native XML parsing I need. The rest of my project is in Java 17, but I thought others may find 11 useful. I am not sure there are any fragments in the library that wouldn’t allow me to go lower, except its 2023, and 11 is an older LTS at this point. An exciting part of that sub-project was getting Maven publishing working. Now I can publish for my domain of ntbl.co.

This project also got me used to using Github Actions. I have used CircleCI before but thought I would try Github Actions as they give you unlimited runtime for public repositories. Thanks Microsoft! I have the library build, get signed, and upload via Actions. I wanted to make sure the library preformed as I wanted and reached out to JetBrains to get an Open-Source maintainer license for Intellij. They kindly approved me!

Java-LAME

The next part of the project was parsing and fingerprinting the audio to search for duplicate segments. I will get more into that at a later time. To be able to fingerprint I needed the WAV/PCM format of the audio. Podcasts tend to be MP3 or AAC files. There are a ton of libraries to convert media in Java, except most of them had a FFMPEG external dependency. That is something I wanted to avoid. By having 100 percent native code, I can more easily create the workers that will handle these duties. Anywhere Java can run, they can run on or be compiled to; instead of having external dependencies.

I found nwaldispuehl‘s java-lame, this is a copy of the fantastic native Java port of LAME; described as “This java port of LAME 3.98.4 was created by Ken Händel for his ‘jump3r – Java Unofficial MP3 EncodeR’ project: http://sourceforge.net/projects/jsidplay2/“. The library hasn’t been updated in a while but does everything I need. It can convert MP3s but needs a file location to be passed in before converting to a byte array. I do not want to have to write to disk. The workflow would be, download podcast, store on disk, read from disk, convert. We should be able to do all this in memory. Doing all these operations in memory also means the workers do not need a bunch of scratch disk space, which is a plus. It’s more memory intensive but cuts down on disk usage. In 2023, I would rather have a slightly more memory intensive application than be doing a ton of extra read/writes to SSDs.

Throughout this project I have been thinking of: if I use it a lot or have friends using the web app, and it is constantly reading and writing audio files, how can I minimize bottlenecks. I forked the Github repo for java-lame, then added in paths to allow in-memory MP3 feeding and processing. This allows me to add a S3 client to the workers, and natively work on those files without ever writing to disk.

This library has a bunch of more functionality than I am using. It was a full LAME port, including the command line system and processing. I am planning to remove that as I go to shrink the library. I also want to replace some of the core conversion to WAV/PCM into having it in-memory compression, and functions to handle chunking the files and processing them piece by piece.

I took a This American Life episode, 1 hour in length, 67MB as MP3. Converting it to WAV/PCM I needed created a 678MB file. About a 10x size difference. Compressing that data lossless-ly with standard ZIP compression got the file down to 437MB, about 65% the size the original WAV/PCM. I can retrieve the ZIP data as a stream, and being audio, I am not jumping around; thus, that works well for me. 678MB for a file doesn’t sound so bad, a worker then just needs 1GB of RAM or so to process it, right? My worry is other podcasts. Shows like Dan Carlin’s Hardcore History can easily be 5 hours per episode, that is a 200+MB MP3, and then would be 2-3 GB of RAM to process one episode. If I can take 35% off for relatively small compute overhead, I want to.

I will post more as I go through the project. If these libraries or the blog have helped anyone, feel free to drop a comment! I always appreciate it when people do.

(The photo is something I through together on Bing image creator, its Java with audio 😊 )

Publishing Java Libraries to Maven Central with GitHub Actions and Gradle (Gradle 7/8 in 2023)

Intro

I recently started a new, grander, project for my spare time. The project involves working with Podcast feeds, and I was going to use this as an opportunity to use a framework I haven’t before, Dropwizard. I found a Java library that did what I needed in MarkusLewis – Podcast Feed Library; except this library only read feeds, I want to be able to read AND write. I decided to make my own, and I wanted to host the library, allowing anyone else to use it if they want. I created the repo and got a basic version working. This is the repo which can be referenced as an example.

I am using GitHub Actions as my CI/CD pipeline. I thought I should easily be able to host the final Jar files there for Gradle. Turns out, this is sort of true… If you host your library on GitHub itself, as this doc goes over, you can easily upload and host the packages; except there is no un-authenticated access to it. No matter what, an end user has to auth with Gradle/Maven before downloading the assets. Instead of dealing with that (specially for a public repo), I thought I would give a try to getting my package into Maven Central. Once I figured out the process, and found out how to publish with up to date Gradle, it as straight forward. I thought I would document it for the greater internet, and my future self (I have already used it). I know others have done this as well, except I wanted to do it with Gradle instead of Maven, with GitHub Actions doing all the work.

Throughout this guide there are items you need to record to bring to the next step, I have underlined the important ones.

Steps:

  • Setup Repo
  • Register For Maven/Sonatype
  • Setup GPG for Repo
  • Configure GitHub
  • Publishing

Setup Repo

Setup a normal GitHub repo, and setup a blank Gradle project. More on the repo/Gradle config later.

Register for Maven/Sonatype

Sonatype is the company who runs Maven Central. They allow free hosting and registration for Java Libraries; the main requirement is for it to be under 1GB in size per file. This adventure starts over at their Jira to register for an account, this Jira account will be your credentials for all future interactions with Maven Central, so make them secure, and have a long password! Once you have an account, use the above like to go to their Jira again to create an issue. This ticket grants you permissions to publish to https://s01.oss.sonatype.org/. You can also login there with the credentials created for Jira. You will have to verify either your GitHub account, or your domain before publishing. A bot handles all this and I had it done in 20 minutes or so. I have a domain I wanted to use, and there is a guide on how to go through this process.

Once you register a group ID you can use this account to publish anything under that ID; for example, I registered my domain of ntbl.co (making the group ID in Java terms “co.ntbl”. First, I published the library above, then I added a fork of a Java-Lame library; I tried to submit a ticket for the second library to be sure, and the bot tells you that you are already good to go.

Setup GPG for the Repo

One requirement for posting assets to Maven Central is to GPG sign the packages. This means we need to generate a key, and then upload the private portion of the key to GitHub secrets, and the public to a public key repository. Below are the commands to do this, the key ID is an example one I have, you will need to replace it with yours:

gpg --gen-key
gpg --list-keys
gpg --export --export-options backup --output public.gpg  co.ntbl.podcastfeedhandler
gpg --export-secret-keys --export-options backup --output private.gpg co.ntbl.podcastfeedhandler
gpg --export-ownertrust > trust.gpg
gpg --list-secret-keys --keyid-format LONG
gpg --export-secret-keys --armor 3F6F38BA13BEBB6941F823DCEFAAE414FF016215
gpg --keyserver keys.openpgp.org --send-keys 3F6F38BA13BEBB6941F823DCEFAAE414FF016215

Line 1 creates the keys, for name you enter the full project name, for example: co.ntbl.podcastfeedhandler . Group ID + the project root name. The email can be any email you have. Then the passphrase which will be used and uploaded to Github secrets. I suggest using a password generator and making it long, you should never have to actually type this in. Next, the exports are for you to back up the key incase the system you are creating it on dies and the data is lost in the GPG instance. You shouldn’t generally need it after this is setup, but it felt like best practice.

Record the output of the 7th line (export-secret-keys), that will need to be added to the Github secrets in the next step.

Configuring GitHub

The last command publishes the public keys to a global repo which is checked against. If this publish is not done, then the verification of the package will fail.

The two items we need to upload to GitHub for GPG are the password added when the key was generated, and the private key we got from the 7th command.

Go to your GitHub repo, then go to the Settings tab. Using the left-hand navigation, go to “Secrets and variables”, and the “Actions” submenu.

We need to create 4 secrets; these need to be kept secret:

  • GPG_SIGNING_KEY – The private key, copy the text from the “–export-secret-keys” command, this formats it correctly. The string should start with “—–BEGIN PGP PRIVATE KEY BLOCK—–“
  • GPG_SIGNING_PASSPHRASE – The password added when generating the key
  • OSSRH_TOKEN – This is the password you set for Sonatypes Jira
  • OSSRH_USERNAME – The Sonatype Jira username

Below is a minimal example build.gradle for your project. I removed a lot of normal extra things you would add to a buidl.gradle, to see a full example, visit this GitHub repo.

Gradle

plugins {
    id 'java-library'
    id 'signing'
    id 'maven-publish'
}

group = 'co.ntbl'
version = '0.1.2-SNAPSHOT'
rootProject.description = 'Read and Write Podcast feeds from Java.'

sourceCompatibility = 11
targetCompatibility = 11

tasks.register('createProperties') {
    doLast {
        new File("$projectDir/src/main/resources/version.properties").withWriter { w ->
            Properties p = new Properties()
            p['version'] = project.version.toString()
            p.store w, null
        }
    }
}

classes {
    dependsOn createProperties
}

jar {
    manifest {
        attributes(
                "Class-Path": "co.ntbl.podcastfeedhandler",
                "Main-Class": "PodcastFeedHandler",
                "Implementation-Title": project.name,
                "Implementation-Version": version,
                "Implementation-Vendor": "Daniel Berkowitz",
                "Build-Jdk": org.gradle.internal.jvm.Jvm.current(),
                "Gradle-Version": GradleVersion.current().toString()
        )
    }
    duplicatesStrategy = DuplicatesStrategy.EXCLUDE
    from {
        configurations.runtimeClasspath.collect { it.isDirectory() ? it : zipTree(it) }
    }
}

java {
    withJavadocJar()
    withSourcesJar()
}

ext.admin = System.getenv("MAVEN_USERNAME")

signing {
    required { admin }
    def signingKey = System.getenv("GPG_SIGNING_KEY")
    def signingPassword = System.getenv("GPG_SIGNING_PASSPHRASE")
    useInMemoryPgpKeys(signingKey, signingPassword)
    sign publishing.publications
}

repositories {
    mavenCentral()
}

dependencies {
...
}

//
// MAVEN
//

publishing {
    publications {
        mavenJava(MavenPublication) {
            from components.java

            pom {
                name = 'PodcastFeedHandler'
                description = rootProject.description
                url = 'https://github.com/daberkow/PodcastFeedHandler'
                licenses {
                    license {
                        name = 'MIT License'
                        url = 'https://github.com/daberkow/PodcastFeedHandler/blob/main/LICENSE'
                        distribution = 'repo'
                    }
                }
                developers {
                    developer {
                        id = 'daberkow'
                        name = 'Daniel Berkowitz'
                        email = 'dansberkowitz@gmail.com'
                    }
                }
                scm {
                    connection = 'scm:git:git://github.com/daberkow/PodcastFeedHandler.git'
                    developerConnection = 'scm:git:ssh://git@github.com:daberkow/PodcastFeedHandler.git'
                    url = 'https://github.com/daberkow/PodcastFeedHandler'
                }
            }
        }
    }
    repositories {
        maven {
            name = "OSSRH"
            if (admin) {
                credentials {
                    username = System.getenv("MAVEN_USERNAME")
                    password = System.getenv("MAVEN_PASSWORD")
                }
            }
            def releasesRepoUrl = 'https://s01.oss.sonatype.org/service/local/staging/deploy/maven2/'
            def snapshotsRepoUrl = 'https://s01.oss.sonatype.org/content/repositories/snapshots/'
            url = version.endsWith('SNAPSHOT') ? snapshotsRepoUrl : releasesRepoUrl
        }
    }
}

A few things to point out. Under publishing you need to enter all the information for this repository/project. If you have another publishing section in your Gradle file you will need to condense them together. Having multiple leads to Gradle getting confused and usually using the first one it sees. You will also see some variables such as “MAVEN_USERNAME”, these get the values of our secrets during the GitHub actions publish process, which we will go over next. I am getting the version, and using the end of it containing “SNAPSHOT” to say if we should publish to a snapshot repo or prod.

I also am using the build.gradle version as the canonical version. This variable could be in a Gradle settings file, or properties, but for ease I have it in the build file. I want 1 version file location; having multiple leads to more confusion during releases. The createProperties task creates a properties file that is added to the build to give the code itself a way to see which version it is. There are more elaborate ways to do this, but it works for me. This function does need the resources folder to be in the “src/main” folder; if your project is not using this the easiest way to add an “empty folder” is add the “resources” folder and then add the following .gitignore to it. This will make sure the contents of this folder are never saved.

# Ignore everything in this directory
*
# Except this file
!.gitignore

Requirements for posting to Maven central are: including source, checksums, Javadocs, and signing your packages. I am using useInMemoryPgpKeys to sign in GitHub Actions. This is part of the signing plugin. I have seen others use sign configuration.packages instead of sign publishing.publications, I found that not to work in many trials.

GitHub Actions

In your repository, create a .github folder, then a workflows folder. Below is my publish.yml, or it is available here. This file is currently set to publish when a new release is tagged, you can also change this to commits or some other trigger.

name: Publish package to the Maven Central Repository and GitHub Packages
on:
  release:
    types: [published]
jobs:
  publish-release:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout latest code
        uses: actions/checkout@v3

      - name: Set up JDK 11
        uses: actions/setup-java@v3
        with:
          distribution: adopt
          java-version: 11
      - name: Validate Gradle wrapper
        uses: gradle/wrapper-validation-action@e6e38bacfdf1a337459f332974bb2327a31aaf4b
      - name: Publish package
        uses: gradle/gradle-build-action@67421db6bd0bf253fb4bd25b31ebb98943c375e1
        with:
          arguments: publish
        env:
          MAVEN_USERNAME: ${{ secrets.OSSRH_USERNAME }}
          MAVEN_PASSWORD: ${{ secrets.OSSRH_TOKEN }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          GPG_SIGNING_KEY: ${{ secrets.GPG_SIGNING_KEY }}
          GPG_SIGNING_PASSPHRASE: ${{ secrets.GPG_SIGNING_PASSPHRASE }}

Here we convert GitHub secrets to local environment variables. Note the change in name from OSSRH_USERNAME to MAVEN_USERNAME and OSSRH_TOKEN to MAVEN_PASSWORD. This is simply to make the variables more clear, and they can be whatever you wish. We also validate Gradle for this final build. Another note, in my setup we are not passing assets from earlier builds into this publish stage, we are rebuilding the jar completely, depending on the size of your job, this may or may not make sense. If you have all this setup correctly, you should be able to commit the code, tag a release with “0.0.1-SNAPSHOT” or any version ending in SNAPSHOT and it should publish to the snapshot repo.

Publishing

Now that we have working snapshot releases, we need to do a full release. This involves you using the credentials created with the Sonatype Jira account earlier and logging into the Nexus panel. When you are ready, go to GitHub, and mark a new release with the version not ending with SNAPSHOT. The GitHub action should finish successfully, yet your asset is not up at https://repo1.maven.org/maven2/ yet. Head over to https://s01.oss.sonatype.org/ and click “Log In” in the top right.

Select “Staged Repositories” on the left. Note: this server seems to be very busy during the day, doubly so if it is a weekday. You will frequently see “There was an error communicating with the server: request timed out”. Come back later or keep hitting refresh.

Clicking a repository will allow you to browse the contents, and make sure it looks how you want it to. When you are ready you click “Close” at the top of the pane to finalize this version. Closing the repository starts all the checks on the repository, this includes making sure GPG signatures are there, the sources, Javadoc, and checksums are there. If they are not, you will get an error and be forced to Drop the release and try again. You also will get a vulnerability scan, including dependencies, to your email on file.

After the repo successfully closed, you can click Release! This is another stage where you can get many timeouts and be forced to wait till the server is less busy. After it successfully releases, it takes about 30 minutes for it to show up in the global Maven repo.

Selecting “Repositories” at the left allows you to browser the global Snapshots and Releases repositories; I have found this screen updates quicker than other locations to see if your assets are starting to propagate, including faster than the main Maven repo.

After about 30 minutes, your release should start to show up at Maven Search, although it can take longer. Another popular place to check packages is mvnrepository, I have found this site seems to take about a day to find new packages.

I hope this guide can help someone (and probably my future self), feel free to drop a comment if it helps or if something is unclear!

Footnotes / Useful links

https://theoverengineered.blog/posts/publishing-my-first-artifact-to-maven-central-using-github-actions

Adding Content Security Policy (CSP) Support to Embedded Tomcat 10

Continuing the series of hardening embedded Tomcat in Java to meet Nessus security scans, I am back with an example of adding a Content Security Policy to your app. There are some ways in a more standard Tomcat server to provide CSP policies, but with an embedded server that can be more difficult.

I have used an embedded Tomcat server for years to build applications. The following example is using Tomcat 10, but the principle is the same or Tomcat 9. The main difference as a Tomcat 9 to 10 transition is moving from the javax namespace to jakarta. With more and more libraries, such as Jooq, moving to more modern Java versions; as well as, some of the new Java versions offering good performance improvements out of the box, it may be time for everyone to move to the Jakarta namespace. (Even if that means leaving some libraries such as Google OAuth behind)

In my recent example project going over how to use Pac4J for Oauth with Tomcat 10, I have added an example here of what the FilterBase class would look like. You then need to initialize the filter where you are starting the Tomcat thread. That will add the needed header to all the web requests your application processes.