Programming

Hamster holding a CD with Java in the background

IsoFileReader

A while ago I was working on a system to handle network boot operations. The main server is written in Java, and I needed to be able to read contents out of ISO files. The easy solution would have been to extract the ISO to the hard drive. I was trying to avoid that to save space; and with all the different images, not have thousands of tiny files littering the drive.

Then I found stephenc repo (java-iso-tools) for reading ISO files in Java. This library worked great! It had examples which helped me get started, and was fast to dive though a file. It supported the traditional ISO-9660 formatted files, which I needed, and I was good to go. Years later, the people over at CentOS and Redhat Linux had the idea to start putting giant SHA hashes as file names. Suddenly the disc images I was getting contained filenames that were 128 characters in length; and sadly java-iso-tools was failing to parse these names. To explain why, we need a bit of a dive into how the ISO-9660 standard works.

ISO-9660 is Developed by ECMA (European Computer Manufacturers Association) as ECMA-119, and then was adopted into ISO-9660. Thus, technically I was able to get the standards documents and investigate how ECMA-119 worked. Images start with a header; pointing to several tables, and the root folder file. The information about files on the disc span out from that root file. The root file, is the root directory on the image. From there every file is either a directory (with/without children) or a file which can be read.

The standard has had many changes to it over the years. While the original ECMA-119/ISO-9660 standard dates back to the start of the CD-ROM, over time people added to the standard. With PC’s at the time running MS-DOS and being able to save files to a FAT file system as 8 letter then 3 letter for extensions, the formatted needed added onto so one day CentOS could have 128 character file names. Some early additions to the format were Rock Ridge, and the Enhanced tables. When reading the first bytes from an image, there are several byte blocks which state which version of the standard they work with; this was forward thinking in this way. The basic tables help simple devices easily be able to read the discs. They can offer short file names, and point to the same binary data other tables later do. Then the enhanced tables can offer more information, and be able to add additional features to the disc. Some of these features can include things like file permissions.

At this point I had decided I needed to fix the problem and was going to write my own library to do it. While it sounds crazy, I enjoy writing these low level libraries. I started with the ECMA-119 standard, and going through the flow, like I was a CD-ROM device reading an image. I would later add on code for Rock Ridge, and reading all the enhanced tables, and even adding on a UDF parser.

I don’t want to spend too much time going through the standard. If you are interested: ECMA-119/ISO 9660 Standard, ECMA-167/ISO_IEC 13346/Original UDF Standard, Rock Ridge, UDF 2.60, there is a collection of the standard documents in depth. This post is more to talk about the project in general, and how I enjoyed working on it. A few constraints I set upon myself were I wanted it to be 100% in Java 8. That way it could be natively compiled if someone wanted to do that, wouldn’t just be connecting to some native binary tool, and would work with older Java code bases. The project currently targets Java 11 being the LTS out at the time I was working on it. I know there are many code bases out there which are Java 8, and I actually dont think there is any code except some tests using Java 9+ features. If someone had a Java 8 project, they could remove the tests and compile to 8. We live in a little bit of an odd time now, where a project like this targets more enterprise users who tend to be back on older versions. And at the same time Java 24 is coming out. I wanted to give high level classes that a user needing a simple tool could use; but at the same time have deeper level objects publicly available.

I was using this in the earlier mentioned network booting environment.There I can be building 100+ servers at a time; speed, small, and fast code were important. I ended up adding as test some performance benchmarks. I test the old library as my control, then I do normal file lookups as well as pre-indexed. I developed a system where certain heuristics of the image are taken and can be stored. Then you can feed in this initial “vector” I called it, of the image and a file vector. If the image matched the initial vector for a few characteristics, we could reasonably assume its the same image originally scanned, then instead of reading all the header tables, we jump to the location of the file vector with trust. This does leave it up to the developer to make sure they are matching pre-indexed images with vectors; but if you do, you can much faster serve files.

This project was fairly straight forward to test, I had many and there are many ISO images out on the internet. And plenty of them are Linux Images! I also had the older library which I could use as a control to test against. I ended up writing many tests which help when people send Pull Requests to make sure nothing has broken. This project I needed done to support what I was working on. There were a few places where I didn’t fully flush out the metadata, but left it to the end user to, if they cared about that data type. I spent a lot of time in Hex Fiend hex editor marking segments and trying to understand where code I had was breaking down.

Over the years of working in Open Source, and going to a technical college, I have seen many strong technical projects that are very impressive code, and can do a ton of interesting things. And then the developer focuses on interesting things they can make their code do, and spends no time putting documentation together. At the same time there are many project that get the job done, but aren’t anything special; these projects put a few documents together and maybe an example, and then get all the usage. The area developers hate to spend time, but can be the most valuable is documentation. That pushed me to spend a lot of time commenting the code, and writing a large README file showing how to work with the library.

I hope you will take a look at the project, maybe use it, and feel free to drop issues as they arise! I have been using the library in production for years now. It doesn’t get a ton of updates, because there hasn’t been a lot I need to add to it. When a PR or Issue arise I take care of it. And with the project being published under my work, I get a lot of automated PRs to help upgrade the library.

Take a look! https://github.com/palantir/isofilereader

Moving From Dropwizard to Spring Boot

Where I work historically, we have used Dropwizard, a Java framework for creating web apps a lot. This framework went head-to-head with Spring Boot, in the last few years and has seem to drop out of favor with the community. Nonetheless, I thought for a new project I would dive into using that to get more acquainted with it. After trying to get two of the basic things I needed done, I ended up giving up on Dropwizard and pivoting to Spring Boot instead.

The first big library I tend to use with Java projects, especially web apps, is jOOQ. This framework creates simple objects and gives many ways to interact with your database. The best feature for me is you can have a Gradle plugin scan your database, then create all the objects automatically in Java. Not only does this save you from handwriting a bunch of SQL queries, but it also means when you update the database (probably using something like Flyway), your objects automatically get updated. Now when you compile your program, if you forgot to add that new field somewhere when editing an object, you get a compilation error instead of the application silently failing SQL queries in production.

Dropwizard does not natively support jOOQ, I went looking for a library to add the support I needed. I found benjamin-bader/droptools library. It seemed to do what I needed. I got it wired in, and soon everything was working! I could make objects and with one or two lines edit objects web requests. Wonderful. Then Dropwizard did a major update; version 3.0 was created to keep the javax namespace, and 4.0 was created to move to the jakarta namespace. These versions also moved a bunch of the internals of the Dropwizard libraries around, meaning supporting libraries like droptools would need updated.

That’s when I saw droptools had not been updated for 3 years… I decided I would open a GitHub issue. With not hearing anything for a few days I started tinkering with it. I got a updated build working for Dropwizard 3.0 and did a pull request back to the main repo. In doing this I realized with the Dropwizard 3.0 and 4.0 split, we would need at least 2 versions of the library created at one time. Then on top of that, jOOQ 3.16 was the last to support Java 11, and jOOQ 3.18 was out as the main community supported branch. This means we need to make 4 versions; 2 with Dropwizard 3.0, and 2 with 4.0, then each one having jOOQ 3.16 and 3.18. I rewrite the build pipeline from the Travis CI the repo had to Github actions, and got all 4 versions compiling with some regex to do the edits in the code that were needed. I then used my earlier article, to publish these 4 assets to Maven Central.

This allowed me to update to Dropwizard 4.0, and the Jakarta namespace.

Next, I need to get basic authentication working. My plan is to use Google OAuth as the login mechanism. I do not feel like writing my own for a side project, and out of the ones out there (Google, Facebook, Twitter, Github) I thought it had the most coverage of people, with the least surveillance factor. It is easy enough to get setup with a developer account and get the client-id and secret I needed for OAuth.

Now I had to wire up the OAuth on the application side, this is not too hard I have done this many times with applications at work, but usually there I am using internal libraries. Heading over to the Dropwizard docs didn’t give me exactly what I wanted. They are pretty sparce, and when it comes to setting it up, they mention how to do OAuth but then mention you need to write your own Authenticator and Authorizer for it. I don’t want to do that. I have done that before for servlet-based apps, but this is supposed to be a fun thing, and on the general internet I want a supported auth library. I went searching for an example of how to use the OAuth system. I could not find anything that got me what I wanted.

Then I remembered using Pac4J before with other Java frameworks, this is a security library that has support for many login methods, and many web frameworks. Dropwizard is listed as supported! But the last time it was touched by a human, and not a bot, was over a year ago, and that was just for a small CI fix… I’ll try to get it working anyway!

The dropwizard-pac4j library is what I need, and there is a dropwizard-pac4j-demo which walks you through setting everything up! I get the demo working, I added in Google login support, which wasn’t there by default. Then I spent a day… Where I wanted to get this auth working on Dropwizard 3.0 or 4.0. I don’t want to start working in the older 2.x framework to get stuck later. I downloaded dropwizard-pac4j and the demo locally and started editing them to get the dependencies updated and try to get everything onto the jakarta namespace.

This is where the dependency hell came in. dropwizard-pac4j-demo depends on dropwizard-pac4j, which makes sense. dropwizard-pac4j sets a lot of your project versions based on what it has in it. After updating a ton of dependencies to try to get it to compile it came down to DropwizardTestSupport.java failing to run because it relies on jax-rs-pac4j. jax-rs-pac4j is still in the javax namespace and hasn’t been touched by a human in 6 months or more. This library would need to be updated, because it links directly to the main Jetty Server project which has a dependency on jakarta.servlet.SingleThreadModel in ServletHolder.java, which has been deprecated and removed (discussion), then and I could not get the demo project to load with any combination of dependencies. They all wanted this Jetty 11 file, which should have jakarta.servlet.SingleThreadModel removed, but doesn’t.

I went back and tried to move to Dropwizard 3.0, going back to the javax namespace, but that opened up a bunch of similar issues and a ton of conflicting dependencies in different versions of code dropwizard-pac4j needed. I have my code on GitHub if anyone wanted to continue this journey, or in the future things are in a better place.

With all of that, I thought I would go and check the documentation for Spring Boot. There is a giant page, with in-depth, step by step instructions on how to get Google or GitHub auth working in your app. There is a night and day difference between the support and thoroughness of Dropwizard docs and Spring Boot. With seeing that, I had to decide to change my plans away from Dropwizard. Many on the Java subreddit will debate Spring Boot vs Quarkus; for me, who has only used servlets in the past with embedded Tomcat, I think starting with the popular Spring Boot makes the most sense.

Adding Content Security Policy (CSP) Support to Embedded Tomcat 10

Continuing the series of hardening embedded Tomcat in Java to meet Nessus security scans, I am back with an example of adding a Content Security Policy to your app. There are some ways in a more standard Tomcat server to provide CSP policies, but with an embedded server that can be more difficult.

I have used an embedded Tomcat server for years to build applications. The following example is using Tomcat 10, but the principle is the same or Tomcat 9. The main difference as a Tomcat 9 to 10 transition is moving from the javax namespace to jakarta. With more and more libraries, such as Jooq, moving to more modern Java versions; as well as, some of the new Java versions offering good performance improvements out of the box, it may be time for everyone to move to the Jakarta namespace. (Even if that means leaving some libraries such as Google OAuth behind)

In my recent example project going over how to use Pac4J for Oauth with Tomcat 10, I have added an example here of what the FilterBase class would look like. You then need to initialize the filter where you are starting the Tomcat thread. That will add the needed header to all the web requests your application processes.

Java Windows Shortcut Library (Parsing and Creating!)

Recently I have been working on a project that involves extracting a bunch of files from zips. The problem I faced was all the shortcuts within the zips were hard coded to locations, making it impossible for me to move the extracted zip data to wherever I may want. I wanted a native library that could read and modify Windows Shortcuts so I could drop my zip data anywhere; my project is in Java, and its instant cross compatibility was needed. I know all my clients have Java installed, so that made its dependency not a issue. After looking around on the internet and finding several options, including the popular https://github.com/jimmc/jshortcut. Now the downside the this popular jShortcut library is you need a DLL, why you need a DLL to write a binary file, I am not sure. More specifically, you need a DLL for your PCs instruction set, ick! After searching the far reaches of github, and getting to the end of my rope I found https://github.com/kactech/jshortcut, written 5 years ago, and not really popular on github I thought I would give it a try. IT’S AMAZING! With no dependencies, and just a single include, you can write, modify, and create new Windows Shortcuts! There is example code included, and it couldn’t be easier to use. I just wanted to make sure anyone who has had the same problem knows about this great library.

6to4 Card Cleaner Github!

Here it is! The source code (all be it not great) for 6to4 card cleaner up on github. This program has become less useful recently because my school recently implemented IPv6 across the network, and that has seemed to fixed the problem, along with new images. Anyway have fun with it, build it out, add to it…

 

https://github.com/daberkow/6to4-Card-Cleaner

Update

Hello Internet, I am back at school; along with working on some new and exciting things. https://github.com/daberkow is going to be my new GitHub place where I put up fun and exciting code, as well as add older things I have worked on and not released code for. Below I will list a couple projects and their state:

Javascript/C# SSL: I am slowly working on putting up open source code of the Javascript/C# SSL code

6to4 Card Cleaner: There was a error in the server side version library, that was fixed and clients should be able to update to newest version of 1.63

Duplicate Image Thing: I havent done much with this, I will be adding it to github for anyway to play with

New Project! Jukebox: I am working on a Jukebox from scratch, it will be based on raspberrypi.org computer. Designed as follows using EL lighting.

CUFU v1.00

Here is my latest little creation, CUFU, which stands for COM USB Finder Utility. The purpose is if you have to plug in and out a USB to COM port cable a lot and it changes which COM it is in windows, making you go to device manager, now you can run this, and hit the icon, and bam it will tell you. Right now it will tell you by default if “Prolific USB-to-Serial Comm Port” is plugged in, and which is the most recent. That can be changed, there is a updater. Its a standard one week project. Also I wanted art for the about page, “art” is stretching it for what I made.

 

Download (Requires .Net v2)

https://github.com/daberkow/daberkow.github.io/tree/master/CUFU

WebViewer

So I’ve started on a new program at the behest of my friend David. It allows you to enter any website that has a bunch of items in a numerical storage system, or through RSS. Thats the idea anyway, some uses are reading webcomics, or looking at EVERY MICROSOFT KNOWLEDGE BASE ARTICLE IN ORDER! That’s the idea, right now it just goes to “Order of The Stick Comics”. But I installed my update class, so I will put updates out when they are ready and it will warn you, then allow you to download them. Links below, my second miror is limited, so please use the first if available.

Download:

https://github.com/daberkow/daberkow.github.io/tree/master/WebViewer

6to4 Card Duplication

Multiple versions of Windows, mostly Windows 7, support IPv6 networking, and if your network doesn’t support it, and the machine wants to link to a site or computer that does, it will make a virtual IPv6 to IPv4 converter card. Now sometimes Windows doesn’t reuse them, or delete them. So in this situation these cards keep being added to the system, adding and adding. Eventually the use will attempt to connect to a local server, or some other service, and Windows will try every card first, and then fail to connect. Timing out even though the real connection has no fault. It appears at around 180 virtual cards we start to see this. The solution is removing all the cards, but Windows wants you to remove them one at a time. Microsoft does have a tool for removing devices in a systematic order, but you need to get the one for your architecture, and some of these need you to download a large package and extract it. So I decided to make it simple, download and run my tool and it will remove these cards for the user. After the removal the user will need to reboot but then they can connect.

Now this doesn’t keep the computer from adding more cards in the future, and I don’t want to disable IPv6, for it is slowly rolling out and when the user actually gets it they should be able to use it. So the current release, v 1.2.2, has a few command line arguments, and will run in graphical mode if no arguments are presented. It is very simple, a few buttons that do their jobs. In the future I want to add a scheduler option, that will allow you to run it monthly automatically so if this problem persists it isn’t a pest, along with a updating function so no need to download a new version if installed.

The download link is below, it requires .Net 2.0, which is already installed on Windows Vista and Windows 7.

https://docs.google.com/leaf?id=0BxiWBWPPWSoLZGEzNGQ3ZTEtMWFlNS00YTBhLTk0NDgtYjY3MDE3NDAxMGU0&sort=name&layout=list&num=50