Software

VMWare EAM Failing, and not Allowing Upgrades

I was attempting to upgrade my homelab which I pushed to VMWare vSphere 8.0 because of… YOLO… and after a recent 8.0.1 update I was no longer able to upgrade individual ESXi hosts. I had already updated vCenter to the latest version, now I wanted to upgrade the hosts. That is my normal course of action, vCenter, then hosts; as recommended. When I went to upgrade the hosts I was told:

"Health check fails to retrieve data about service 'vSphere ESX Agent Manager' on '3 Node And Friends'. Verify that the service 'vSphere ESX Agent Manager' is running and try again."

This had me SSH into the appliance and looking at logs. (To quickly mention EAM = “vSphere ESX Agent Manager“) Here are some of the fun errors I was getting in “/var/vmware/eam/eam.log”:

  • “Re-login to vCenter because method: currentTime of managed object: null::ServiceInstance:ServiceInstance failed due to expired client session: null”
  • “failed to authenticate extension com.vmware.vim.eam to vCenter”

Some older guides mentioned unregistering EAM and then re-registering it. This broke my install even worse, and I ended up reverting to a snapshot. (Always snapshot before upgrades…) When I reverted back to before the vCenter upgrade, I realized that EAM was actually failing before the vCenter upgrade; except now I had EAM back in my extension list both on https://vcenter/mob/?moid=ExtensionManager and in vCenter, which was missing after I followed the guide saying to un-register it.

Now that I had the plugin registered, again, I found this KB, and this persons blog very helpful. I ran the recommended commands:

mkdir /certificate

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.crt

/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.key

python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.vim.eam -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s vcenter.my.domain -u Administrator@vsphere.local

And then EAM suddenly showed happy, and the log started showing useful things:

2023-06-06T16:53:37.573Z |  INFO | vim-monitor | ExtensionSessionRenewer.java | 190 | [Retry:Login:com.vmware.vim.eam:f86509907b4cb7c6] Re-login to vCenter b
ecause method: currentTime of managed object: null::ServiceInstance:ServiceInstance failed due to expired client session: null
2023-06-06T16:53:37.573Z |  INFO | vim-monitor | OpId.java | 37 | [vim:loginExtensionByCertificate:443bbd7c03dce9c6] created from [Retry:Login:com.vmware.vim
.eam:f86509907b4cb7c6]
2023-06-06T16:53:37.947Z |  INFO | vim-async-2 | OpIdLogger.java | 35 | [vim:loginExtensionByCertificate:443bbd7c03dce9c6] Completed.

Thats it! Now I can run updates again! If anyone has the same issue, drop a line in the comments. I hope this isn’t a big new vSphere 8.0 issue. I had upgraded this appliance from 7.0, and perhaps that or a cert issue caused issues.

Below is some of my eam.log to help people:

2023-06-06T02:20:29.728Z | ERROR | vlsi | DispatcherImpl.java | 468 | Internal server error during dispatch
com.vmware.vim.binding.eam.fault.EamServiceNotInitialized: EAM is still loading from database. Please try again later.
        at com.vmware.eam.vmomi.EAMInitRequestFilter.handleBody(EAMInitRequestFilter.java:57) ~[eam-server.jar:?]
        at com.vmware.vim.vmomi.server.impl.DispatcherImpl$SingleRequestDispatcher.handleBody(DispatcherImpl.java:373) [vlsi-server.jar:?]
        at com.vmware.vim.vmomi.server.impl.DispatcherImpl$SingleRequestDispatcher.dispatch(DispatcherImpl.java:290) [vlsi-server.jar:?]
        at com.vmware.vim.vmomi.server.impl.DispatcherImpl.dispatch(DispatcherImpl.java:246) [vlsi-server.jar:?]
        at com.vmware.vim.vmomi.server.http.impl.CorrelationDispatcherTask.run(CorrelationDispatcherTask.java:58) [vlsi-server.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_362]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_362]
        at java.lang.Thread.run(Thread.java:750) [?:1.8.0_362]
2023-06-06T02:20:31.769Z |  INFO | vim-monitor | ExtensionSessionRenewer.java | 190 | [Retry:Login:com.vmware.vim.eam:9ae94019eb8cb9a2] Re-login to vCenter b
ecause method: currentTime of managed object: null::ServiceInstance:ServiceInstance failed due to expired client session: null
2023-06-06T02:20:31.769Z |  INFO | vim-monitor | OpId.java | 37 | [vim:loginExtensionByCertificate:b63ca4cf0b995a54] created from [Retry:Login:com.vmware.vim
.eam:9ae94019eb8cb9a2]
2023-06-06T02:20:34.775Z |  INFO | vim-async-2 | OpIdLogger.java | 43 | [vim:loginExtensionByCertificate:b63ca4cf0b995a54] Failed.
2023-06-06T02:20:34.775Z |  WARN | vim-async-2 | ExtensionSessionRenewer.java | 227 | [Retry:Login:com.vmware.vim.eam:9ae94019eb8cb9a2] Re-login failed, due
to:
com.vmware.eam.security.NotAuthenticated: Failed to authenticate extension com.vmware.vim.eam to vCenter.
        at com.vmware.eam.vim.security.impl.SessionManager.convertLoginException(SessionManager.java:329) ~[eam-server.jar:?]
        at com.vmware.eam.vim.security.impl.SessionManager.lambda$loginExtension$4(SessionManager.java:154) ~[eam-server.jar:?]
        at com.vmware.eam.async.remote.Completion.onError(Completion.java:86) [eam-server.jar:?]
        at com.vmware.eam.vmomi.async.FutureAdapter.setException(FutureAdapter.java:81) [eam-server.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$ClientFutureAdapter.setException(MethodInvocationHandlerImpl.java:731) [vlsi-c
lient.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$RetryingFuture.fail(MethodInvocationHandlerImpl.java:578) [vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$RetryingFuture$RetryActionImpl.proceed(MethodInvocationHandlerImpl.java:625) [
vlsi-client.jar:?]
        at com.vmware.eam.vim.security.impl.ExtensionSessionRenewer.retry(ExtensionSessionRenewer.java:149) [eam-server.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$RetryingFuture.setException(MethodInvocationHandlerImpl.java:541) [vlsi-client
.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setResponse(ResponseImpl.java:239) [vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:286) [vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.HttpExchange.invokeWithinScope(HttpExchange.java:54) [vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.TracingScopedRunnable.run(TracingScopedRunnable.java:24) [vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.run(HttpExchangeBase.java:60) [vlsi-client.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_362]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_362]
        at java.lang.Thread.run(Thread.java:750) [?:1.8.0_362]
Caused by: com.vmware.vim.binding.vim.fault.InvalidLogin: Cannot complete login due to an incorrect user name or password.
        at sun.reflect.GeneratedConstructorAccessor58.newInstance(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_362]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_362]
        at java.lang.Class.newInstance(Class.java:442) ~[?:1.8.0_362]
        at com.vmware.vim.vmomi.core.types.impl.ComplexTypeImpl.newInstance(ComplexTypeImpl.java:174) ~[vlsi-core.jar:?]
        at com.vmware.vim.vmomi.core.types.impl.DefaultDataObjectFactory.newDataObject(DefaultDataObjectFactory.java:25) ~[vlsi-core.jar:?]
        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.ComplexStackContext.<init>(ComplexStackContext.java:30) ~[vlsi-core.jar:?]
        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.parse(UnmarshallerImpl.java:167) ~[vlsi-core.jar:?]
        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.unmarshall(UnmarshallerImpl.java:105) ~[vlsi-core.jar
:?]
        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:92) ~[vlsi-core.jar:?]
        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:86) ~[vlsi-core.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.SoapFaultStackContext.setValue(SoapFaultStackContext.java:41) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.processNextElement(ResponseUnmarshaller.java:127) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.unmarshal(ResponseUnmarshaller.java:70) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.unmarshalResponse(ResponseImpl.java:284) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setResponse(ResponseImpl.java:241) ~[vlsi-client.jar:?]
        ... 7 more
2023-06-06T02:20:34.777Z | ERROR | vim-monitor | VcListener.java | 124 | An unexpected error in the changes polling loop
com.vmware.eam.EamRemoteSystemException: Unexpected error communicating with the vCenter server.
        at com.vmware.eam.vim.server.impl.VimRoot.rootOperation(VimRoot.java:106) ~[eam-server.jar:?]
        at com.vmware.eam.vim.server.impl.VimRoot.currentTime(VimRoot.java:78) ~[eam-server.jar:?]
        at com.vmware.eam.vc.VcListener.main(VcListener.java:140) ~[eam-server.jar:?]
        at com.vmware.eam.vc.VcListener.call(VcListener.java:118) [eam-server.jar:?]
        at com.vmware.eam.vc.VcListener.call(VcListener.java:58) [eam-server.jar:?]
        at com.vmware.eam.async.impl.AuditedJob.call(AuditedJob.java:58) [eam-server.jar:?]
        at com.vmware.eam.async.impl.FutureRunnable.run(FutureRunnable.java:55) [eam-server.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_362]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_362]
        at java.lang.Thread.run(Thread.java:750) [?:1.8.0_362]
Caused by: com.vmware.vim.binding.vim.fault.NotAuthenticated: The session is not authenticated.
        at sun.reflect.GeneratedConstructorAccessor57.newInstance(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_362]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_362]
        at java.lang.Class.newInstance(Class.java:442) ~[?:1.8.0_362]
        at com.vmware.vim.vmomi.core.types.impl.ComplexTypeImpl.newInstance(ComplexTypeImpl.java:174) ~[vlsi-core.jar:?]
        at com.vmware.vim.vmomi.core.types.impl.DefaultDataObjectFactory.newDataObject(DefaultDataObjectFactory.java:25) ~[vlsi-core.jar:?]
        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.ComplexStackContext.<init>(ComplexStackContext.java:30) ~[vlsi-core.jar:?]
        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.parse(UnmarshallerImpl.java:167) ~[vlsi-core.jar:?]
        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.unmarshall(UnmarshallerImpl.java:105) ~[vlsi-core.jar
:?]
        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:92) ~[vlsi-core.jar:?]
        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:86) ~[vlsi-core.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.SoapFaultStackContext.setValue(SoapFaultStackContext.java:41) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.processNextElement(ResponseUnmarshaller.java:127) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.unmarshal(ResponseUnmarshaller.java:70) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.unmarshalResponse(ResponseImpl.java:284) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setResponse(ResponseImpl.java:241) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:286) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.HttpExchange.invokeWithinScope(HttpExchange.java:54) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.TracingScopedRunnable.run(TracingScopedRunnable.java:24) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.run(HttpExchangeBase.java:60) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:229) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:114) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:693) ~[vlsi-client.jar:
?]
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:674) ~[vlsi-client.j
ar:?]
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:371) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:322) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:195) ~[vlsi-client.jar:?]
        at com.sun.proxy.$Proxy51.currentTime(Unknown Source) ~[?:?]
        at com.vmware.eam.vim.server.impl.VimRoot.rootOperation(VimRoot.java:101) ~[eam-server.jar:?]
        ... 9 more
2023-06-06T02:20:34.778Z |  INFO | vim-monitor | VcListener.java | 125 | Full stack trace: com.vmware.eam.EamRemoteSystemException: Unexpected error communic
ating with the vCenter server.
        at com.vmware.eam.vim.server.impl.VimRoot.rootOperation(VimRoot.java:106)
        at com.vmware.eam.vim.server.impl.VimRoot.currentTime(VimRoot.java:78)
        at com.vmware.eam.vc.VcListener.main(VcListener.java:140)
        at com.vmware.eam.vc.VcListener.call(VcListener.java:118)
        at com.vmware.eam.vc.VcListener.call(VcListener.java:58)
        at com.vmware.eam.async.impl.AuditedJob.call(AuditedJob.java:58)
        at com.vmware.eam.async.impl.FutureRunnable.run(FutureRunnable.java:55)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
Caused by: (vim.fault.NotAuthenticated) {
   faultCause = null,
   faultMessage = null,
   object = ManagedObjectReference: type = ServiceInstance, value = ServiceInstance, serverGuid = f0ee8343-1721-4676-9069-1a837625c60b,
   privilegeId = ,
   missingPrivileges = null
}
        at sun.reflect.GeneratedConstructorAccessor57.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.lang.Class.newInstance(Class.java:442)
        at com.vmware.vim.vmomi.core.types.impl.ComplexTypeImpl.newInstance(ComplexTypeImpl.java:174)
        at com.vmware.vim.vmomi.core.types.impl.DefaultDataObjectFactory.newDataObject(DefaultDataObjectFactory.java:25)
        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.ComplexStackContext.<init>(ComplexStackContext.java:30)
        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.parse(UnmarshallerImpl.java:167)
        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.unmarshall(UnmarshallerImpl.java:105)
        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:92)
        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:86)
        at com.vmware.vim.vmomi.client.common.impl.SoapFaultStackContext.setValue(SoapFaultStackContext.java:41)
        at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.processNextElement(ResponseUnmarshaller.java:127)
        at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.unmarshal(ResponseUnmarshaller.java:70)
        at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.unmarshalResponse(ResponseImpl.java:284)
        at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setResponse(ResponseImpl.java:241)
        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:286)
        at com.vmware.vim.vmomi.client.http.impl.HttpExchange.invokeWithinScope(HttpExchange.java:54)
        at com.vmware.vim.vmomi.client.http.impl.TracingScopedRunnable.run(TracingScopedRunnable.java:24)
        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.run(HttpExchangeBase.java:60)
        at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:229)
        at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:114)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:693)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:674)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:371)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:322)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:195)
        at com.sun.proxy.$Proxy51.currentTime(Unknown Source)
        at com.vmware.eam.vim.server.impl.VimRoot.rootOperation(VimRoot.java:101)
        ... 9 more

2023-06-06T02:20:34.778Z |  INFO | vim-monitor | VcListener.java | 131 | Retrying in 10 sec.

Moving From Dropwizard to Spring Boot

Where I work historically, we have used Dropwizard, a Java framework for creating web apps a lot. This framework went head-to-head with Spring Boot, in the last few years and has seem to drop out of favor with the community. Nonetheless, I thought for a new project I would dive into using that to get more acquainted with it. After trying to get two of the basic things I needed done, I ended up giving up on Dropwizard and pivoting to Spring Boot instead.

The first big library I tend to use with Java projects, especially web apps, is jOOQ. This framework creates simple objects and gives many ways to interact with your database. The best feature for me is you can have a Gradle plugin scan your database, then create all the objects automatically in Java. Not only does this save you from handwriting a bunch of SQL queries, but it also means when you update the database (probably using something like Flyway), your objects automatically get updated. Now when you compile your program, if you forgot to add that new field somewhere when editing an object, you get a compilation error instead of the application silently failing SQL queries in production.

Dropwizard does not natively support jOOQ, I went looking for a library to add the support I needed. I found benjamin-bader/droptools library. It seemed to do what I needed. I got it wired in, and soon everything was working! I could make objects and with one or two lines edit objects web requests. Wonderful. Then Dropwizard did a major update; version 3.0 was created to keep the javax namespace, and 4.0 was created to move to the jakarta namespace. These versions also moved a bunch of the internals of the Dropwizard libraries around, meaning supporting libraries like droptools would need updated.

That’s when I saw droptools had not been updated for 3 years… I decided I would open a GitHub issue. With not hearing anything for a few days I started tinkering with it. I got a updated build working for Dropwizard 3.0 and did a pull request back to the main repo. In doing this I realized with the Dropwizard 3.0 and 4.0 split, we would need at least 2 versions of the library created at one time. Then on top of that, jOOQ 3.16 was the last to support Java 11, and jOOQ 3.18 was out as the main community supported branch. This means we need to make 4 versions; 2 with Dropwizard 3.0, and 2 with 4.0, then each one having jOOQ 3.16 and 3.18. I rewrite the build pipeline from the Travis CI the repo had to Github actions, and got all 4 versions compiling with some regex to do the edits in the code that were needed. I then used my earlier article, to publish these 4 assets to Maven Central.

This allowed me to update to Dropwizard 4.0, and the Jakarta namespace.

Next, I need to get basic authentication working. My plan is to use Google OAuth as the login mechanism. I do not feel like writing my own for a side project, and out of the ones out there (Google, Facebook, Twitter, Github) I thought it had the most coverage of people, with the least surveillance factor. It is easy enough to get setup with a developer account and get the client-id and secret I needed for OAuth.

Now I had to wire up the OAuth on the application side, this is not too hard I have done this many times with applications at work, but usually there I am using internal libraries. Heading over to the Dropwizard docs didn’t give me exactly what I wanted. They are pretty sparce, and when it comes to setting it up, they mention how to do OAuth but then mention you need to write your own Authenticator and Authorizer for it. I don’t want to do that. I have done that before for servlet-based apps, but this is supposed to be a fun thing, and on the general internet I want a supported auth library. I went searching for an example of how to use the OAuth system. I could not find anything that got me what I wanted.

Then I remembered using Pac4J before with other Java frameworks, this is a security library that has support for many login methods, and many web frameworks. Dropwizard is listed as supported! But the last time it was touched by a human, and not a bot, was over a year ago, and that was just for a small CI fix… I’ll try to get it working anyway!

The dropwizard-pac4j library is what I need, and there is a dropwizard-pac4j-demo which walks you through setting everything up! I get the demo working, I added in Google login support, which wasn’t there by default. Then I spent a day… Where I wanted to get this auth working on Dropwizard 3.0 or 4.0. I don’t want to start working in the older 2.x framework to get stuck later. I downloaded dropwizard-pac4j and the demo locally and started editing them to get the dependencies updated and try to get everything onto the jakarta namespace.

This is where the dependency hell came in. dropwizard-pac4j-demo depends on dropwizard-pac4j, which makes sense. dropwizard-pac4j sets a lot of your project versions based on what it has in it. After updating a ton of dependencies to try to get it to compile it came down to DropwizardTestSupport.java failing to run because it relies on jax-rs-pac4j. jax-rs-pac4j is still in the javax namespace and hasn’t been touched by a human in 6 months or more. This library would need to be updated, because it links directly to the main Jetty Server project which has a dependency on jakarta.servlet.SingleThreadModel in ServletHolder.java, which has been deprecated and removed (discussion), then and I could not get the demo project to load with any combination of dependencies. They all wanted this Jetty 11 file, which should have jakarta.servlet.SingleThreadModel removed, but doesn’t.

I went back and tried to move to Dropwizard 3.0, going back to the javax namespace, but that opened up a bunch of similar issues and a ton of conflicting dependencies in different versions of code dropwizard-pac4j needed. I have my code on GitHub if anyone wanted to continue this journey, or in the future things are in a better place.

With all of that, I thought I would go and check the documentation for Spring Boot. There is a giant page, with in-depth, step by step instructions on how to get Google or GitHub auth working in your app. There is a night and day difference between the support and thoroughness of Dropwizard docs and Spring Boot. With seeing that, I had to decide to change my plans away from Dropwizard. Many on the Java subreddit will debate Spring Boot vs Quarkus; for me, who has only used servlets in the past with embedded Tomcat, I think starting with the popular Spring Boot makes the most sense.

A cup of java with audio waves behind it, from Bing image creator

PodcastFeedHandler and Java-LAME

As I have mentioned in previous posts, I am working on a side project where I work with podcast feeds. Part of the idea is being able to act as a sort of middleware for podcasts. Give the application your podcasts, and you will be able to do things like search transcripts; and, if a podcast has a long intro on each episode, remove that. Searching for where someone mentioned a thing in an episode can be hard to research by just scrolling through an episode; if computer transcripts are 95% effective, that is better than nothing. For trimming podcasts, I find when you go through a big backlog of podcasts, hearing the same 1-minute intro every 30 minutes takes a good chunk of time.

Being that I use Java a lot for work, and am comfortable there, I wanted to make this project with Dropwizard and React. The first bit of the project has been working on the audio recognition engine, which will be a whole post in and of itself. After that I needed to start getting the supporting libraries I wanted. I tend to try to make as much of my code native to the language I am using as possible. That means we want to do as much in Java itself as possible. There are a ton of libraries that call out to FFMPEG or a command line app to handle the feeds; I don’t want to do that. If a side effect of this project is helping the community and writing some additional libraries, that is a win in my book.

PodcastFeedHandler

The first library I needed was a library to read AND write podcast feeds. With this app being middleware, we need to be able to do both. I found MarkusLewis‘s Podcast-Feed-Library, this works great for reading feeds in, but does not support writing. I took a look at his library and architecture a library similarly, except added the ability to get your feed object and then write it out again. In the end I made https://github.com/daberkow/PodcastFeedHandler. This library is written entirely in Java with no dependencies. Using Java 11, I can have all the native XML parsing I need. The rest of my project is in Java 17, but I thought others may find 11 useful. I am not sure there are any fragments in the library that wouldn’t allow me to go lower, except its 2023, and 11 is an older LTS at this point. An exciting part of that sub-project was getting Maven publishing working. Now I can publish for my domain of ntbl.co.

This project also got me used to using Github Actions. I have used CircleCI before but thought I would try Github Actions as they give you unlimited runtime for public repositories. Thanks Microsoft! I have the library build, get signed, and upload via Actions. I wanted to make sure the library preformed as I wanted and reached out to JetBrains to get an Open-Source maintainer license for Intellij. They kindly approved me!

Java-LAME

The next part of the project was parsing and fingerprinting the audio to search for duplicate segments. I will get more into that at a later time. To be able to fingerprint I needed the WAV/PCM format of the audio. Podcasts tend to be MP3 or AAC files. There are a ton of libraries to convert media in Java, except most of them had a FFMPEG external dependency. That is something I wanted to avoid. By having 100 percent native code, I can more easily create the workers that will handle these duties. Anywhere Java can run, they can run on or be compiled to; instead of having external dependencies.

I found nwaldispuehl‘s java-lame, this is a copy of the fantastic native Java port of LAME; described as “This java port of LAME 3.98.4 was created by Ken Händel for his ‘jump3r – Java Unofficial MP3 EncodeR’ project: http://sourceforge.net/projects/jsidplay2/“. The library hasn’t been updated in a while but does everything I need. It can convert MP3s but needs a file location to be passed in before converting to a byte array. I do not want to have to write to disk. The workflow would be, download podcast, store on disk, read from disk, convert. We should be able to do all this in memory. Doing all these operations in memory also means the workers do not need a bunch of scratch disk space, which is a plus. It’s more memory intensive but cuts down on disk usage. In 2023, I would rather have a slightly more memory intensive application than be doing a ton of extra read/writes to SSDs.

Throughout this project I have been thinking of: if I use it a lot or have friends using the web app, and it is constantly reading and writing audio files, how can I minimize bottlenecks. I forked the Github repo for java-lame, then added in paths to allow in-memory MP3 feeding and processing. This allows me to add a S3 client to the workers, and natively work on those files without ever writing to disk.

This library has a bunch of more functionality than I am using. It was a full LAME port, including the command line system and processing. I am planning to remove that as I go to shrink the library. I also want to replace some of the core conversion to WAV/PCM into having it in-memory compression, and functions to handle chunking the files and processing them piece by piece.

I took a This American Life episode, 1 hour in length, 67MB as MP3. Converting it to WAV/PCM I needed created a 678MB file. About a 10x size difference. Compressing that data lossless-ly with standard ZIP compression got the file down to 437MB, about 65% the size the original WAV/PCM. I can retrieve the ZIP data as a stream, and being audio, I am not jumping around; thus, that works well for me. 678MB for a file doesn’t sound so bad, a worker then just needs 1GB of RAM or so to process it, right? My worry is other podcasts. Shows like Dan Carlin’s Hardcore History can easily be 5 hours per episode, that is a 200+MB MP3, and then would be 2-3 GB of RAM to process one episode. If I can take 35% off for relatively small compute overhead, I want to.

I will post more as I go through the project. If these libraries or the blog have helped anyone, feel free to drop a comment! I always appreciate it when people do.

(The photo is something I through together on Bing image creator, its Java with audio 😊 )

Publishing Java Libraries to Maven Central with GitHub Actions and Gradle (Gradle 7/8 in 2023)

Intro

I recently started a new, grander, project for my spare time. The project involves working with Podcast feeds, and I was going to use this as an opportunity to use a framework I haven’t before, Dropwizard. I found a Java library that did what I needed in MarkusLewis – Podcast Feed Library; except this library only read feeds, I want to be able to read AND write. I decided to make my own, and I wanted to host the library, allowing anyone else to use it if they want. I created the repo and got a basic version working. This is the repo which can be referenced as an example.

I am using GitHub Actions as my CI/CD pipeline. I thought I should easily be able to host the final Jar files there for Gradle. Turns out, this is sort of true… If you host your library on GitHub itself, as this doc goes over, you can easily upload and host the packages; except there is no un-authenticated access to it. No matter what, an end user has to auth with Gradle/Maven before downloading the assets. Instead of dealing with that (specially for a public repo), I thought I would give a try to getting my package into Maven Central. Once I figured out the process, and found out how to publish with up to date Gradle, it as straight forward. I thought I would document it for the greater internet, and my future self (I have already used it). I know others have done this as well, except I wanted to do it with Gradle instead of Maven, with GitHub Actions doing all the work.

Throughout this guide there are items you need to record to bring to the next step, I have underlined the important ones.

Steps:

  • Setup Repo
  • Register For Maven/Sonatype
  • Setup GPG for Repo
  • Configure GitHub
  • Publishing

Setup Repo

Setup a normal GitHub repo, and setup a blank Gradle project. More on the repo/Gradle config later.

Register for Maven/Sonatype

Sonatype is the company who runs Maven Central. They allow free hosting and registration for Java Libraries; the main requirement is for it to be under 1GB in size per file. This adventure starts over at their Jira to register for an account, this Jira account will be your credentials for all future interactions with Maven Central, so make them secure, and have a long password! Once you have an account, use the above like to go to their Jira again to create an issue. This ticket grants you permissions to publish to https://s01.oss.sonatype.org/. You can also login there with the credentials created for Jira. You will have to verify either your GitHub account, or your domain before publishing. A bot handles all this and I had it done in 20 minutes or so. I have a domain I wanted to use, and there is a guide on how to go through this process.

Once you register a group ID you can use this account to publish anything under that ID; for example, I registered my domain of ntbl.co (making the group ID in Java terms “co.ntbl”. First, I published the library above, then I added a fork of a Java-Lame library; I tried to submit a ticket for the second library to be sure, and the bot tells you that you are already good to go.

Setup GPG for the Repo

One requirement for posting assets to Maven Central is to GPG sign the packages. This means we need to generate a key, and then upload the private portion of the key to GitHub secrets, and the public to a public key repository. Below are the commands to do this, the key ID is an example one I have, you will need to replace it with yours:

gpg --gen-key
gpg --list-keys
gpg --export --export-options backup --output public.gpg  co.ntbl.podcastfeedhandler
gpg --export-secret-keys --export-options backup --output private.gpg co.ntbl.podcastfeedhandler
gpg --export-ownertrust > trust.gpg
gpg --list-secret-keys --keyid-format LONG
gpg --export-secret-keys --armor 3F6F38BA13BEBB6941F823DCEFAAE414FF016215
gpg --keyserver keys.openpgp.org --send-keys 3F6F38BA13BEBB6941F823DCEFAAE414FF016215

Line 1 creates the keys, for name you enter the full project name, for example: co.ntbl.podcastfeedhandler . Group ID + the project root name. The email can be any email you have. Then the passphrase which will be used and uploaded to Github secrets. I suggest using a password generator and making it long, you should never have to actually type this in. Next, the exports are for you to back up the key incase the system you are creating it on dies and the data is lost in the GPG instance. You shouldn’t generally need it after this is setup, but it felt like best practice.

Record the output of the 7th line (export-secret-keys), that will need to be added to the Github secrets in the next step.

Configuring GitHub

The last command publishes the public keys to a global repo which is checked against. If this publish is not done, then the verification of the package will fail.

The two items we need to upload to GitHub for GPG are the password added when the key was generated, and the private key we got from the 7th command.

Go to your GitHub repo, then go to the Settings tab. Using the left-hand navigation, go to “Secrets and variables”, and the “Actions” submenu.

We need to create 4 secrets; these need to be kept secret:

  • GPG_SIGNING_KEY – The private key, copy the text from the “–export-secret-keys” command, this formats it correctly. The string should start with “—–BEGIN PGP PRIVATE KEY BLOCK—–“
  • GPG_SIGNING_PASSPHRASE – The password added when generating the key
  • OSSRH_TOKEN – This is the password you set for Sonatypes Jira
  • OSSRH_USERNAME – The Sonatype Jira username

Below is a minimal example build.gradle for your project. I removed a lot of normal extra things you would add to a buidl.gradle, to see a full example, visit this GitHub repo.

Gradle

plugins {
    id 'java-library'
    id 'signing'
    id 'maven-publish'
}

group = 'co.ntbl'
version = '0.1.2-SNAPSHOT'
rootProject.description = 'Read and Write Podcast feeds from Java.'

sourceCompatibility = 11
targetCompatibility = 11

tasks.register('createProperties') {
    doLast {
        new File("$projectDir/src/main/resources/version.properties").withWriter { w ->
            Properties p = new Properties()
            p['version'] = project.version.toString()
            p.store w, null
        }
    }
}

classes {
    dependsOn createProperties
}

jar {
    manifest {
        attributes(
                "Class-Path": "co.ntbl.podcastfeedhandler",
                "Main-Class": "PodcastFeedHandler",
                "Implementation-Title": project.name,
                "Implementation-Version": version,
                "Implementation-Vendor": "Daniel Berkowitz",
                "Build-Jdk": org.gradle.internal.jvm.Jvm.current(),
                "Gradle-Version": GradleVersion.current().toString()
        )
    }
    duplicatesStrategy = DuplicatesStrategy.EXCLUDE
    from {
        configurations.runtimeClasspath.collect { it.isDirectory() ? it : zipTree(it) }
    }
}

java {
    withJavadocJar()
    withSourcesJar()
}

ext.admin = System.getenv("MAVEN_USERNAME")

signing {
    required { admin }
    def signingKey = System.getenv("GPG_SIGNING_KEY")
    def signingPassword = System.getenv("GPG_SIGNING_PASSPHRASE")
    useInMemoryPgpKeys(signingKey, signingPassword)
    sign publishing.publications
}

repositories {
    mavenCentral()
}

dependencies {
...
}

//
// MAVEN
//

publishing {
    publications {
        mavenJava(MavenPublication) {
            from components.java

            pom {
                name = 'PodcastFeedHandler'
                description = rootProject.description
                url = 'https://github.com/daberkow/PodcastFeedHandler'
                licenses {
                    license {
                        name = 'MIT License'
                        url = 'https://github.com/daberkow/PodcastFeedHandler/blob/main/LICENSE'
                        distribution = 'repo'
                    }
                }
                developers {
                    developer {
                        id = 'daberkow'
                        name = 'Daniel Berkowitz'
                        email = 'dansberkowitz@gmail.com'
                    }
                }
                scm {
                    connection = 'scm:git:git://github.com/daberkow/PodcastFeedHandler.git'
                    developerConnection = 'scm:git:ssh://git@github.com:daberkow/PodcastFeedHandler.git'
                    url = 'https://github.com/daberkow/PodcastFeedHandler'
                }
            }
        }
    }
    repositories {
        maven {
            name = "OSSRH"
            if (admin) {
                credentials {
                    username = System.getenv("MAVEN_USERNAME")
                    password = System.getenv("MAVEN_PASSWORD")
                }
            }
            def releasesRepoUrl = 'https://s01.oss.sonatype.org/service/local/staging/deploy/maven2/'
            def snapshotsRepoUrl = 'https://s01.oss.sonatype.org/content/repositories/snapshots/'
            url = version.endsWith('SNAPSHOT') ? snapshotsRepoUrl : releasesRepoUrl
        }
    }
}

A few things to point out. Under publishing you need to enter all the information for this repository/project. If you have another publishing section in your Gradle file you will need to condense them together. Having multiple leads to Gradle getting confused and usually using the first one it sees. You will also see some variables such as “MAVEN_USERNAME”, these get the values of our secrets during the GitHub actions publish process, which we will go over next. I am getting the version, and using the end of it containing “SNAPSHOT” to say if we should publish to a snapshot repo or prod.

I also am using the build.gradle version as the canonical version. This variable could be in a Gradle settings file, or properties, but for ease I have it in the build file. I want 1 version file location; having multiple leads to more confusion during releases. The createProperties task creates a properties file that is added to the build to give the code itself a way to see which version it is. There are more elaborate ways to do this, but it works for me. This function does need the resources folder to be in the “src/main” folder; if your project is not using this the easiest way to add an “empty folder” is add the “resources” folder and then add the following .gitignore to it. This will make sure the contents of this folder are never saved.

# Ignore everything in this directory
*
# Except this file
!.gitignore

Requirements for posting to Maven central are: including source, checksums, Javadocs, and signing your packages. I am using useInMemoryPgpKeys to sign in GitHub Actions. This is part of the signing plugin. I have seen others use sign configuration.packages instead of sign publishing.publications, I found that not to work in many trials.

GitHub Actions

In your repository, create a .github folder, then a workflows folder. Below is my publish.yml, or it is available here. This file is currently set to publish when a new release is tagged, you can also change this to commits or some other trigger.

name: Publish package to the Maven Central Repository and GitHub Packages
on:
  release:
    types: [published]
jobs:
  publish-release:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout latest code
        uses: actions/checkout@v3

      - name: Set up JDK 11
        uses: actions/setup-java@v3
        with:
          distribution: adopt
          java-version: 11
      - name: Validate Gradle wrapper
        uses: gradle/wrapper-validation-action@e6e38bacfdf1a337459f332974bb2327a31aaf4b
      - name: Publish package
        uses: gradle/gradle-build-action@67421db6bd0bf253fb4bd25b31ebb98943c375e1
        with:
          arguments: publish
        env:
          MAVEN_USERNAME: ${{ secrets.OSSRH_USERNAME }}
          MAVEN_PASSWORD: ${{ secrets.OSSRH_TOKEN }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          GPG_SIGNING_KEY: ${{ secrets.GPG_SIGNING_KEY }}
          GPG_SIGNING_PASSPHRASE: ${{ secrets.GPG_SIGNING_PASSPHRASE }}

Here we convert GitHub secrets to local environment variables. Note the change in name from OSSRH_USERNAME to MAVEN_USERNAME and OSSRH_TOKEN to MAVEN_PASSWORD. This is simply to make the variables more clear, and they can be whatever you wish. We also validate Gradle for this final build. Another note, in my setup we are not passing assets from earlier builds into this publish stage, we are rebuilding the jar completely, depending on the size of your job, this may or may not make sense. If you have all this setup correctly, you should be able to commit the code, tag a release with “0.0.1-SNAPSHOT” or any version ending in SNAPSHOT and it should publish to the snapshot repo.

Publishing

Now that we have working snapshot releases, we need to do a full release. This involves you using the credentials created with the Sonatype Jira account earlier and logging into the Nexus panel. When you are ready, go to GitHub, and mark a new release with the version not ending with SNAPSHOT. The GitHub action should finish successfully, yet your asset is not up at https://repo1.maven.org/maven2/ yet. Head over to https://s01.oss.sonatype.org/ and click “Log In” in the top right.

Select “Staged Repositories” on the left. Note: this server seems to be very busy during the day, doubly so if it is a weekday. You will frequently see “There was an error communicating with the server: request timed out”. Come back later or keep hitting refresh.

Clicking a repository will allow you to browse the contents, and make sure it looks how you want it to. When you are ready you click “Close” at the top of the pane to finalize this version. Closing the repository starts all the checks on the repository, this includes making sure GPG signatures are there, the sources, Javadoc, and checksums are there. If they are not, you will get an error and be forced to Drop the release and try again. You also will get a vulnerability scan, including dependencies, to your email on file.

After the repo successfully closed, you can click Release! This is another stage where you can get many timeouts and be forced to wait till the server is less busy. After it successfully releases, it takes about 30 minutes for it to show up in the global Maven repo.

Selecting “Repositories” at the left allows you to browser the global Snapshots and Releases repositories; I have found this screen updates quicker than other locations to see if your assets are starting to propagate, including faster than the main Maven repo.

After about 30 minutes, your release should start to show up at Maven Search, although it can take longer. Another popular place to check packages is mvnrepository, I have found this site seems to take about a day to find new packages.

I hope this guide can help someone (and probably my future self), feel free to drop a comment if it helps or if something is unclear!

Footnotes / Useful links

https://theoverengineered.blog/posts/publishing-my-first-artifact-to-maven-central-using-github-actions

Adding Content Security Policy (CSP) Support to Embedded Tomcat 10

Continuing the series of hardening embedded Tomcat in Java to meet Nessus security scans, I am back with an example of adding a Content Security Policy to your app. There are some ways in a more standard Tomcat server to provide CSP policies, but with an embedded server that can be more difficult.

I have used an embedded Tomcat server for years to build applications. The following example is using Tomcat 10, but the principle is the same or Tomcat 9. The main difference as a Tomcat 9 to 10 transition is moving from the javax namespace to jakarta. With more and more libraries, such as Jooq, moving to more modern Java versions; as well as, some of the new Java versions offering good performance improvements out of the box, it may be time for everyone to move to the Jakarta namespace. (Even if that means leaving some libraries such as Google OAuth behind)

In my recent example project going over how to use Pac4J for Oauth with Tomcat 10, I have added an example here of what the FilterBase class would look like. You then need to initialize the filter where you are starting the Tomcat thread. That will add the needed header to all the web requests your application processes.

Bitbucket: Convert From Standalone ElasticSearch to Embedded OpenSearch

At work I maintain random stacks of software, and sometimes help people with other stacks that they maintain. Recently I was asked to help bring a Atlassian Bitbucket stack up to date. In the past Atlassian always included a built-in ElasticSearch (ES) server. This was used to index code in Bitbucket and allow searching. It’s not a hard requirement for the server to function, but important for user experience.

When an environment moves from Bitbucket Server to Bitbucket enterprise you are supposed to go to a standalone ES over the embedded one for performance. I don’t know if people elsewhere commonly do this, but the stacks I have seen have just continued to use the embedded version. Admittedly, these are smaller instances; at scale I would understand that. That was until recently, when due to a licensing change Atlassian could no longer embed a up to date ElasticSearch. For a while they decided the best way to move forward was to keep bundling the one from before the licensing change (I think 7.10).

This works until you have an infosec team use Nessus and find you have an out-of-date ES sitting around when 7.16, or the 8.0 branch are out. From all that, this one stack had moved to a standalone ES cluster. We also now had to install the Atlassian security plugin into ES; this was not a simple task, and this plugin only supports a few versions of ES, none of which were current. At least then we are at a BETTER spot with security.

Now fast forward a few months of this mess going on, and Atlassian moved Bitbucket from ElasticSearch to OpenSearch. OpenSearch is a fork of ElasticSearch at version 7.10.2 from Amazon to get around these new licensing terms. Normally if you were still using the embedded version of ES, when you did your next upgrade of Bitbucket it would move you to OpenSearch. Because this stack had already moved to standalone instance it did not migrate over. We are now in the worst of both worlds, off the supported path, and can’t get back on it. If you search the Atlassian documentation there are guides on how to move to a standalone version, but not back. A big catch I found was they use default passwords in the embedded version, that are not easy to find, which lead you making it hard to migrate back.

Migrating Back

Below are some notes I have on migrating back. Hopefully they help someone.

There are two main folders we will work in, one is your Atlassian Bitbucket installation folder for this version, I will call it %atlassian-install%, then there is your Bitbucket data folder that moves between your versions, with your upgrades, we will call that %bitbucket-home%. (Note: I did all this on Linux, but I am calling the variables that because it is easy)

Default %atlassian-install% is /opt/atlassian/bitbucket/7.21.7, or your current version. Default %bitbucket-home% is /var/atlassian/application-data/bitbucket, but I tend to move that to /opt.

Under %atlassian-install%/opensearch/plugins/opensearch-security/securityconfig/internal_user.yml is the details Bitbucket needs to connect to this OpenSearch instance. The default password is “bitbucket-changeit”. To create a new hash of a password, the following file needs to be given execute privileges and does not come with that on Linux; %atlassian-install%/opensearch/plugins/opensearch-security/tools/hash.sh .

Go into %bitbucket-home%/shared/bitbucket.properties if you have one, this file is created as you migrate between versions or databases; and remove any legacy elasticsearch username/password/url settings. For example: plugin.search.elasticsearch.baseurl or plugin.search.config.baseurl as shown in the documentation. The properties file overrides settings you have in the instance/database. You may have a SystemD service file to automatically start Bitbucket, this file has the start-bitbucket.sh file starting with -ns or --no-search to run a standalone instance, remove the no search option.

Now start Bitbucket and go to Administration -> Troubleshooting and support tools -> System Information, you will see Search failed to connect. Go to Administration -> Server settings, then enter your new search information there. If you just removed ElasticSearch, and started OpenSearch with the server, all you have to do is make sure the port is right, by default 7992 I believe, then make sure the username is “bitbucket” and the password is “bitbucket-changeit”. If you get a connection error it may be that you have to setup a TLS trust between Bitbucket and Opensearch, but that is outside the scope of this guide.

Below is the default %bitbucket-home%/shared/search/config/opensearch.yml

cluster.name: bitbucket_search
node:
  name: bitbucket_bundled

network.host: _local_
discovery.type: single-node

path:
  logs: ${BITBUCKET_HOME}/log/search
  data: ${BITBUCKET_HOME}/shared/search/data

action.auto_create_index: false

http.port: 7992
transport.tcp.port: 7993

# The OpenSearch security plugin stores its configuration in an index in the cluster itself. On startup if the
# security index doesn't exist yet, sitting this to true will cause the security plugin to read the yml files and
# configure the index using the contents of the files.
plugins.security.allow_default_init_securityindex: true

# Using the yml files with default initialisation, we create a bitbucket user and give it the all_access in-built role.
# However, access to the REST API is disabled by default even for the all_access role so we need to explicitly give
# it permission here so that the bitbucket user can access the OpenSearch REST API.
plugins.security.restapi.roles_enabled: ["all_access"]

# Mandatory TLS setup for transport layer
plugins.security.authcz.admin_dn:
  - CN=BITBUCKET
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.transport.pemcert_filepath: bitbucket.pem
plugins.security.ssl.transport.pemkey_filepath: bitbucket-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem

# Logs audit events to bitbucket_search_server.json
plugins.security.audit.type: log4j
plugins.security.audit.config.log4j.logger_name: audit
plugins.security.audit.config.log4j.level: INFO
Stable Diffusion Image, "computer with redhat logo on screen, in a field with mountains and a dinosaur in the background"

Clean Tenable Nessus Scans for RHEL 7 with Podman

There can be an alert misfire for Tenable Nessus plugins 137561, 138032, 142002 based on your YUM repo configuration. This leads to 3 medium alerts that should not be there.

Plugins:

  • RHEL 7 : OpenShift Container Platform 4.3.25 containernetworking-plugins (RHSA-2020:2443) (Plugin 137561)
  • RHEL 7 : OpenShift Container Platform 4.2.36 containernetworking-plugins (RHSA-2020:2592) (Plugin 138032)
  • RHEL 7 / 8 : OpenShift Container Platform 4.6.1 package (RHSA-2020:4297) (Plugin: 142002)

If you have a stack that is using podman with RHEL 7 and does not have the default redhat.repo file, then packages are installed that have newer versions in the OpenShift repos. Normally this would be fine, but the Nessus scanner is supposed to check if you have OpenShift repos enabled, and if not then stop and say the latest versions from RHEL 7 OS is good; but the check fails if you are missing the RHEL 7 OS repos. The OS repo HAS to be enabled also, or the check will show as failing. This situation can easily happen if you have an air gapped system or a system on Satellite where you are not using the default repo in redhat.repo. Luckily the baseurl does not matter, as long as you set the name to “rhel-7-server-rpms”, and I put the name= line in there for good measure, then the check will come back clean.

/etc/yum.repos.d/redhat.repo

[rhel-7-server-rpms]
name = Red Hat Enterprise Linux 7 Server (RPMs)
baseurl=file:///opt/rhel_7_x86_64_os/
enabled = 1
gpgcheck = 0
gpgkey = file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release

Before (or after) setting that, you will need to disable the YUM Redhat Subscription Manager plugin, or the next time you run “yum” it will wipe your redhat.repo and reload it from subscription manager. To do this, go to /etc/yum/pluginconf.d/subscription-manager.conf and set “enabled=0”. Also # subscription-manager config --rhsm.manage_repos=0

Below are examples of the errors you can see from Nessus. 

Remote package installed : containernetworking-plugins-0.8.3-3.el7_8
Should be                : containernetworking-plugins-0.8.6-1.rhaos4.2.el7
OR
Remote package installed : runc-1.0.0-69.rc10.el7_9
Should be : runc-1.0.0-81.rhaos4.6.git5b757d4.el7

The default thinking may be, it says I need to update to the OpenShift packages; then it makes sense to install the OpenShift repos. And if you go get a Redhat developer account to debug this, you have the OpenShift repos there. That is because the developer account gives you a lot of entitlements including OpenShift, and if you add the OpenShift repos to a bunch of systems, you may be liable to get OpenShift licenses, or get errors because those systems do not have the entitlements. The key is the packages say “.el7_8″/”.el7_9″ instead of “.rhaos4.2”. This is a plugin misclassification, not a need for updates.

Note: The image is a random AI generated one: Stable Diffusion Image, “computer with redhat logo on screen, in a field with mountains and a dinosaur in the background”. I think they are fun.

Missing Email Alerts from LibreNMS

I realized recently that I haven’t gotten any alerts from LibreNMS recently, including when I rebooted devices for patching. After going to the “Alert Transport”, and attempting to send a message I got “SNMP Error: Could not authenticate.” Others seem to recently get this as well. (Link)

Turns out after May 31st (although for me it seems more like June 6th, 2022) Google disabled simple password logins for Gmail accounts. You need to enable two factor auth, then enable an app specific password for LibreNMS. This was a good quick guide on how to do that. With LibreNMS sending alerts when something is wrong, but not having a alert that it is working, it may be worth going and checking if you use LibreNMS and Gmail.

Computer Vision for Datacenter Auditing

I am going to start a series of posts of random ideas I have had but not had time to fully implement. The first in this series is a idea I have worked on about ~3 years ago (November 2019)  for being able to audit a datacenter as well as map systems physicals location to their logical one in a network.

The core of the idea is to use cameras in a datacenter to see servers in the rack, these could be security cameras, then use that data to map out the datacenter and save the administrators time from having to manually preform these actions. The process begins by training a machine vision learning model on what a server looks like. Most of the time at work I am working with Dell servers so I thought that was a good starting point. To make the model generic enough I was just attempting to train it on what a 1U server looks like vs what a 2U server looks like.

At this point I needed A LOT of photos of servers with different lighting and angles. I took a bunch myself as a seed set of different racks I had, then I turned to the web. Where could I get a large assortment of photos of Dell servers in different configurations and lightings? The homelab section on reddit! People all the time post their setups at home and what they have. I went through and downloaded several hundreds photos of different peoples setups. Another place to get photos was from eBay where a lot of sellers put up photos of servers in different settings; the downside is that a lot of people reuse the same photos again and again. I don’t know if the internet has yet to figure out what the copyright rules of using photos from online to train a model.

I researched a bunch of different techniques and was playing around with OpenCV, but then found a tutorial that seemed to be in line with what I was looking to do. (This one is also good, and very similar material) I researched different image processing models, and played around with several.

Now that I had the photos, I downloaded GitHub – tzutalin/labelImg: 🖍️ LabelImg is a graphical image annotation tool and label object bounding boxes in images. This is a tool where you go to each photo, select the item you are trying to learn, and label it. This took a while, its fully manual work. A lot of the photos from the web had multiple servers in the photo, and each one would need to be selected. This proved to be one of the more time consuming parts of the project. I had to manipulate the photos to allow the rectangular bounding boxes to be able to fit the servers, even when the photos are at weird angles.

I had to pick some of the photos to be the training set, than other photos to be the testing set. With that metadata ready and everything marked, I converted the final metadata from XML to CSV, using xml_to_csv.py provided at the above example repos. That was then fed into Tensorflow. The system I had to start with for this other than a laptop was a CentOS 7 server, this proved to be very annoying because some dependencies such as protobuf were not available at new enough versions and had to be custom compiled.

It was time to let the model run for a while and see what it could learn. Several important things were learnt in this process. First, if you have GPUs makes sure you have a Tensorflow that is compiled and ready to use them. The speed you speed you get with and without them is kind of crazy. Also, more RAM and GPU helps a lot speed up the process. At first I was playing with this on just a laptop, and that one didn’t have the GPU drivers for CUDA. This was taking DAYS to work on the model. Later I switched to using GPUs I had in a server, and this greatly increased the iteration cycle speed.

Off the bat it was able to get a decent percentage recognition of the servers in the photos I had presented it! I do think a lot of the photos I then tested it on were fairly ideal conditions, with good lighting and camera angle. This may give a better than real world experience with it working. To improve the model I can always find more photos and train it with more images. I was able to get the model to recognize about 80% of the servers in racks I showed it at this time. Another factor that could help in the future is the evolution of cameras. A lot of places are replacing 720p/1080P cameras with 4k cameras, the more resolution the system has to work with the better.

The next step I wanted to do was start matching physical location to logical. The idea behind this is, I can find regions in a photo or video where servers are, and each server through its iDrac/IPMI allows me to blink front chassis lights. So one host at a time I will have automation send the command to blink the front chassis lights, and perhaps some lights on the HDDs, then scan for which region in the photo has started to blink!

This is the idea I have slowly worked on for the last little while, I have prototypes of most of it working, but have not had a lot of time to put into it.  The hope would be we could use existing cameras to get the footage we need to map existing datacenters we have. Then perhaps in the future port this system to something like Hololens, or Apple/Meta AR system. Once we have that mapping, now we can start to draw out the physical servers and their location in the world/racks on a webpage, and make it easier for people working in a datacenter to find boxes they need. Hopefully one day allowing for people to click a server on a webpage, and then connect into its controller without a human painstakingly going to each box and doing this mapping. Of course all of this is fixed by a team labeling each server, but where is the fun in that.

CentOS 8 Migration

I have a pipeline which creates live images to network boot different systems. Historically this has been based on CentOS. A little while ago I moved it to CentOS 8 because I had some newer hardware that was not supported on the older kernel of 7. Everything was working well until recently when CentOS 8 went end of life, and I could no longer rely on the CentOS 8 Docker containers.

The journey began for a new EL8 system. I wanted to keep on EL8 instead of switching to Streams because all the other systems I had running were EL8 (CentOS 8 or RHEL8), and I wanted to keep compatibility. At the same time, I didn’t want to do a new build of the image, have things break, and not realize it was because of a CentOS Streams change upstream. I also used the CentOS 8 docker container which seems to have been pulled, so that forced me to do this change now.

My first thought was Oracle Linux. It has been around for a while, is ALMOST drop in compatible, and can be used without going and getting licenses (RHEL). (There are some small silly things like instead of “epel-release” the package is “oracle-epel-release-el8”) This lead to nothing but issues. I replaced all the repos I had in the image creation stage with Oracle Linux ones, then every build I got a ton of “nothing provides module(platform:el8)” lines for any package that used yum/dnf modules. I spent a chunk of time on this, finding no real answers, and one Oracle support page that looked like it could help saying I needed to buy a support contract. Classic Oracle. At one point I thought it had something to do with Commit – rpms/centos-release – 89457ca3bf36c7c29d47c5d573a819dd7ee054fe – CentOS Git server where a line in os-release confuses dnf, but then that line was there. Also Oracle doesn’t seem to have a kickstart url repo, which is needed to do this sort of network boot. They wanted the end user to set that repo up, which may be the source of my issues. This also touched on the issue Disable Modular Filtering in Kickstart Repos – Red Hat Customer Portal, but I wasn’t even getting to a base OS setup, then I could make changes to the os and dnf for how it processes modules.

In my searches I did find this nice script to get bash variables for OS and version. https://unix.stackexchange.com/a/6348

Then I figured I would try either AlmaLinux or Rocky Linux. They both came out around when Redhat said Cent 8 was going away. Looking into both projects, they both are backed by AWS and Equinix who are big players, which made me feel a bit better about it. I had heard a bit more about Rocky and its support, so I tried that. I dropped in the new repos, and kickstart location, and everything just worked… Even things that were a issue when playing with Oracle Linux went away. For example, epel-release was once again called what it should be.

In the end so far it seems to be happy! We will see if any other small differences pop up and bite me…

Below is an example of the top of the kickstart I am using, if anyone is interested in more of how I create live images, leave a comment and I can do a post on it:

lang en_US.UTF-8
keyboard us
timezone Europe/Brussels --isUtc
auth --useshadow --enablemd5
selinux --disable
network --device=eno1 --bootproto=dhcp
skipx
part / --size 4096 --fstype ext4
part /opt --size 4096 --fstype ext4
firewall --disabled

url --url=https://download.rockylinux.org/pub/rocky/8/BaseOS/x86_64/kickstart/

# Root password
rootpw --iscrypted <Insert encrypted password here>

repo --name=baseos --baseurl=https://download.rockylinux.org/pub/rocky/8/BaseOS/x86_64/os/ --install
repo --name=extras --baseurl=https://download.rockylinux.org/pub/rocky/8/extras/x86_64/os/ --install
repo --name=appstream --baseurl=https://download.rockylinux.org/pub/rocky/8/AppStream/x86_64/os/ --install