NullPointerExceptions in catalina.log during startup

GOhrner · November 14, 2023, 6:33pm

It seems to be more of a cosmetical issue, is not new according to my logs, and does not seem to trigger any actual misbehaviour.

However, after a Tomcat restart, is see tons of NullPointerExceptions in my catalina.log:

[2023-11-14 19:14:18] [info] 2023-11-14 19:14:18,301 [main] ERROR DefaultExtensionLicenseManager - Failed to load license file at [null]
[2023-11-14 19:14:18] [info] java.lang.NullPointerException: Cannot invoke "java.net.URL.getPath()" because "licenseUrl" is null
[2023-11-14 19:14:18] [info] #011at org.xwiki.extension.internal.DefaultExtensionLicenseManager.initialize(DefaultExtensionLicenseManager.java:94)
[2023-11-14 19:14:18] [info] #011at org.xwiki.component.embed.InitializableLifecycleHandler.handle(InitializableLifecycleHandler.java:39)
[2023-11-14 19:14:18] [info] #011at org.xwiki.component.embed.EmbeddableComponentManager.createInstance(EmbeddableComponentManager.java:365)
[2023-11-14 19:14:18] [info] #011at org.xwiki.component.embed.EmbeddableComponentManager.getComponentInstance(EmbeddableComponentManager.java:451)
[2023-11-14 19:14:18] [info] #011at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:201)
[2023-11-14 19:14:18] [info] #011at org.xwiki.component.embed.EmbeddableComponentManager.getDependencyInstance(EmbeddableComponentManager.java:406)
[2023-11-14 19:14:18] [info] #011at org.xwiki.component.embed.EmbeddableComponentManager.createInstance(EmbeddableComponentManager.java:355)
[2023-11-14 19:14:18] [info] #011at org.xwiki.component.embed.EmbeddableComponentManager.getComponentInstance(EmbeddableComponentManager.java:451)
[2023-11-14 19:14:18] [info] #011at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:201)
[2023-11-14 19:14:18] [info] #011at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:189)
[2023-11-14 19:14:18] [info] #011at org.xwiki.extension.repository.internal.local.LocalExtensionStorage.<init>(LocalExtensionStorage.java:91)
[2023-11-14 19:14:18] [info] #011at org.xwiki.extension.repository.internal.local.DefaultLocalExtensionRepository.initialize(DefaultLocalExtensionRepository.java:100)
[2023-11-14 19:14:18] [info] #011at org.xwiki.component.embed.InitializableLifecycleHandler.handle(InitializableLifecycleHandler.java:39)

Is this supposed to happen?

Xwiki is 14.10.19, Debian Bookworm/Bullseye, Tomcat 9, Debian packages

tmortagne · November 15, 2023, 7:22am

It’s generally safe to assume that a NullPointerException is not expected.

GOhrner:

[2023-11-14 19:14:18] [info] 2023-11-14 19:14:18,301 [main] ERROR DefaultExtensionLicenseManager - Failed to load license file at [null]
[2023-11-14 19:14:18] [info] java.lang.NullPointerException: Cannot invoke "java.net.URL.getPath()" because "licenseUrl" is null
[2023-11-14 19:14:18] [info] #011at org.xwiki.extension.internal.DefaultExtensionLicenseManager.initialize(DefaultExtensionLicenseManager.java:94)

This error is very strange, it’s not really clear to me how it’s possible. Do you have any error or warning before that in the log ?

Which version of Java are you using (the error suggest it’s Java 17 or more) ? No problem was reported with Java 17, but it was not tested much with Java 21 yet AFAIK.

This specific nullpointer won’t have any bad consequence other than the log, but it might be the sign that there is a more serious root cause.

slauriere · November 23, 2023, 10:54am

I’m hitting this issue as well, with Java 17. I’m wondering if this could be due to Loading... since I’m running Tomcat 9.0.70. I wish I could give a try to a distinct version of Tomcat but on Ubuntu mantic I can’t figure how to downgrade to for instance Tomcat9 9.0.58 which is available in Ubuntu jammy: after adding the jammy apt sources list and running apt update it seems apt search won’t find version 9.0.58-1ubuntu0.1 while it’s listed in jammy. Probably a misuse of apt by me…

slauriere · November 23, 2023, 4:55pm

Downgrading to a Tomcat9 version lower than 9.0.70 fixed the issue for me. I downgraded because at the time of writing there’s no more recent Tomcat9 package available for the distribution I’m using (Ubuntu).

I managed to downgrade on Ubuntu as 23.10 follows:

Add Ubuntu 22.04 repositories apt sources

/etc/apt/sources.list

deb http://archive.ubuntu.com/ubuntu/ jammy main restricted universe multiverse
# deb-src http://archive.ubuntu.com/ubuntu/ jammy main restricted universe multiverse

deb http://archive.ubuntu.com/ubuntu/ jammy-updates main restricted universe multiverse
# deb-src http://archive.ubuntu.com/ubuntu/ jammy-updates main restricted universe multiverse

Run downgrade commands

sudo apt-get install -t jammy tomcat9=9.0.58-1ubuntu0.1
sudo apt-get install -t jammy tomcat9-common=9.0.58-1ubuntu0.1
sudo apt-get install -t jammy libtomcat9-java=9.0.58-1ubuntu0.1

There’s probably a single command which performs the three updates directly though.

tmortagne · November 24, 2023, 8:07am

Indeed, I just noticed it on Ubuntu – Package Search Results -- tomcat9. That’s not great…

Strangely, the last stable Debian distribution with tomcat9 (Debian 11/bullseye) stopped at 9.0.43 (they did upgrade to 9.0.70 on the unstable branch, which is probably why Ubuntu has it, but it never ended up in Debian 12).

I just reported Bug #2044495 “Upgrade tomcat9 to a version greater than 9.0.70” : Bugs : tomcat9 package : Ubuntu.

Also added a note on https://www.xwiki.org/xwiki/bin/view/Documentation/AdminGuide/Installation/InstallationWAR/InstallationTomcat/#HNullPointerExceptionsincatalina.logduringstartuponUbuntu23.x.

GOhrner · November 24, 2023, 1:08pm

Sorry for the delayed feedback, I was “under water” the last few weeks.

Side note: In Ubuntu, Tomcat is only contained in universe, so to my knowledge is not covered by official security updates. Keep this in mind if you run it on a server.

I’m running Debian 12 “Bookworm”, but with Bullseye (Debian 11) repos also entabled to get a security supported Tomcat version for Xwiki. I’ll probably switch to the new Xwiki-Jetty-packages soon now that they are available, but have not migrated yet.

I just listed the involved packages to provide feedback here and noticed something interesting:

root@XXX:~# dpkg -l | egrep 'tomcat|openjdk'
ii  libtomcat9-java                      9.0.70-2                                all          Apache Tomcat 9 - Servlet and JSP engine -- core libraries
ii  openjdk-11-jre-headless:amd64        11.0.21+9-1~deb11u1                     amd64        OpenJDK Java runtime, using Hotspot JIT (headless)
ii  openjdk-17-jre:amd64                 17.0.9+9-1~deb12u1                      amd64        OpenJDK Java runtime, using Hotspot JIT
ii  openjdk-17-jre-headless:amd64        17.0.9+9-1~deb12u1                      amd64        OpenJDK Java runtime, using Hotspot JIT (headless)
ii  tomcat9                              9.0.43-2~deb11u9                        all          Apache Tomcat 9 - Servlet and JSP engine
ii  tomcat9-common                       9.0.43-2~deb11u9                        all          Apache Tomcat 9 - Servlet and JSP engine -- common files
ii  xwiki-tomcat9-common                 14.10.19                                all          XWiki is a free wiki software platform written in Java with a design emphasis
ii  xwiki-tomcat9-mariadb                14.10.19                                all          XWiki is a free wiki software platform written in Java with a design emphasis
root@XXX:~#

(Note that openjdk-11 is installed as the Xwiki packages depend on it, but openjdk-17 is the used version.)

So actually libtomcat9-java, which seems to provide the much of Tomcat 9’s core, still seems to be included in and shipped with Debian Bookworm in version 9.0.70, while some parts (tomcat9-common) were discontinued and are only available in Bullseye and version 9.0.43.

Strangely, the last stable Debian distribution with tomcat9 (Debian 11/bullseye) stopped at 9.0.43

That’s not strange, at all.

Version 9.0.43, released in January 2021, was probably the current version when Bullseye was frozen (hard freeze in March 2021), and once the distribution is released, the packages are not updated any more except in really exceptional circumstances. Any security related and critical bugfixes are backported into the package versions included in the distribution to avoid unwanted disruptions / breakages in running installations. For most packages this works fine, for others which outdate faster and where you you don’t rely that much on stability, there’s the -updates repositories, where some packages are provided in more recent versions indeed. If you go even further, you have to resort to the Backports repositories, which I would not really recommend for servers.

Other Linux distributions take exactly the same or very similar approaches.

As we see here with Tomcat 9.0.70, and there were also other “stable” Tomcat 9 releases with critical regressions, we can see that’s warranted.

To get back on topic, the thing with libtomcat9-java in Bookworm is peculiar, though.

In any case, my Tomcat indeed reports itself as 9.0.70, and also libapr it uses got updated:

[2023-09-21 19:05:22] [info] Server version name:   Apache Tomcat/9.0.43 (Debian)
[2023-09-21 19:05:22] [info] Server version number: 9.0.43.0
[2023-09-21 19:05:22] [info] Loaded Apache Tomcat Native library [1.2.26] using APR version [1.7.0].
(...)
[2023-11-14 18:50:11] [info] Server version number: 9.0.70.0
[2023-11-14 18:50:11] [info] Loaded Apache Tomcat Native library [1.2.35] using APR version [1.7.2].

I don’t see any misbehaviour of the running Xwiki application, though.

GOhrner · November 24, 2023, 1:13pm

Ok, if I understand Loading... correctly, I would

either need to dowgrade to Tomcat 9.0.43 from Debian Bullseye,
or open an issue against Tomcat 9.0.70 libraries in Debian Bookworm and hope for a backport, right?

(BTW, Links to Xwiki Jira seem to be broken here… At least for me it only displays “Loading…” except of the issue key or additional information:

)

tmortagne · December 1, 2023, 3:31pm

Not sure I understand, Debian Bookworm does not have tomcat9 at all anymore. They moved to tomcat10 (which cannot be used with XWiki currently).

tmortagne · December 1, 2023, 3:33pm

The link works fine, the problem is the preview (not sure if the problem is on Discourse or Jira side but we have the problem with all links to https://jira.xwiki.org).

GOhrner · December 1, 2023, 3:43pm

That’s also what I thought until I stumbled across my findings above.

If this really were the case, I’d never had problems with these NullPointerExceptions.

root@XXX:~# apt-cache policy libtomcat9-java 
libtomcat9-java:
  Installiert:           9.0.43-2~deb11u9
  Installationskandidat: 9.0.43-2~deb11u9
  Versionstabelle:
     9.0.70-2 500
        500 https://mirror.netcologne.de/debian bookworm/main amd64 Packages
        500 http://ftp.de.debian.org/debian bookworm/main amd64 Packages
 *** 9.0.43-2~deb11u9 1200
        500 http://security.debian.org/debian-security bullseye-security/main amd64 Packages
        100 /var/lib/dpkg/status
     9.0.43-2~deb11u6 1200
        500 https://mirror.netcologne.de/debian bullseye/main amd64 Packages

“Only” the tomcat9 and tomcat9-common packages are missing, libtomcat9-java which contains most of Tomcat is shipped in version 9.0.70.

As you can see in the log in one of my previous postings above, Tomcat (accidentally) started with tomcat9 and tomcat9-common from Bullseye and libtomcat9-java from Bookworm announces itself as version 9.0.70 in the logs and exhibits the NPE / paths-with-spaces bug.

I now used apt pinning to also downgrade libtomcat9-java to the version in Debian Bullseye / Bullseye Security, so I’m not longer running a “Franken-Tomcat”. However if Debian had patched the paths-with-spaces bug in their libtomcat9-java package in Bookworm, it would also have worked probably (and I may never have noticed that I was running non-matching package versions…)

The link works fine, the problem is the preview

Yes, my wording was not clear. I was just referring to the link’s rendering / visual appearance.

tmortagne · December 1, 2023, 3:48pm

I saw your message about libtomcat9-java but I thought it was just some lib leftover from the name, are you saying that libtomcat9-java alone install you a fully functional Tomcat 9 (.0.70) service ?

GOhrner · December 1, 2023, 6:17pm

No, unfortunately that’s also not the case. tomcat9 and tomcat9-common are really missing and would be required to get a fully functional service.

Nevertheless, most of Tomcat is contained in libtomcat9-java, that’s why the (accidentally, because I didn’t know about libtomcat9-java) mixed-up Tomcat on my system exhibited the buggy behaviour of Tomcat 9.0.70.

From my point of view, the current situation in Debian is quite strange, I don’t really understand why they are shipping most of Tomcat but not the systemd service unit and some other required stuff, while most of Tomcat is still contained in the distribution… I can only imagine that it’s maybe because of other packages which depend on it - but then it may make sense to patch the spaces-in-filenames bug of Tomcat 9.0.70…)

root@XXX:~# apt-cache rdepends libtomcat9-java
libtomcat9-java
Reverse Depends:
  tomcat9-common
  tomcat9-common
  libjetty9-extra-java
  libtomcatjss-java
  libjetty9-extra-java
  libresteasy3.0-java
  libmina2-java
  liblogback-java
  libspring-context-java
  libjetty9-extra-java
  libequinox-jsp-jasper-java
  libjetty9-extra-java
  liblogback-java
root@XXX:~#