Xwiki do not start after serverreboot

Help / Discuss
1

If I restart tomcat9 or the server xwiki do not start. I think that the problem is “solr”. This is what I see in catalina.out:

[2021-02-13 12:04:38] [info] Starting Servlet engine: [Apache Tomcat/9.0.31 (Debian)]
[2021-02-13 12:04:38] [info] Deploying web application directory [/var/lib/tomcat9/webapps/xwiki]
[2021-02-13 12:04:48] [info] 2021-02-13 12:04:48,325 [main] INFO  iPropertiesConfigurationSource - Loading [xwiki.properties] from [file:/var/lib/tomcat9/webapps/xwiki/WEB-INF/xwiki.properties]
[2021-02-13 12:04:48] [info] 2021-02-13 12:04:48,588 [main] INFO  o.x.e.i.ServletEnvironment     - Using permanent directory [/var/lib/tomcat9/webapps/xwiki/data]
[2021-02-13 12:04:55] [info] 2021-02-13 12:04:55,563 [main] INFO  o.x.s.s.i.EmbeddedSolr         - Starting embedded Solr server...
[2021-02-13 12:04:55] [info] 2021-02-13 12:04:55,567 [main] INFO  o.x.s.s.i.EmbeddedSolr         - Using Solr home directory: [/var/lib/tomcat9/webapps/xwiki/data/store/solr]
[2021-02-13 12:04:57] [info] 2021-02-13 12:04:57,068 [main] WARN  o.e.j.u.s.S.config             - Trusting all certificates configured for Client@573f95b2[provider=null,keyStore=null,trustStore=null]
[2021-02-13 12:04:57] [info] 2021-02-13 12:04:57,069 [main] WARN  o.e.j.u.s.S.config             - No Client EndPointIdentificationAlgorithm configured for Client@573f95b2[provider=null,keyStore=null,trustStore=null]
[2021-02-13 12:04:57] [info] 2021-02-13 12:04:57,370 [main] WARN  o.e.j.u.s.S.config             - Trusting all certificates configured for Client@3eea7c4b[provider=null,keyStore=null,trustStore=null]
[2021-02-13 12:04:57] [info] 2021-02-13 12:04:57,370 [main] WARN  o.e.j.u.s.S.config             - No Client EndPointIdentificationAlgorithm configured for Client@3eea7c4b[provider=null,keyStore=null,trustStore=null]
[2021-02-13 12:04:57] [info] 2021-02-13 12:04:57,395 [main] WARN  o.a.s.c.CoreContainer          - Not all security plugins configured!  authentication=disabled authorization=disabled.  Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external.  See https://s.apache.org/solrsecurity for more info

… if I stop tomcat9 and delete all folders (including the files) in /var/lib/tomcat9/webapps/xwiki/data/store/solr and restart tomcat9 xwiki starts as usual.

I installed tomcat9 from debian packages and xwiki-all-solr (from debian packages). This problem I get with release 12.10 and 13.0.

Is there a solution for this?
Here the output of uname -a:

Linux xwiki 4.19.0-14-amd64 #1 SMP Debian 4.19.171-2 (2021-01-30) x86_64 GNU/Linux

output of “systemctl status tomcat9”

root@xwiki:/# systemctl status tomcat9
● tomcat9.service - Apache Tomcat 9 Web Application Server
   Loaded: loaded (/lib/systemd/system/tomcat9.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2021-02-13 12:07:51 CET; 13min ago
     Docs: https://tomcat.apache.org/tomcat-9.0-doc/index.html
  Process: 697 ExecStartPre=/usr/libexec/tomcat9/tomcat-update-policy.sh (code=exited, status=0/SUCCESS)
 Main PID: 701 (java)
    Tasks: 86 (limit: 2358)
   Memory: 1.2G
   CGroup: /system.slice/tomcat9.service
           └─701 /usr/lib/jvm/default-java/bin/java -Djava.util.logging.config.file=/var/lib/tomcat9/conf/logging.proper

Has somebody a solution for this?

Best

2

Hi, I don’t see what is the problem. I don’t see any error in your logs.

3

Hi Vincent! Thank you for your response. I try to explain what my problem is:

I deleted this morning the database and all subfolders for environment.home in “data”. I also deleted the cache - so it is a system totally new installed with release 13.0. I get this output from catalina.out:


    [2021-02-17 06:32:34] [info] NOTE: Picked up JDK_JAVA_OPTIONS:  --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
    [2021-02-17 06:32:34] [info] Server version name:   Apache Tomcat/9.0.31 (Debian)
    [2021-02-17 06:32:34] [info] Server built:          Jan 19 2021 22:31:47 UTC
    [2021-02-17 06:32:34] [info] Server version number: 9.0.31.0
    [2021-02-17 06:32:34] [info] OS Name:               Linux
    [2021-02-17 06:32:34] [info] OS Version:            4.19.0-14-amd64
    [2021-02-17 06:32:34] [info] Architecture:          amd64
    [2021-02-17 06:32:34] [info] Java Home:             /usr/lib/jvm/java-11-openjdk-amd64
    [2021-02-17 06:32:34] [info] JVM Version:           11.0.9.1+1-post-Debian-1deb10u2
    [2021-02-17 06:32:34] [info] JVM Vendor:            Debian
    [2021-02-17 06:32:34] [info] CATALINA_BASE:         /var/lib/tomcat9
    [2021-02-17 06:32:34] [info] CATALINA_HOME:         /usr/share/tomcat9
    [2021-02-17 06:32:34] [info] Command line argument: --add-opens=java.base/java.lang=ALL-UNNAMED
    [2021-02-17 06:32:34] [info] Command line argument: --add-opens=java.base/java.io=ALL-UNNAMED
    [2021-02-17 06:32:34] [info] Command line argument: --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
    [2021-02-17 06:32:34] [info] Command line argument: -Djava.util.logging.config.file=/var/lib/tomcat9/conf/logging.properties
    [2021-02-17 06:32:34] [info] Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
    [2021-02-17 06:32:34] [info] Command line argument: -Djava.awt.headless=true
    [2021-02-17 06:32:34] [info] Command line argument: -Djdk.tls.ephemeralDHKeySize=2048
    [2021-02-17 06:32:34] [info] Command line argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
    [2021-02-17 06:32:34] [info] Command line argument: -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
    [2021-02-17 06:32:34] [info] Command line argument: -Dignore.endorsed.dirs=
    [2021-02-17 06:32:34] [info] Command line argument: -Dcatalina.base=/var/lib/tomcat9
    [2021-02-17 06:32:34] [info] Command line argument: -Dcatalina.home=/usr/share/tomcat9
    [2021-02-17 06:32:34] [info] Command line argument: -Djava.io.tmpdir=/tmp
    [2021-02-17 06:32:34] [info] An older version [1.2.21] of the APR based Apache Tomcat Native library is installed, while Tomcat recommends a minimum version of [1.2.23]
    [2021-02-17 06:32:34] [info] Loaded APR based Apache Tomcat Native library [1.2.21] using APR version [1.6.5].
    [2021-02-17 06:32:34] [info] APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true].
    [2021-02-17 06:32:34] [info] APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true]
    [2021-02-17 06:32:34] [info] OpenSSL successfully initialized [OpenSSL 1.1.1d  10 Sep 2019]
    [2021-02-17 06:32:34] [info] Initializing ProtocolHandler ["http-nio-8080"]
    [2021-02-17 06:32:34] [info] Server initialization in [360] milliseconds
    [2021-02-17 06:32:34] [info] Starting service [Catalina]
    [2021-02-17 06:32:34] [info] Starting Servlet engine: [Apache Tomcat/9.0.31 (Debian)]
    [2021-02-17 06:32:34] [info] Deploying web application directory [/var/lib/tomcat9/webapps/xwiki]
    [2021-02-17 06:32:37] [info] 2021-02-17 06:32:37,632 [main] INFO  iPropertiesConfigurationSource - Loading [xwiki.properties] from [file:/var/lib/tomcat9/webapps/xwiki/WEB-INF/xwiki.properties]
    [2021-02-17 06:32:37] [info] 2021-02-17 06:32:37,767 [main] INFO  o.x.e.i.ServletEnvironment     - Using permanent directory [/var/lib/tomcat9/webapps/xwiki/data]
    [2021-02-17 06:32:41] [info] 2021-02-17 06:32:41,120 [main] INFO  o.x.s.s.i.EmbeddedSolr         - Generating a new Solr home directory at [/var/lib/tomcat9/webapps/xwiki/data/store/solr]
    [2021-02-17 06:32:41] [info] 2021-02-17 06:32:41,136 [main] INFO  o.x.s.s.i.EmbeddedSolr         - Starting embedded Solr server...
    [2021-02-17 06:32:41] [info] 2021-02-17 06:32:41,136 [main] INFO  o.x.s.s.i.EmbeddedSolr         - Using Solr home directory: [/var/lib/tomcat9/webapps/xwiki/data/store/solr]
    [2021-02-17 06:32:42] [info] 2021-02-17 06:32:42,271 [main] WARN  o.e.j.u.s.S.config             - Trusting all certificates configured for Client@6859f102[provider=null,keyStore=null,trustStore=null]
    [2021-02-17 06:32:42] [info] 2021-02-17 06:32:42,272 [main] WARN  o.e.j.u.s.S.config             - No Client EndPointIdentificationAlgorithm configured for Client@6859f102[provider=null,keyStore=null,trustStore=null]
    [2021-02-17 06:32:42] [info] 2021-02-17 06:32:42,411 [main] WARN  o.e.j.u.s.S.config             - Trusting all certificates configured for Client@6e8627e6[provider=null,keyStore=null,trustStore=null]
    [2021-02-17 06:32:42] [info] 2021-02-17 06:32:42,411 [main] WARN  o.e.j.u.s.S.config             - No Client EndPointIdentificationAlgorithm configured for Client@6e8627e6[provider=null,keyStore=null,trustStore=null]
    [2021-02-17 06:32:42] [info] 2021-02-17 06:32:42,430 [main] WARN  o.a.s.c.CoreContainer          - Not all security plugins configured!  authentication=disabled authorization=disabled.  Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external.  See https://s.apache.org/solrsecurity for more info
    [2021-02-17 06:32:45] [info] 2021-02-17 06:32:45,595 [main] INFO  o.x.s.s.i.EmbeddedSolr         - Started embedded Solr server.
    [2021-02-17 06:32:52] [info] 2021-02-17 06:32:52,385 [main] INFO  o.x.s.f.i.FilesystemStoreTools - Using filesystem store directory [/var/lib/tomcat9/webapps/xwiki/data/store/file]
    [2021-02-17 06:32:52] [info] Deployment of web application directory [/var/lib/tomcat9/webapps/xwiki] has finished in [17,935] ms
    [2021-02-17 06:32:52] [info] Deploying web application directory [/var/lib/tomcat9/webapps/ROOT]
    [2021-02-17 06:32:53] [info] At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
    [2021-02-17 06:32:53] [info] Deployment of web application directory [/var/lib/tomcat9/webapps/ROOT] has finished in [503] ms
    [2021-02-17 06:32:53] [info] Starting ProtocolHandler ["http-nio-8080"]
    [2021-02-17 06:32:53] [info] Server startup in [18,503] milliseconds
    [2021-02-17 06:42:50] [info] 2021-02-17 06:42:50,723 [XWiki initialization] INFO  .HibernateDataMigrationManager - Checking Hibernate mapping and updating schema if needed for wiki [xwiki]
    [2021-02-17 06:42:51] [info] 2021-02-17 06:42:51,159 [XWiki initialization] INFO  .HibernateDataMigrationManager - Storage schema updates and data migrations are enabled
    [2021-02-17 06:42:51] [info] 2021-02-17 06:42:51,332 [XWiki initialization] INFO  .HibernateDataMigrationManager - No data migration to apply for wiki [xwiki] currently in version [121001000]
    [2021-02-17 06:42:51] [info] 2021-02-17 06:42:51,332 [XWiki initialization] INFO  .HibernateDataMigrationManager - Checking Hibernate mapping and updating schema if needed for wiki [xwiki]
    [2021-02-17 06:42:55] [info] 2021-02-17 06:42:55,131 [solr/indexer job group daemon thread - org.xwiki.search.solr.internal.job.IndexerJob@319ed137([solr, indexer])] INFO  o.x.s.s.i.j.IndexerJob         - Starting job of type [solr.indexer] with identifier [[solr, indexer]]
    [2021-02-17 06:42:55] [info] WARNING: An illegal reflective access operation has occurred
    [2021-02-17 06:42:55] [info] WARNING: Illegal reflective access by com.thoughtworks.xstream.core.util.SerializationMembers (file:/var/lib/tomcat9/webapps/xwiki/WEB-INF/lib/xstream-1.4.15.jar) to method java.util.concurrent.CopyOnWriteArrayList.readObject(java.io.ObjectInputStream)
    [2021-02-17 06:42:55] [info] WARNING: Please consider reporting this to the maintainers of com.thoughtworks.xstream.core.util.SerializationMembers
    [2021-02-17 06:42:55] [info] WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
    [2021-02-17 06:42:55] [info] WARNING: All illegal access operations will be denied in a future release
    [2021-02-17 06:42:55] [info] 2021-02-17 06:42:55,305 [solr/indexer job group daemon thread - org.xwiki.search.solr.internal.job.IndexerJob@319ed137([solr, indexer])] INFO  o.x.s.s.i.j.IndexerJob         - 0 documents added, 0 deleted and 0 updated during the synchronization of the Solr index.
    [2021-02-17 06:42:55] [info] 2021-02-17 06:42:55,305 [solr/indexer job group daemon thread - org.xwiki.search.solr.internal.job.IndexerJob@319ed137([solr, indexer])] INFO  o.x.s.s.i.j.IndexerJob         - Finished job of type [solr.indexer] with identifier [[solr, indexer]]

After I started xwiki I have to install the user and flavour as usual - everything works.


But if I restart the server - the catalina.out stops with this message (tomcat is up but xwiki do not start):
[2021-02-17 05:52:47] [info] Deploying web application directory [/var/lib/tomcat9/webapps/xwiki]
[2021-02-17 05:52:56] [info] 2021-02-17 05:52:56,977 [main] INFO  iPropertiesConfigurationSource - Loading [xwiki.properties] from [file:/var/lib/tomcat9/webapps/xwiki/WEB-INF/xwiki.properties]
[2021-02-17 05:52:57] [info] 2021-02-17 05:52:57,231 [main] INFO  o.x.e.i.ServletEnvironment     - Using permanent directory [/var/lib/tomcat9/webapps/xwiki/data]
[2021-02-17 05:53:04] [info] 2021-02-17 05:53:04,848 [main] INFO  o.x.s.s.i.EmbeddedSolr         - Starting embedded Solr server...
[2021-02-17 05:53:04] [info] 2021-02-17 05:53:04,852 [main] INFO  o.x.s.s.i.EmbeddedSolr         - Using Solr home directory: [/var/lib/tomcat9/webapps/xwiki/data/store/solr]
[2021-02-17 05:53:06] [info] 2021-02-17 05:53:06,152 [main] WARN  o.e.j.u.s.S.config             - Trusting all certificates configured for Client@38eb2140[provider=null,keyStore=null,trustStore=null]
[2021-02-17 05:53:06] [info] 2021-02-17 05:53:06,152 [main] WARN  o.e.j.u.s.S.config             - No Client EndPointIdentificationAlgorithm configured for Client@38eb2140[provider=null,keyStore=null,trustStore=null]
[2021-02-17 05:53:06] [info] 2021-02-17 05:53:06,444 [main] WARN  o.e.j.u.s.S.config             - Trusting all certificates configured for Client@49f2994f[provider=null,keyStore=null,trustStore=null]
[2021-02-17 05:53:06] [info] 2021-02-17 05:53:06,445 [main] WARN  o.e.j.u.s.S.config             - No Client EndPointIdentificationAlgorithm configured for Client@49f2994f[provider=null,keyStore=null,trustStore=null]
[2021-02-17 05:53:06] [info] 2021-02-17 05:53:06,469 [main] WARN  o.a.s.c.CoreContainer          - Not all security plugins configured!  authentication=disabled authorization=disabled.  Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external.  See https://s.apache.org/solrsecurity for more info

I think that solr is not configured well - however - I searched a lot but I didn’t find any solution or how to configure solr right for xwiki. I configured solr as “embedded” (as I’m not so familiar with solr and tomcat).

4

I fixed this issue. However - I found no solution to get it working with Tomcat 9. I removed Tomcat9 completely and installed jetty9 (from the Debian packages). Moved the xwiki - folder to the jetty9/webapps and restarted it. Worked without any problems.

The conclusion: I think it has something to do with the solr-implementation in Tomcat9 - the thing is, that I didn’t find any solution for a correct configuration. With jetty9 it works out-of-the-box.

5

But you did not use the xwiki-tomcat9-* package, right ? If you did the permanent directory would not be /var/lib/tomcat9/webapps/xwiki but /var/lib/xwiki/data.

There is nothing wrong in the log you pasted (yes the warning about Solr are expected and you definitely have them with Jetty too), but maybe you did not paste all the log.

6

Thank you Thomas!
I used the tomcat9 from the Debian-packages and installed xwiki from the war-file. I followed the recommendations from the documentation. I configured the environment.permantDirectory in xwiki.properties (var/lib/tomcat9/webapps/xwiki/data). By default this setting is commentet out - but I configured it there.

7

I think I experienced the same problem with XWiki 13.1 on MacOS 12.0.1 with Tomcat 9.0.56:

After installed XWiki and restarting Tomcat, Tomcat hangs on deploying XWiki without an error message.

Switching to XWiki 13.10.1 the problem disappeared.