Some updates on this and also a few new questions:
Regarding clustering, for now I’ve decided keep the current state that all job statuses and logs are local basically - they’re stored in the same database table but have a node ID column that is filled with the current node ID. Similarly, job status blobs have a prefix for the node ID. New APIs to read job statuses from other cluster nodes can be introduced later.
An option that we discussed during the last developer meeting and that I have now implemented as a first PoC is to produce plain text log files in the file system for every job whose logs are isolated, i.e., not part of the main log. All log events inside jobs are now handled by Logback which decides, based on MDC metadata, to route some logs into separate log files, currently inside {environment.permanentDirectory}/logs/jobs/ (configurable in xwiki.properties or in logback.xml). The idea is that this allows admins to easily change the configuration of Logback to collect all logs including job logs in a central log collection facility while having metadata about the fact that a log is from a job, the type of the job and the exact job ID.
Where I’m unsure is if we should configure any kind of log cleanup. With the current way, job logs wouldn’t even be removed when the job status is deleted. So even if a job ID was re-used, which currently clears the logs, logs would keep appending. With Logback, we can’t easily get this globally for all the different logs across all jobs. What we could do:
- A custom cleanup job that runs on startup or once every X hours with a custom configuration in XWiki.
- Don’t have one log file per job ID, but a log file per job type (so, e.g., one log file for all rename jobs) and then configure log rotation and retention rules per job type, meaning that Logback would, e.g., produce daily archives of rename jobs and keep the last X of them (or up to a certain total size). This would be fully configurable in the Logback configuration file.