Since I am currently working on the migration of cristal from the xwiki-contrib to the xwiki GitHub organization, I’d like to discuss the splitting of the projects in repositories.
Introduction
Note: module refers to atomic Maven entities, while package refers to atomic npm entities. repository is only used as a git-related term.
Blocknote “headless”
See below the current repositories organization (the schema excludes many packages and modules that are not useful for the rest of the discussion).
In xwiki-contrib, the cristal repository contains:
all the code required for cristal and all of its backends (XWiki, File System, Nextcloud, GitHub)
a Blocknote package named blocknote-headless made to be reusable
In xwiki, the xwiki-platform repository contains a xwiki-platform-blocknote-webjar modules that reuse blocknote-headless to integrate Blocknote in XWiki (as an alternative to CKeditor 4).
The main problem is that while blocknote-headless is conceptually decoupled from the project in which it is integrated, this is currently not true for two reasons:
it is part of the cristal repository, and follows the same releases cycles as cristal
it depends on cristal APIs, and uses the same dependency injection mechanism (inversify based)
For those reasons, blocknote-headless is too connected to cristal, which makes its reusability at risk.
XWiki/Cristal front-end component reuse
A second aspect to consider is the goal to have more front-end components being reused between XWiki and Cristal.
As a thought experiment, I’m going to use Live Data (LD) as an example. How would we split the repositories if Live Data was extracted out of XWiki to be reusable by Cristal and XWiki?
Proposal
See below a few options for repository splitting.
Option 1: minimal
In this case we consider that the LD case is too anticipated, and we only extract Blocknote in its repository.
we reintroduce new APIs dedicated to Blocknote, and both XWiki and Cristal have to be modified to provide components for the new APIs (easy bridge to existing components this should be only glue code)
On last option is to “do nothing” and to only move cristal in the xwiki organization without changing anything else.
Conclusion
I’m +1 to go directly with option 2 as this is where we want to go.
@ClementEXWiki@mflorea I’d like to have your opinion regarding the code separation for Blocknote, in particular regarding the APIs location.
WDYT?
Thanks!
PS: This change will have an impact on how we deal with “snapshot” code and how we release, but to keep this proposal reasonably large, I kept them for a separate proposal. You are welcome to ask for details if you feel like you need to know more to make an educated choice.
Why is it named blocknote-headless? For me “headless” means that there’s no UI. is that true? Or maybe I have a bad definition of “headless”
In your description you don’t mention the dependency between xwiki-platform-blocknote-webjar and Cristal APIs. Why is that needed? Why isn’t the dep on blocknote-headless enough to transitively bring the Cristal APIs required by it?
I’ll check the rest but wanted to make sure I understood the current situation first.
You are right, the name needs to be changed too. If the package is moved to a separate repository, I propose to simply rename it @xwiki/blocknote.
Blocknote “headless” expects to have access to the implementation of a few inversify components (e.g., for links suggestions).
For each component, an API is defined in the form of an interface that needs to be implemented and injected (the same pattern an XWiki basically).
For now, all those APIs are Cristal specific.
Note: It took me a while to understand option 1 since it contradicts what’s said above. Maybe phrase it differently?
Could you elaborate on what “Blocknote APIs” are (with some example)? Are they APIs (in the sense of Java interfaces) that need to be implemented by both XWiki and Cristal?
I think you’re missing an arrow from Blocknote (Cirstal) to Cristal APIs, no?
PS: I don’t think we need to name it BlocknoteHeadless. We could just name it Blocknote and then in XWiki it’s BlocknoteXWiki and in Cristal it’s BlocknoteCristal. WDYT? It would match better the repo name too.
Why is there no “LiveData API” package in xwiki-frontend-commons?
Re the name I’d use the xwiki-commons prefix and change the suffix. There’s also the possibility of having:
xwiki-commons
|_ java/
|_ (current content from xwiki-commons, ie java libraries independent of the concept of wikis)
|_ javascript/
|_ (location of where we put commons libraries coded in javascript, independent of the concept of wikis)
Note that I have the feeling that your xwiki-frontend-commons javacript libs might depend on the concept of wikis which is a bit of a problem in commons. If they are, then it’s more xwiki-platform that would need to be split into 2: java and javascript directories. Actually both can exist (xwiki-commons/javascript and xwiki-platform/javascript).
Ofc we could also have xwiki-commons-java, xwiki-commons-javascript, etc. (ie different repos).
Option 2 seems better to me too, even if we don’t share more front end code between XWiki and Cristal as we could want to move our XWiki javascript code to a npm build instead of using Maven.
I’m personally in favor of option 2 as it would be much more simple I think.
Having all a shared repository containing all the parts that are common to both Cristal and XS would greatly reduce the risks of misaligning versions and reduce the complexity of ensuring everything is compatible with both projects.
we have code we want to share between XWiki and Cristal
we don’t want to keep that shared code in existingxwiki organization repos because they are too Maven (Java) oriented
we don’t want to keep that shared code in Cristal repo either because it risks becoming too Cristal oriented? And maybe because XWiki would depend on xwiki-contrib code, which we try to avoid?
so we need one or more separate repos; it doesn’t make sense to create another GitHub organization for this, and between existing xwiki-contrib and xwiki the latter makes the most sense, because this code is going to be bundled in XWiki Standard
then the choice is basically between using a single repo or multiple repos inside the xwiki GitHub organization
this choice usually boils down to the release cycle we want to have on these shared modules / packages:
if we want to release them all at once, with the same version, then we should use a single repo
otherwise, if we think we may need to release individually (some more often than others), we need separate repos
We tried the separate release cycle long time ago, and it wasn’t pretty, which is why we’re currently releasing all commons/rendering/platform modules at once. So it makes sense to do the same with the shared code we have between XWiki and Cristal, which is option 2.
I’m not sure about the repo name though. Technically it could contain server-side JavaScript code as well, which we might need in the future. So I wouldn’t limit it to “frontend”. The main difference from xwiki-platform is that we’re going to use node/pnpm instead of Maven, but I’m not sure what’s the best way to express that in the repo name.
But if I’m correct your proposal was about having different directories inside the existing xwiki-platform repository, while in my case I’m talking about having a new single repository dedicated to those javascript module. The main advantage of my solution is that it probably wouldn’t use maven at all, while in your case we’d need some tooling to use both maven and pnpm.
That we should use the same prefix as we currently have for java, ie xwiki-commons-xxx, xwiki-platform-xxx. And not xwiki-javascript-platform for ex but instead xwiki-platform-javascript.
That we could have xwiki-commons-java (optional) and xwiki-commons-javascript (and xwiki-platform-java and xwiki-platform-javascript)
OR that we keep the existing repos but introduce a subdirectory inside: xwiki-commons/java and xwiki-commons/javascript (where xwiki-commons/java would have a maven build and xwiki-commons/javascript would be a npm build). And same for xwiki-platform.
Since the release cycle should be the same, I’m more inclined to go towards 3).
Not exactly. Each directory would use its own build system. There’s no mix of build toosl in my ideas above. The only difference between 2) (what you also proposed but with 1) being different) and 3) is the # of repos and the fact that there’s no rename of the xwiki-commons/xwiki-platform repos (in solution 3)).
Let’s focus the discussion on details of Option 2, as it seems to be the agreed-upon option.
I’m +1 to preserve this approach as much as possible.
I had initially in mind to keep it “client-side only”, but following the discussion, I think we can also include other aspects such as the electron distribution, so +1 to choose a less specific name.
I’m not a fan of using specific technologies for name, but here using “npm” or “node” makes sense.
+1
And (p)npm is easily called from maven (as we already do for a few webjars). So we could keep centralizing all operations from maven commands.
I let others answer, but I’ll probably move to a PoC of what it could look like.
See a very broad draft of what it could look like below:
That brings the question of what we do for our non-node javascript modules that we already have in xwiki-platform-core. I was thinking that we could put all javascript “modules” in the same directory. What build tool would be the best to build pure client-side javascript (and execute tests)? Is there a problem building them with npm?
I think your diagram is missing the repo for Cristal “Standalone” (ie not XS) and its dependencies. xwiki-platform is there to build XS for me, not Cristal. I’m wondering if it’s ok for the cristal repo to depend on xwiki-platform…
Also, I think we should drop the term “Cristal” inside xwiki-platform. Unless we want to name the frontend of XWiki with the name “Cristal” (but how do we call the backend then? Do we really need a name?). It’s also ambiguous to have 2 Cristal (the cristal repo + some modules in XS with the name “Cristal”).
I had in mind to gradually migrate them to the same directory indeed.
This would make it easier to run tests and to uniformize their build process overall.
Good point. This is indeed not so clear.
If xwiki-platform is here only to build XS. Then all the components that are more generic than XS alone must be located outside (e.g., electron related packages, the cristal APIs, the cristal backends that are not XWiki).
In this case, my schema is not ok we need to move everything except the code that integrates reusable components in XS into a separate repository.
Same here. We should not setup anything that would make XWiki Standard end up with xwiki-contrib material, that’s something to fix, not something to make worse.
+1 for the xwiki organization
I’m not sure either what’s the best name, but it feels the difference it should be the last part of the name, so I’m more in favor of xwiki-platform-javascript than xwiki-javascript-platform.
I don’t think having two sub folders with different release cycles in the same repository is a good idea (even if you release them the same days with the same version, you cannot really have a single tag with a single release of the whole repository because they use 2 completely different release tools), so -1 for me for now.
Of course if those two are using the same build (and so release) tool (so building all those javascript module with Maven, but isn’t it a huge constraint ?) then it’s fine and the less repositories the better.
xwiki-commons, xwiki-rendering, and xwiki-platform keep having their usual release lifecycle.
The main difference is that they now also contain npm packages.
Those packages would be released to npm registries (https://nexus.xwiki.org/ for “snapshot” versions, https://www.npmjs.com/ for final versions).
The node part of each repository would contain reusable node packages of the relevant level of abstraction (i.e., wiki agnostic in commons, dedicated to rendering in rendering, and wiki aware in platform).
xwiki-cristal will contain everything that is not related to XS, i.e.,:
Everything related to electrons.
All the code related to non-xwiki backends (i.e., Nextcloud, File System, and GitHub).
Steps
Move of xwiki-contrib/cristal to xwiki/xwiki-cristal (simple ownership move, no code change). This step still has some consequences on the location of documentation, moving from cristal.xwiki.org to other locations on xwiki.org (including release plans, etc.).
Move gradually code from xwiki/xwiki-cristal to xwiki/xwiki-platform, xwiki/xwiki-rendering and xwiki/xwiki-commons. The step will also have consequences on the location of documentation but can be discussed case by case.
While step 1 is reasonably easy, step 2 raises many technical questions that I’m currently untangling:
How to publish “snapshot” versions of packages on nexus.xwiki.org?
How to publish npm packages automatically to registries during the release
What is the best organization inside the repositories to clearly separate the Node and Maven code while keeping a nice development environment (e.g., keep having webjars easily, have a clear way to rebuild artifacts during development)?
I’ll open separate discussions for those topics. Since some of those changes can have negative consequences on the stability of our release process, I tend to be careful and to be as confident as possible before actually deploying them.
I think I’d leave the Nextcloud + GH backends to contrib FTM as I’m not sure it should be the ownership of the XWiki Dev Team to develop them and it would be easier to get contributions in contrib.
It makes sense to have FS and XWiki backends in the XWiki GH org though.
Conceptually I’m fine.
Be aware that currently, backends can only be embedded at build time.
So, until we have a way to load backends at runtime (i.e., having a proper extensions mechanism and possibly a store), we can either
Keep having the Nextcloud and GH backends packaged in the Electron distribution by building the Electron distribution on the contrib repository.
Distribute the XWiki and File System backends only by building the Electron distribution on the GH org repository.
Technically they are not exclusive, and we could build and distribute two distributions until we invest time in the extension and store mechanisms.
