Standardizing our use of node in maven projects

Development
#1

Hello all,

I’d like to propose a standardization of our use of node in maven projects.

There is many benefit to it:

  • easier and more uniform definition of new front-end modules
  • easier definition of relationships between front-end modules (e.g., typescript type checking)
  • access to node specific code minification tools (e.g., Vite)
  • easier definition of front-end tests (e.g., Vitest)

This proposal requires some understanding of how maven and npm are working. I’ll try to be as didactic as possible but let me know if something is not clear.
This proposal suggests a lot of new technologies. They are based on our my experience selecting technologies for Cristal. I’ll try to motivate them quickly, but can’t fully develop to keep this proposal at an acceptable size.

We discussed recommending typescript as the default for front-end implementation. While this is part of the examples provided here, this proposal is not about that. Though, once this proposal implemented, it will be much easier to use typescript.
The same is true for other aspects such as testing, or migrating to Vue 3 (which uses Vite by default as its build tool).

Once this proposal is accepted, I’ll work on migrating xwiki-platform-livedata-webjar and xwiki-platform-notifications-webjar, as they are the two modules we have involving node in their build process.

I’ve created two draft pull request as a base for the following explanations:

Note: Testing is not present in the PoC, but is quite easy to integrate following the same principles.

Involved technologies

  • pnpm
  • typescript
  • vite

Design principles

  • Each module is independent of the others. In particular, the generation of the JavaScript from Typescript should not involve the inlining of code from imported modules
  • Modules can be type checked against the code they reuse from other modules
  • Defining a new webjar-node should involve as little code declaration as possible

Packaging choices

The history of dependency management in JavaScript lead to the introduction of a LOT of formats.
The most recent is ES modules (esm), but this is not supported by requirejs, which is the library we currently use for modules resolution.
Therefore, I suggest to use amd which is the format best supported by requirejs. Note that this is easy to switch to move to esm later simply by changing some vite configurations.

npm vs pnpm

The default dependency manager for node is npm. But, I propose to use pnpm because:

  • installation time and disk usage as much lower (local node_modules are built using symbolic links)
  • built in notion of “monorepo” (see pnpm-workspace.yaml)
    • linking modules using npm would require relative paths, which is really not convenient
  • the list of resolved dependencies is grouped in a single pnpm-lock.yaml at the root of the project instead of scattered in package-lock.json files on all modules (pnpm lock file is also easier to diff)

Vite

While other bundlers exist (e.g., esbuild, rollup), Vite has several advantages:

  • large community with a lot of plugins
  • default bundler for Vue, which will greatly simplify the migration to Vue 3
  • default bundler for Cristal

Module structure

├── pom.xml (see https://github.com/xwiki/xwiki-platform/pull/2880/files#diff-b817a3603e73d5e2abe089978a6024629e37ebfe6743e230093be7774dad37fb)
├── src
│   └── main
│       └── node // root of the node module
│           ├── dist // produced by "vite build", later copied on the webjar
│           │   ├── main.amd.js // actual code
│           │   ├── main.amd.js.map // source map between the typescript code and the compiled javascript
│           │   ├── main.d.ts // exported types 
│           │   └── main.d.ts.map // source map of the types
│           ├── lib // root of the source code
│           │   └── main.ts // import and reuse a simple demo function from lib-a
│           ├── node_modules // resolved dependencies, created when calling "pnpm install"
│           │   ├── typescript -> ../../../../../../node_modules/.pnpm/typescript@5.3.3/node_modules/typescript
│           │   ├── vite -> ../../../../../../node_modules/.pnpm/vite@5.1.1/node_modules/vite
│           │   ├── xwiki-platform-lib-a -> ../../../../../xwiki-platform-lib-a/src/main/node
│           │   ├── xwiki-platform-tool-node-tsconfig -> ../../../../../../xwiki-platform-tools/xwiki-platform-tool-node/xwiki-platform-tool-node-tsconfig/src/main/node
│           │   └── xwiki-platform-tool-node-viteconfig -> ../../../../../../xwiki-platform-tools/xwiki-platform-tool-node/xwiki-platform-tool-node-viteconfig/src/main/node
│           ├── package.json // node project dependencies
│           ├── tsconfig.json // type script configuration
│           └── vite.config.js // vite configuration
└── target
    ├── checkstyle-result.xml
    ├── classes
    │   └── META-INF
    │       └── resources
    │           └── webjars
    │               └── xwiki-platform-lib-b
    │                   └── 16.1.0-SNAPSHOT // copied from src/main/node/dist
    │                       ├── main.amd.js
    │                       ├── main.amd.js.map
    │                       ├── main.d.ts
    │                       └── main.d.ts.map
    ├── maven-archiver
    │   └── pom.properties
    ├── maven-shared-archive-resources
    │   └── META-INF
    │       ├── LICENSE
    │       └── NOTICE
    ├── site
    │   └── license-plugin-report.xml
    ├── test-classes
    │   └── META-INF
    │       ├── LICENSE
    │       └── NOTICE
    └── xwiki-platform-lib-b-16.1.0-SNAPSHOT.jar // final jar

Important details:

  • module lib-b depends on lib-a. This dependency is declared at two places: pom.xml to make sure that the EM install lib-a whenever lib-b is installed. In package.json for the bundler; the typeckecker (and also IDE support and link resolutions).
  • I’ve made the choice to keep the dist and node_modules outside target for several reasons:
    • copying the content of src/main/node to target causes trouble because of the duplicated module declaration
    • this is following the usual structure of node project, making it easier to have a good IDE support

Tools

xwiki-platform-tool-node-tsconfig

Provide the default configuration to be inherited by all webjar-node modules.

{
  "compilerOptions": {
    "declaration": true,                                 /* Generate .d.ts files from TypeScript and JavaScript files in your project. */
    "declarationMap": true,                              /* Create sourcemaps for d.ts files. */
    "forceConsistentCasingInFileNames": true,
    "strict": true,
  }
}

xwiki-platform-tool-node-viteconfig

Provide the default vite configuration to be inherited by all webjar-node modules.

import {resolve} from 'path'
import {defineConfig} from 'vite'
import dts from "vite-plugin-dts"
import peerDepsExternal from 'rollup-plugin-peer-deps-external';

export default defineConfig({
  build: {
    sourcemap: true, // Generate the source maps
    lib: {
      entry: resolve(process.cwd(), 'lib/main.ts'), // main.ts is the default entry point
      fileName: 'main',
      formats: ['amd'] // Default export format.
    },
  },
  plugins: [
    dts(), // Generate typescript types 
    peerDepsExternal() // All peer dependencies are excluded from the bundle.
  ]
})

Build steps

  • download node and pnpm
  • call “pnmp install”
  • execute “tsc --noEmit” to type check the module (vite will later do the compilation of typescript to javascript, without typechecking)
  • call “vite build” to bundle the project (involving the compilation of typescript here, but could also involve the transformation of esm based javascript modules to amd)
  • call “vitest”
  • copy the content of “src/main/node/dist” to the webjar
  • package and install the maven module

Conclusion

With this setup, setting up a node based webjar will be much easier and will allow us to more easily:

  • test our front-end code without having to deploy it in a browser
  • import and reuse code from other modules

WDYT?

Cristal projects repositories splitting
#2

Hi,

thanks for working on this, it looks good so +1 from me!

Any idea if we could simplify this in the future? Or if we could have at least some kind of checks to ensure there’s no inconsistencies?

#3

I guess we could imagine something where dependencies with a “workspace:*” version on dependencies or peerDependencies are considered as internal dependencies, and should also be part of the pom. And they, either derive the pom automatically, or introduce a check.
But this is a bit hackish, and based on many assumptions.
I would suggest going without it in a first step, and to do this later where we have a better experience of how we do this at a larger scale.

#4

Hmmm here you’re only talking about the case where we defined in same project lib-a and lib-b, but I guess we should do same declaration in both the pom and the package.json if it’s an external dep no? You said:

so doesn’t it mean that for any npm module that I might need for my JS to work I also need to declare it in the pom.xml so that it gets installed with EM? Or am I missing something here?

#5

Indeed, this is also true for external webjar dependencies. Making it even harder to identify correctly.

If you want the dependency to be declared in a separate file, yes.
There are two cases the npm dependency is not needed in the pom:

  1. this is not a runtime dependency (i.e., everything in the devDependencies section)
  2. the dependency can be bundled in the module (i.e., the parts of the code of the dependency that are used in the current module are “copy-pasted” and inlined inside the code generated for the module)

Case 2 can be interesting if a single module only uses a given dependency. Or, for small utility libraries where it’s better to inline.

#6

Hmmm right it’s always webjar dependencies in the pom.xml so it even always need a manual check to ensure the webjar dep exists with right version or to import it, right?

Indeed, and there’s a way to express that in package.json? Or it has to be configured elsewhere?

#7

Yes. Note that this is basically our current practice regarding webjars.

For now, the general rule I’ve defined is:

  • dependencies part of the ‘dependencies’ section are inlined (case 2)
  • dependencies part of the peerDependencies section are not inlined
  • dependencies part of the devDependencies section are, by definition, not inlined as they are only here for tool support

This can be specialized by editing the vite.config.js of a given module.
The base version is very lightweight as it’s basically just inheriting from the common config provided by xwiki-platform-tool-node-viteconfig (you just need to give a name to the module).

For instance:

import {defineConfig, mergeConfig} from 'vite'
import viteconfig from "xwiki-platform-tool-node-viteconfig"

export default mergeConfig(viteconfig, defineConfig({
  // More module specific config here if needed.
  build: {
    lib: {
      name: 'xwiki-platform-lib-a'
    }
  }
}))
1 Like
#8

AFAIK amd is what we use today in XS, right? See https://www.xwiki.org/xwiki/bin/view/Documentation/DevGuide/FrontendResources/JavaScriptAPI/#HRequireJSandjQueryAPIs

Any reason to have 2 modules instead of just one named xwiki-platform-tool-node? (BTW this could be required in xwiki-commons in case we want to share JS code in commons)

How do you encapsulate the usage of these files (I haven’t checked the source code of your examples) so that developers don’t need to configure the resources maven plugin? Is this done automatically by the webjar-node packaging type?

I’m curious especially since for functional tests we still have this todo:

  <!-- TODO: Move to use "functional-test" in the future when http://jira.codehaus.org/browse/MNG-1911 is fixed,
       see https://jira.xwiki.org/browse/XWIKI-7683 -->
  <packaging>jar</packaging>

If I understand correctly calling mvn install will execute internally all the steps mentioned in:

right?

I think an example of a minimal pom.xml would help in the proposal. I’ve tried checking your PRs but they’re full of lots of files and even stuff that don’t seem reated (like security advisories configs).

You shouldn’t copy them but instead configure the tools to use a different location. However your point about external tools (e.g. IDE) using these defaults make sense to me.

So in conclusion this proposal is about proposing both to use the mentioned tools (pnpm, vite) and the defined build practices for XWiki development, right?

For typescript, this will be another proposal, and what is currently proposed will work with javascript.

Looks good to me, +1

The only question I’m asking myself is whether we’ll be better off having 2 separate builds:

  • one for front end modules, using the native javascript/typescript ecosystem of tools directly
  • one for backend modules (java), using maven

Do we have limitation of using maven to build javascript/typescript stuff vs a native non-maven build?

Thanks

#9

exactly!

Yes, those could probably fit in a single module.
Moving it to commons is slight more difficult as for now they are easy to reuse in other modules because they are in the same repo. I’ll look into it though.

For xwiki-platform-tool-node-tsconfig and xwiki-platform-tool-node-viteconfig they are resolved as node dependencies of lib-a and lib-b

Those don’t need to go though maven dependencies as they are only used for npm build (as long as they stay internal to xwiki-platform).

Yes, the webjar-node packaging involve a default set of steps, defined in the components.txt (see PNPM Experiment by manuelleduc · Pull Request #2880 · xwiki/xwiki-platform · GitHub) from a new xwiki-commons-tool-webjar-node-handlers module in commons (following what we do already for webjar).

The security advisory file is a mistake, I’ll remove it.
All the rest is useful, a example of a minimal pom can be found here: PNPM Experiment by manuelleduc · Pull Request #2880 · xwiki/xwiki-platform · GitHub

<?xml version="1.0" encoding="UTF-8"?>

<!--
 * See the NOTICE file distributed with this work for additional
 * information regarding copyright ownership.
 *
 * This is free software; you can redistribute it and/or modify it
 * under the terms of the GNU Lesser General Public License as
 * published by the Free Software Foundation; either version 2.1 of
 * the License, or (at your option) any later version.
 *
 * This software is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
 * Lesser General Public License for more details.
 *
 * You should have received a copy of the GNU Lesser General Public
 * License along with this software; if not, write to the Free
 * Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA
 * 02110-1301 USA, or see the FSF site: http://www.fsf.org.
-->

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <parent>
    <groupId>org.xwiki.platform</groupId>
    <artifactId>xwiki-platform-core</artifactId>
    <version>16.1.0-SNAPSHOT</version>
  </parent>
  <packaging>webjar-node</packaging> <!-- define the default set of operations. -->
  <artifactId>xwiki-platform-lib-b</artifactId>
  <name>Lib B NPM</name>
  <properties>
    <xwiki.extension.name>Lib B NPM</xwiki.extension.name>
  </properties>
  <dependencies>
    <dependency>
      <groupId>org.xwiki.platform</groupId>
      <artifactId>xwiki-platform-lib-a</artifactId><!-- explicit runtime dependency to lib-a at maven level. -->
      <version>${project.version}</version>
      <scope>runtime</scope>
    </dependency>
  </dependencies>
</project>

I only copy at the end to the location expected by the webjar packaging (${project.build.outputDirectory}/META-INF/resources/webjars/${project.artifactId}/${project.version})

Exacly

Yes, this is already quite a big proposal and I don’t want to diverge on discussion with specific tools.
Overall, I think with the proposed structure, we make it way easier to integrate node base tool in our build without adding too much extra work on maven end.

That’s another interesting approach.
In this case we would also have to declare dependencies toward specific node modules in our packaging somehow.
By splitting node and maven project in separate modules, we wouldn’t benefit from the build order resolution of maven, and we’d need yet another extra node tool (e.g., nx) just for that.

As this is an actual pnpm project weaved inside a maven project, I don’t see any cons as of now.

#10

Hi Manuel, thanks for working on this. What’s not clear to me is how is pnmp able to find modules that were built with maven. When running mvn clean install inside lib-b (without building lib-a before), maven will see that pom.xml declares a dependency on lib-a and will download it to the local maven repository from Nexus (where it was built by the CI). Then, when pnmp comes into play, how does it know to look for lib-a in the local maven repository? I assume you are unpacking all these maven WebJar dependencies, like lib-a, in src/main/node/node_modules but where does this happen? I don’t see it in your PRs. Maybe I missed it.

Thanks,
Marius

#11

This is a limitation I should have mentioned earlier. All of this works well as long as all the dependencies are part of the same repo, and all the dependencies are build first (i.e., vite build).
pnpm is able to resolve dependencies by name without publishing them in a remote reposity as long as all the projects are part of the same pnpm workspaces.
But, pnpm is not able to find sources and typescript types (.d.ts) from the webjars themselves.
Meaning that it can be used when developing webjar-node modules outside xwiki-platform (e.g., a contrib extension).

For that, we’d need to add an extra step where we actually publish the result both as a webjar and as a npm package (on https://www.npmjs.com/ for instance, but nexus can also be used as a npm repository afaik).
From there, from the point of view of the extension, they would be usual external npm dependencies.
This is definitely something we can do, but this is an extra work that I suggest to do in a next step.

1 Like