Reusable client-side packages translations handling

Development
1

Hello all,

Context

With each 18+ release, we publish npm packages on https://www.npmjs.com/.
These packages are located in the xwiki-platform-node module of xwiki-platform.
We publish them publicly because they are meant to be reusable outside xwiki-platform. The main target is the Cristal projects, but they are meant to be reusable even by other projects (similar to what we do with xwiki-commons and xwiki-rendering for server-side code).

Localization challenge

Since these packages are often UI oriented, they can contain localized string that are expected to be translated in the user’s locale, and to be translatable by contributors (i.e., using weblate).

Localization in Cristal

I’m mentioning this because many of the packages in xwiki-platform-node have been initialized in Cristal and moved to XWiki Platform later.

In Cristal, there is conceptually no server supporting the translation.
Translations files are defined as json files with the translation keys are the object keys, and the translation strings are the object values (see the example below). They are bundled in the final packaging and the translation is done fully client-side.

Example:
From BlockNote.

// langs/translation-en.json
{
  "blocknote.image.insertView.loading": "Loading...",
  "blocknote.image.insertView.upload": "Upload",
  "blocknote.image.insertView.uploadOrSelect": "Upload or select an attachment.",
  "blocknote.image.insertView.noResults": "No results",
  "blocknote.image.insertView.search.placeholder": "Image name or image URL",
  "blocknote.document.parsingError": "Failed to load document: {reason}"
}

Note that there is also inconsistencies because the Live Data packages, extracted from the Live Data webjar are a first effort to make them reusable, only resolve using server-side translation keys.
This module would need to be refactored as well to stay aligned with the rest of the code base.

What’s missing?

Having client-side only translations is too limiting for XWiki, where:

  1. (R1) We might want to have access to some translation keys server-side. This constraint is possibly optional, see below.
  2. (R2) Translations can be updated and overridden server-side

Challenges

(C1) While we aim for flexibility and the possibility to adapt the translation key resolution mechanism to rely on server-side features when they are available, we also don’t want to add more work for translators and developers.
Therefore, we aim for a solution that keep a single source of truth for translation keys.

(C2) We also of course want to keep translations modularity:

  • Translations are packaged along the module using them
  • When a module is installed, the translations are provided, when a module is removed, the translations are gone as well.

Proposals

Option 1 - Naive

Offer a way to override the translation component, and fully over-write it for XWiki.
XWiki doesn’t use the translations from Cristal.
This goes against C1.

Option 2 - Adaptative

Offer a fallback mechanism when an optional localization component is loaded.
When the component is missing, the client-side keys are used.
When the component is available, it is called first for resolution, then the client-side keys are used.

Pros:

  • Fulfils C1 and R2

Cons:

  • A request is sent server side for keys that might not be there. With the way the client-side translation module works, keys are cached, so this would only happen once per page rendering.
  • Does not comply with R1, unless the keys are re-declared in a format that XWiki can handle server-side (i.e., ApplicationResources.properties or translation wiki pages).

Option 3 - Server-side support

This is the same as option 2, but we also introduce a mechanism to load translations from json files stored in a standard location in webjars.
I haven’t fully worked out the implementation of this one, but mentioning it to have you opinion on how much R1 is critical.
This option can also be done in a second step if we want to work incrementally.

Another limitation to be aware of is that the format used client-side and server side are not strictly equivalent.
For instance, client-side The total is {count} (with named parameters) is supported, while this is not true server-side. This can be fixed with best practices but I still want to mention it.

We’ll also need to be careful with translation key naming because client-side only translation keys could be part of the global server-side translation key set in the future.

A note on Weblate.

The JSON format I presented above is the standard JSON File format natively proposed by Weblate.
I believe we can safely enable it for json translations in XWiki if the adoption option requires it.

Conclusion

I’m +1 for option 2 as I believe it covers most of our needs.
Option 3 can be added later on top of Option 2 and I propose to postpone it in the spirit of YAGNI.

Thanks for reading!

2

I’m not sure to see how it goes against C1? We could have the translations performed only once in Weblate and automatically duplicated in XWiki with some scripts no?

I don’t understand latest sentence: “when the component is available” you mean “when a server side translation is available”? then it’s this one that should be used, not the client side one, no? There might be some misunderstanding with the vocabulary here.

3

That’s indeed another option I did not mention. It is indeed also possible to have a mechanism to copy translations. In terms of translations relation mechanism, this would pair well with option 1.
An approach could be to have a special plugin that run when building weblate. This script could copy the translations json files from a declared list of npm dependencies, and bundle them in the XWiki compatible format in the resources.

Let’s call it Option 4.

In this case:

  • R1 is fulfilled - modulo the potential misalignment between translation formats as mentioned in my first message
  • R2 as well
  • C1 as well
  • C2 as well

Cons:

  • There is a risk for the translations files from a given module to be packaged in several webjars by accident. But I don’t think the consequences are bad except for marginally larger jars. But the keys being the same, the translations would stay consistent.

The sequences I have is mind is:

  • Using the component manager, check for the present of a component implementing role ServerSideTranslationsResolver (named to be refined).
  • When it exists, the component is called first to resolve the translations keys (performing an http request most likely)
    • If the keys are not all resolved server-side, the resolver falls back to the client-side resolution mechanism
  • When it does not exist, the client-side mechanism is used alone

I hope it is clearer.

4

My opinion is that having the translations in two places is a pain for developers and translators. Even if we could automate it in some way, it feels difficult as I suppose the two translation formats are quite different. Always asking the server if there is a translation also feels like a total waste of resources.

On the other hand, I fear that the ability to customize strings that are displayed in the UI is a feature that is expected by some admins and developers. Could we maybe add this as an extra feature? So a feature that allows specifying client-side translations in an XObject that override translations that are shipped in a node module? I guess this could also be a part of a feature that allows defining client-side modules inside wiki pages. [Edit] I don’t know how client-side translations are loaded, but in case it should make things easier, I think as a starter, it would be perfectly fine to just load such translation overrides in every request (with some caching), assuming that they’re small as they’re just customizations.

If we need client-side translations to be available server-side, I assume that implementing a way to load the JSON in Java should be doable. As far as I’m aware, we want to have support for named parameters, anyway, so this would be a good occasion then to implement it.