Brainstorming: Should XWiki support locales with a script (e.g. zh_Hans, sr_Latn)?

Development
1

Hi devs,

Context

The upgrade to Commons Lang 3.21.0 (XCOMMONS-3819) broke LocaleUtilsTest#isValid on master (build 8992).

org.xwiki.localization.LocaleUtils.isValid() (which is what filters the JVM locales returned by LocaleUtils.getAvailableLocales(), i.e. what we offer in the UI) considers a locale valid if its toString() can be parsed back by Commons Lang’s LocaleUtils.toLocale(). Until 3.20.0, that call threw for locales with a script (zh__#Hans, sr_BA_#Latn) or extensions (ja_JP_#u-ca-japanese, th_TH_#u-nu-thai), so we implicitly excluded them. Since 3.21.0 they parse, so suddenly:

  • 3.20.0: 765 valid locales out of the 1069 the JVM provides
  • 3.21.0: all 1069 are “valid” (+304 locales, the vast majority being script variants like af_ZA_#Latn which duplicate a plain af_ZA)

Note that with 3.21.0, toLocale("zh__#Hans") does not even give back the original locale (equals() is false), so it’s not real support either.

I’ve opened a regression fix that rejects locales with a script or extensions explicitly in isValid(), which restores exactly the 765 locales we had before (no behavior change compared to 18.8): XCOMMONS-3819: Upgrade to Apache Commons Lang 3.21.0 by vmassol · Pull Request #6617 · xwiki/xwiki-platform · GitHub

As @tmortagne said on Matrix, this shows we’ve never explicitly defined which kind of locales we allow, and we’ve been relying on Commons Lang’s (and the JVM’s) behavior instead. Recent Java versions expose a lot of weird stuff and many duplicates. So I’d like us to decide what we want for the future.

Options

A. Keep XWiki locales to language[_COUNTRY[_variant]] only, and say so explicitly (what the fix does)

  • Make it an explicit, documented rule and test it ourselves (instead of relying on Commons Lang).
  • Simple, no migration, no change for stored data.
  • Script-based languages keep being represented with a country (e.g. zh_CN / zh_TW, sr_RS), which is what we already do for translations.

B. Support locales with a script (BCP 47 style, e.g. zh-Hans, sr-Latn)

Consequences to evaluate:

  • Serialization: Java’s toString() form (zh__#Hans) is not a sane identifier. We’d need to switch to the language tag form (toLanguageTag() / Locale.forLanguageTag()) everywhere we serialize a locale: document locale stored in the database, references (Space.Page(zh_CN)), URLs (?language=), translation file names (ApplicationResources_zh_CN.properties), wiki preferences (supported languages, default language), user preferences, Solr fields, REST, filter/XAR export (<locale>). And keep reading the old zh_CN form forever. That’s potentially a large change with migrations.
  • Fallback chain: LocaleUtils.getParentLocale() and the translation lookup would need to know that zh_Hant_TW → zh_Hant → zh (or zh_TW?) and how it maps to the existing zh_CN/zh_TW translations from Weblate.
  • Duplicates in the UI: the language pickers would show af_ZA and af_ZA (Latin) etc. We’d probably need to filter to “meaningful” scripts only (zh, sr, bs, uz, az, pa…), which means maintaining a list.
  • Browser negotiation: browsers do send zh-Hans-CN in Accept-Language, so there could be a real benefit in matching them better.

C. A middle ground: keep A as the storage/identifier rule, but map incoming script locales (e.g. from Accept-Language) to our existing country-based ones (zh-Hans → zh_CN, zh-Hant → zh_TW).

And separately, what about extension locales (ja_JP_#u-ca-japanese, th_TH_#u-nu-thai)? I’d say they describe formatting preferences (calendar, numbering) rather than a language, so they shouldn’t be XWiki locales at all. WDYT?

My opinion

+1 for A (explicit rule + our own tests), possibly with C later if someone has a real need. I don’t see enough demand for B to justify the migration cost right now, but I’d be happy to be proven wrong, especially by people using Chinese or Serbian wikis.

WDYT?

Thanks

2

+1 for A.

Thanks,
Marius

3

+1 for A for now, mainly because it’s been XWiki’s behavior in practice for a long time without real complains. We can re-discuss supporting more if someone really have the need.

While the concept of script might make more sense than the country code in some cases, I doubt the migration really worth it right now.

We actually do rely on the Locale (or rather, Java does) to display properly dates, numbers etc. so it could make sense, but here again I doubt it really worth the complexity.

If there really are very common cases where the local sent by the browser is not properly interpreted, we probably need a jira issue about that.

4

+1 for A thanks

5

Conclusion: we go with A. XWiki locales are language[_COUNTRY[_variant]] only, and locales with a script or extensions are rejected. This is now an explicit, tested rule (XCOMMONS-3819: Upgrade to Apache Commons Lang 3.21.0 by vmassol · Pull Request #6617 · xwiki/xwiki-platform · GitHub) and no longer depends on Commons Lang’s behavior. We can discuss B again if someone has a real need. For C, we’ll create a JIRA issue if there’s a common case where the browser’s locale isn’t matched correctly.

Thanks all!