Components in the extension index

[tmortagne]

Hi devs,

Before I start documenting that in a more official fashion here is a summary of a new extension index feature I just committed in master.

The extension API and the extension index now contains metadata related to components you can find in an extension. This allows various uses cases generally related to discoverability (list remote macros right in the WYSIWYG macro picker, propose some extensions to install in the missing macro error, etc.).

The metadata is pretty simple: an Extension expose a list of role type/hint couples provided by the extension.

The extension index have dynamic fields to associate a role type to a list of hints. It also has a (currently hardcoded) list of aliases for some well known components (just added a few and more will come before 13.3 final to have at least a set of very common user oriented components provided by the contrib extensions) allowing to put things like component_macro:map directly in the search input of the EM UI.

Since we need this metadata to be part of the extension descriptor (we don’t want to download the extension file just to know if its worth downloading the extension) and that this new feature would be unusable without a way for contrib extensions to start listing components they want to make public right away I came up with the following to start playing with it: XWiki now understand a new custom property that you can define through a Maven <xwiki.extension.components> property where you list components role/hint couples. For example in the pom.xml file of the Map Macro contrib extension you would have:

<propertie>
  <xwiki.extension.component>
    org.xwiki.rendering.macro.Macro/map
  </xwiki.extension.components>
</properties>

Extensions located on https://extensions.xwiki.org but which are not built with Maven can also set the same property through the field Properties in the xobject ExtensionCode.ExtensionClass.

[surli]

@tmortagne

The metadata is pretty simple: an Extension expose a list of role type/hint couples provided by the extension.

Just to be sure I understand: it should expose all components provided by the extension, even those for which the extension itself provide the role interface?
Or it’s more a case by case basis where we decide which component we want to expose?
Basically the question is: what should be the best practice for documenting those?

[tmortagne]

@surli

Basically the question is: what should be the best practice for documenting those?

On the short term at least the minimum (not sure if we can talk about best practice at this stage) I have in mind is to expose public roles implementations an extension has been written to provide. Exposing some internal component that nobody is supposed to use does not really have any value.
It does not mean it should only be things related to XWiki Standard extension points, it of course also make sense to expose implementations of public component roles introduced by some extension. Having the extension itself expose implementations of this public role it introduces is probably often of not much value, but it does not arm, and it can be interesting in some cases (I can think of things like reviewing new components provided by a new version of the extension).

[mleduc]

I must say this is a cool new feature

Does this change have other impacts on the UI (among providing new search keywords)?
For instance, the list of provided components listed in the extensions UI?
Or the other way around, this list of extensions providing component for a given role.

[tmortagne]

@mleduc

Does this change have other impacts on the UI (among providing new search keywords)?

On the UI not yet. This is the foundation of two new features which are in the 13.x XWiki roadmap:

• easily find possible extensions to resolve the error you get when a page use a not existing macro
• integrate “remote” macros in the WYSIWYG macro picker

@mleduc

For instance, the list of provided components listed in the extensions UI?

Was a bit short for 13.3RC1 which is planned on Monday but yes I plan to add the list of components in the extension details UI before 13.3 final (so next week).

@mleduc

Or the other way around, this list of extensions providing component for a given role.

That’s something you can do with the search field, but maybe you are talking about a helper in the EM UI to generate advanced search queries ? This is something that could be nice, but it’s not in my TODO list currently.

[vmassol]

Sounds good, thanks!

@surli

Just to be sure I understand: it should expose all components provided by the extension, even those for which the extension itself provide the role interface?

yes ideally it should expose all (for roles that are not in internal packages ofc) since they’re public stuff.

@tmortagne

Since we need this metadata to be part of the extension descriptor (we don’t want to download the extension file just to know if its worth downloading the extension) and that this new feature would be unusable without a way for contrib extensions to start listing components they want to make public right away I came up with the following to start playing with it: XWiki now understand a new custom property that you can define through a Maven <xwiki.extension.components> property where you list components role/hint couples. For example in the pom.xml file of the Map Macro contrib extension you would have:

Couldn’t this be part of the XAR format and then have the XAR maven plugin automatically extract all components (by reading components.txt for example and then doing some checks to verify that the implemented role is public) and fill the XAR metadata with it?

If we want this to work well we shouldn’t rely on the extension developer to provide it (or forget to do so, or forget to list some components) so it would be nice to generate it automatically.

WDYT?

[tmortagne]

@vmassol

Couldn’t this be part of the XAR format and then have the XAR maven plugin automatically extract all components (by reading components.txt for example and then doing some checks to verify that the implemented role is public) and fill the XAR metadata with it?

You seem to be mixing very different things here. components.txt is not a XAR thing and there is no generic way to find components in a XAR.
Anyway as I said the first step was to have a way for any contrib extensions to be able to expose components they want to make public without a big refactoring of their build. Helpers to find components to expose in specific use cases will probably come later.

[vmassol]

@tmortagne

You seem to be mixing very different things here. components.txt is not a XAR thing and there is no generic way to find components in a XAR.

I’m talking build time.

[tmortagne]

@vmassol

I’m talking build time.

I know, but I still don’t see the relationship between the XAR plugin and components.txt.

[vmassol]

@tmortagne

I know, but I still don’t see the relationship between the XAR plugin and components.txt.

Ah right, indeed, good point, I mixed stuff!

So let me rephrase what I meant:

• I meant, have a maven plugin that we execute in JAR modules, and which will generate the metadata and put them in some META-INF file in the JAR, instead of having the dev manually modifying his pom.xml file.

[tmortagne]

@vmassol

them in some META-INF file in the JAR

But this means that you have to download the JAR file to get this information which is not great from extension index point of view. As I said ideally we want to have those metadata in the descriptor.

[mleduc]

Couldn’t these JAR metadatas be imported in exo (as we do currently with pom properties currently)?

[vmassol]

@tmortagne

But this means that you have to download the JAR file to get this information which is not great from extension index point of view. As I said ideally we want to have those metadata in the descriptor.

Indeed, but isn’t that needed anyway? Either it’s done at build time or at indexing time. In any case this indexing will need to extract the metadata and if the jar is stored on nexus, there’s no way to add metadata to it there except inside the JAR, right?

[tmortagne]

Guys please keep in mind that we need a short term solution to let contrib extensions easily expose any components they want to make public whatever kind of extension it is.

Discussing how to implement helpers for each type of extension that can help find the list of components is definitely interresting but something to discuss on case by case and in their own threads IMO. Because of the scale of this taks (don’t forget that there is no obvious way to figure out which components a XAR extensions contains without knowing all the possible kinds of bridges used for all kinds of wiki components) it’s not in my immediate TODO, and it should really not be a blocker before we start using this new concept.

@vmassol

In any case this indexing will need to extract the metadata and if the jar is stored on nexus, there’s no way to add metadata to it there except inside the JAR, right?

Finding the components in old released extensions can definitely only be done by analyzing the extension file. But the ideal location to read generic extension metadata for all the tools involved in the management of available extensions (EM and its API, the extension index, the various repositories, etc.) is the extension descriptor and that’s something I would really like to keep as long as we technically can. On the build side you can perfectly imagine tools which help update the descriptor (the pom in the case of Maven) and check if the descriptor is missing things (for example we do that a lot with the XAR plugin for other source files than the pom, but it’s the same logic). And even if for some reason using the pom.xml would really become too painful you still don’t have to put everything in the main file, for example with Maven you can very easily generate another metadata file (a .xed file for example) during the build and install/deploy it along with the pom and the main files that the XWiki Maven repository handler could download with (or even instead of the pom in the case of a .xed).

@mleduc

Couldn’t these JAR metadatas be imported in exo (as we do currently with pom properties currently)?

Right now this kind of modification to exo is not easy since it’s located in Java in a code which is located in xwiki-platform so generally require an upgrade of xwiki.org to be clean. Also, I would really like that we keep trying to be able to express in a plain Maven descriptor all the static metadta you can find in an exo extension descriptor as much as we can (and in general make sure they are in sync).