XObjects listing macro

Development
1

Hello all,

I’d like to propose the introduction of a new macro dedicated to the custom listing of XObjects by non-technical users.
By custom listing I mean the possibility to decide, for a given XClass:

  • The entries to display
  • The filtering criteria
  • The sorting

By scoping to non-technical users, this means the following constraints:

  • Use of non-technical vocabulary
  • Everything is achievable using WYSIWYG
  • Auto-suggestions for all the relevant fields
  • Non-technical error messages
  • Available to users with edit rights only
  • Available to non-advanced users

In short, the goal is to let anyone add listing in their documents freely. For instance, being able to add a listing of all the meetings a given user was involved in December, sorted by date. For each, only displaying the executive summary and the title (assuming a Meeting XClass exists in the wiki).

This macro would allow picking a XClass, its properties, and sorting and filtering criteria.
The rendering output a Live Data (LD).

The way the XClasses and XObjects were created is out of the scope of this proposal (including AWM, used only as an example). We have other work for this topic:

  • (future work) The ability to define the structure of a LD (ultimately impacting the structure of an XClass for the liveTable source)
  • (ongoing) The ability to add entries directly from LD UI (here, a new entry is an XObject in a new page in the case of the liveTable source).

Motivation

  • Currently, it is difficult for non-technical users to add listings of their structured data from the editor.
  • Creating a structure is decently easy with App Withing Minute (AWM). But, from there, adding a list of the data added in an AWM, with custom filtering and sorting, is not easily achievable.
  • Users can use the Live Data macro, but its generic nature (e.g., supports several sources) and lack of suggestions and error reporting makes it hard to use.
  • User can also use the documents (pretty name: Pages) macro, but it is limited to pages, and suffer from the same lack of auto-completion support, making it hard to configure for non-technical users
  • This is especially important in the context of users familiar with Confluence or Notion, where such listings are easily done.

Use cases

  • UC1:A non-technical user (e.g., a project manager) can insert a listing of XObject from an XClass they created (e.g., using AWM).
  • UC2: An author who does not know XWiki syntax configures that table entirely from the WYSIWYG macro dialog.
  • UC3: A reader of the page filters and sorts the table themselves, without editing anything.
  • UC4: An author restricts the table to one sub-page, and chooses whether nested pages are displayed too.
  • UC5: An author wants a plain list of pages with no data type at all. Equivalent to the {{documents}} macro. Already covered by the documents macro
  • UC6: An author reopens the macro months later to add a column, and finds their existing configuration rather than defaults.
  • UC7: (optional, probably best covered by a contrib extension) A Confluence migration turns a Page Properties Report into this macro, with the migrator creating the class. Covered by a dedicate extension
  • UC8: A property is renamed or removed from the class after the macro was configured, and the page still renders.
  • UC9: A reader without permission on some of the matching pages sees a table containing only what they may see.
  • UC10: A reader with edit rights is able to add entries to the table, and edit existing entries, in-place, if the author allowed it.

Specification

You can see more details of the proposed parameters and how they interact in the design page.

The main scenario to have in mind is:

  1. The author types /macroame in the editor (see naming below)
  2. A macro dialog opens
  3. The author is asked for a XClass (exact vocabulary to be defined)
  4. The author is asked for properties, sorting and filtering choices
  5. The author inserts the macro, a Live Data is displayed in the editor, with a visible content

Options

This section will describe a few options that all fulfil the same features, but with variations on the level of support provided to users.

Option 1 - No improvements

This is basically a rewrite of the Live Data (LD) macro, but with a specialized UI focused on XObjects listing.
No custom displayers, no conditional display of parameters.
With no specialized pickers, everything is free text and the author has to know what to input, and where to copy it from.

Option 1.1: No parameter validation, everything is passed to the LD, LD error messages are presented to the user

Option 1.2: Server side parameters validation

Pros:

  • Cheapest

Cons:

  • Poor usability

The Records macro dialog with empty Class, Properties and Location text fields.
The Records macro dialog with empty Class, Properties and Location text fields.750×812 30.8 KB

Option 2.1 - With custom displayers

The parameters are based on custom displayers. For instance, a XClass picker is used for the XClass selection, instead of asking for a string.
The author has suggestions and live validation at all steps.
Parameters that depend on each other are hidden or disabled until the required value is set. For instance, the sort is displayed until the XClass is selected.
Or, the sorting picker only suggests fields that are sortable.

Pros

  • Improved usability

Cons

  • Higher development and maintenance costs

See below two UI mockups. The first one while the author is picker an XClass, the second one once an XClass is picked.

The Records macro dialog with an XClass picker open, showing search results for “proj” listing the Client Project, Project review and Project risk applications.
The Records macro dialog with an XClass picker open, showing search results for “proj” listing the Client Project, Project review and Project risk applications.750×869 38.8 KB

The same dialog with Client Project selected, showing a column chooser grouped into page metadata and Client Project properties, with checkboxes and drag handles.
The same dialog with Client Project selected, showing a column chooser grouped into page metadata and Client Project properties, with checkboxes and drag handles.750×911 38.4 KB

Option 2.2 - With custom displayers and custom preview

Same are 2.1, but a preview of the LD that will be displayed once the macro is inserted is presented directly in the macro form.

Pro:

  • Quick usability win over 2.1

Cons:

  • The notion of macro preview is already something we want to work on, and introducing an ad-hoc implementation for this macro is wasteful unless we absolutely don’t want to wait

See below a mockup UI of what it could look like.

The Records macro dialog with the parameter form on the left and a preview pane on the right rendering the resulting Live Data table.
The Records macro dialog with the parameter form on the left and a preview pane on the right rendering the resulting Live Data table.1862×690 64.6 KB

Option 3 - In-place edition

Instead of displaying the usual macro modal, the macro has no parameters editable in the modal. Instead, the LD is tweaked to offer interactions that result in a change in parameters internally (and a refresh of the display), directly from the editor.
This is the closest to what Notion proposes.

Pro:

  • Seamless integration, zero back and forth to a macro dialog to tweak the display

Cons:

  • Experimental, likely the more expensive option
  • Probably also harder to generalize, meaning more custom development for each macro that would match this look & feel.

I would only pick this option only if we consider XObjects listing an absolutely primordial feature that deserves a special treatment.

See below two mockup UI of what the user interactions could look like. First before an XClass is piked, then once it is selected.

A wiki page in the editor containing an empty Records block that asks “What kind of data should this table list?” with an application search field and suggestions.
A wiki page in the editor containing an empty Records block that asks “What kind of data should this table list?” with an application search field and suggestions.747×757 40.5 KB

The same page with the table configured, a toolbar of parameter controls above it, and an “Add a column” menu open from the table header.
The same page with the table configured, a toolbar of parameter controls above it, and an “Add a column” menu open from the table header.747×522 33.7 KB

Questions

Q1 - Macro name

I’m proposing the naming below, but feel free to contribute to the vocabulary proposal, and we’ll adapt the naming accordingly.
The goal being to provide a name that is easy to discover and to remember.

Macro ID: records
Macro name: Records

Q2 - Options choice

Let me know you options choice.

Conclusion

  • Q1: +1 for records

  • Q2:

    • I’m -1 for Option 1 as it is not suited for non-technical users
    • I’m -1 for Option 3 as it feels too much of change in comparison to what we currently do
    • I’m -0 for Option 2.2 as I think the macro preview needs more design before implementation, and an ad-hoc supports does not feel ok (I would only go for that option if we absolutely want the preview and a generic implementation cannot fit the 18.x cycle)
    • I’m +1 for Option 2.1 that is I believe already a significant usability improvement for non-technical users, thanks for the improved ease of use with suggestions and error reporting.

cc @lucaa @caubin

References

2

It’s very easy and it’s even done automatically by the AWM app, on the home page of the app.

Unless the focus is not so much the listing but the custom filtering (what does it mean?). Not sure it’s so hard by editing the generated AWM home page (or we could make that page simpler to understand with comments for ex, in edit mode).

I think this is already covered by the AWM app itself which creates such a listing on the home page of the app by default.

Or do you mean doing this even when no AWM has been created? If so, then the user has to be a developer or an advanced user since they’ve been able to create an XClass and XObjects…

I don’t want to nit pick but I fail to really understand the motivation aspect based on what’s written. I think it’s probably just that the motivation section is missing explaining the real issues.

For example, if the idea is to add such listings on any page the user wants to, then suddenly AWM doesn’t work anymore. For non-tech users, AWM should still be used to create the structure (XClass) but the macro you propose can then be used to display instances of that structure (XObjects). BTW the AWM-generated home page should be modified to use the new macro.

Note that this is already achieved in the AWM-generated home page. The key difference is if you need to do this outside of the generated AWM app.

I don’t understand this UC.

Note that this competes with AWM in which you can edit the app to either change the structure or change the columns displayed on the generated home page.

What we’ve started doing is to move the AWM capability inplace, ie at the location where the need arises, ie. on any page the user wants to add a listing of xobjects. This needs to be stated IMO in the motivation area.

Today we’re doing 2 things in this direction:

  • LD edit mode to add/remove/modify rows being xobjects
  • this macro

And in the future we’ll also add:

  • Ability to create directly an XClass from a LD

Once we have these 3 features, we’ll probably be able to replace almost full the current AWM app. We’ll then need to decide what we do with AWM. One option will be to not install it by default in XS. OTOH, we will need to still find a way to make these features discoverable.

I need to go but I’ll review the proposed option a bit later.

Thx for working on this. My comment above is just some observations to try to make the motivation better explained and to position this proposal more globally in the replacement of AWM direction.

3

Hi Manuel, thanks for all the work on this topic.

I liked the original options on the referenced topic. Record is not bad I just find it too generic.

IMO this goes against * UC1:A non-technical user (e.g., a project manager) can insert a listing of XObject from an XClass they created (e.g., using AWM).

This one is better, and if we need this implemented sooner rather than later this is the option I’d go for.

Even better but I would make the preview generic and work with all macros. Could be a further improvement after 2.1

For me an ideal scenario would be a mix of options 2 and 3. But ofc the development cost would be very high so it might not be realistic.

  • Creating a basic table is done on the page itself. Here we can add and remove columns, sort, filter using LD.
  • Modal for advanced stuff and a comprehensive view of all the options. Yes, that means we would have two ways of achieving the same results.

In short, 2.1 and 2.2 are a good middle ground between ease of use and cost of development.

Thanks!

4

yes, I’ve improved a bit the start of the proposal, I hope it’s clearer.

I think we need to aim for non-advanced users (i.e., with no access to the wiki edition mode), and using WYSIWYG only.
Also, a point I updated in the introduction, is that I want to let any user add their custom listings in their documents.
Having a single central listing in the main page of the AWM is nice, but is not enough.

Same here, user must be able to define custom listings.

Yes, the proposal has nothing to do with AWM. The listing of any XClass should be possible.
I only used AWM as an example here and there because it is what’s available currently.

AWM is on way to created XClass, the XClass editor is another one. Both are imo not suited for non-technical users. We have plans to address this issues, but this is out of the scope of the current proposal.

Yes, and this capability is already available for LD in 18.7.0. The point of this UC is to make sure that it is not lost in this new macro.

Thanks for all the feedback :slight_smile:

I have a hard time find a right name for the macro. The naming needs to convey the notion of listing (be it as a table, or a set of cards, or any other layout).
An alternative name could be “Entries” that would match the vocabulary proposal, but as a user that’s not what I would type in the quick action.

5

Users would propably start by searching the page itself since that’s what they already know and worked with. Searching /meeting-notes intead of /macro-name for example, but I don’t know if that’s feasible for us.

6

Thanks a lot for working on this :folded_hands:

For Q1: I agree records is too generic, at the very least I would expect users to look for a macro that has “data” in its name but I can’t find a good fit… “Data collection” is the best I can come up with but it still sounds weird.
As a nitpick: looking at your mockups, I’m not sure the UI should mention “pages”? The only reason we store entries in pages is for technical reasons, but what the user should be interested in is the structured data instead of the container.

For Q2: +1 for Option 2.1, it would be overkill to start working on macro preview before we have the generic design for it. -1 for Option 1, and +0 for Option 3 (I like the UX a lot but that seems extremely hard to maintain).

7

That’s an interesting idea, but I don’t think that’s possible currently. Also, I fear it could clutter the quick actions with too many entries.
I think we should stick to a generic entry in the quick actions, then an easy to use XClass picker at the next step.

Thanks for the idea. Collection sounds interesting but feels quite broad as well.
In all cases, reminder that the quick actions and macro search both rely on the macro name and its description.
So we can cheat a bit by doing something like the example below, hitting several usual keywords at the same time.

  • Macro name: Records
  • Macro description: The records macro displays a collection of entries of the same kind.

Don’t look too closely at the mockup. They are here to give a general feeling of what each approach would look like. But, you’ll quickly realize that the details are inconsistent.
I wanted this discussion to converge to a collectively agreed one approach before investing more time in the details.
In this option 2.1 is the likely agreement, and I’m going to continue in this direction.

8

Status post.
I opened a draft PR: XWIKI-24831: Implement an xobjects display macro by manuelleduc · Pull Request #6374 · xwiki/xwiki-platform · GitHub

The PR contains several screenshots of the proposed UI.
It is not a final implementation, and many aspects need to be improved incrementally, but I think having a first usable version shipped in the product is important.
What I focused on is the stability of the serialization form. As long as what is persisted is stable, it is “easy” to improve the UI.

I’d also like to extract (in a second step) the custom displayers in a reusable location.
The XClass picker also need to be improved but we have a separate discussion for it.

Look forward to you feedbaks. Thanks

9

Hello @mleduc ,

finally got around to this, I have a couple of comments based on the proposals that were made here (unfortunately, I didn’t check the result in the PR to compare to what was actually implemented).

Naming:
I believe I can llive with records, but I think there is another discussion that we need to have here. The macro is called XObjects and the specs / proposal is talking about displaying XObjects. I didn’t check the implementation, but I suppose the objective, as for the home of the app within minutes, is to display pages that contain at least one object of the specified type. This means that if a page contains 2 objects of the chosen type, there will always be a single row in the table displayed by this macro for that page (with the data of the first object, I suppose).
At least this is how I see the usage of this macro. The display of multiple objects from a page is something that could be added, but later, as a bonus.
Because of this, the macro could also be called “pages” or “documents”, or something similar to the existing macro that lists pages - the one you mentioned on UC5.

As a nitpick: looking at your mockups, I’m not sure the UI should mention “pages”? The only reason we store entries in pages is for technical reasons, but what the user should be interested in is the structured data instead of the container.

actually, @pjeanjean , no, not really, the storage is not only technical. The classes and objects are a method of adding an extra layer of data on a page: a standard page has title, content and attachments list as “properties” (places where data can be stored), a class/object will allow you to add some additional things to those 3 properties, but they shouldn’t be seen as “replacing” the page. Even if we would store the objects differently, it would still be a good idea that that layer of extra data is “tied” to a page, added along with a page, like an “embellishment”, for the things it “inherits” from pages: access rights, history, authorship metadata, placement on the content tree of the wiki, the possibility to store a large rich text (the content), a smaller text that works as an identifier in listings (title) and associated attachments.
To me, I think we’re complicating too much the concept space of XWiki if we’re trying to hide, for all applications, that the entries are pages.
Indeed, some applications may have very specialized objects, for which we don’t want to promote necessarily this embellishment model (a base which is a page + an extra layer), such as diagrams, for example, or tasks. For these needs, I think the applications in question should / could also provide a specialized macro, that would hide away the concept of “page with extra things”.

@mleduc this made me think that we have a filter in the default search for type of object present in a page, and we should ensure to align the vocabulary we use here for the selected class with the vocabulary used there for that facet (as they’re basically select the same pages).

Whether the displayed livedata is editable or not should be a parameter of the macro (I tried to read the design page to understand whether it’s the case, it’s not clear to me).

For the edit options (Q2), 2.1 is perfectly fine, I am -1 for 3, for now at least, and 2.2 should be handled by a proposal that is larger than only this macro. Of course -1 for option 1, that can be a first version, but not the final one.

I know it’s late for these, but for the record.

Anca

10

Thanks a lot for the answer @lucaa
My turn to apologize for answering back with a delay.

I get you point, sadly a documents macro already exists with the pretty name Pages and the description Displays a list of pages in a Livetable.
It is defined as a wiki macro in XWiki.DocumentsMacro.

I reconsidered building on top of this existing macro instead of introducing the new records macro.

  1. The documents macro is defined in a wiki page and I think we’d need to migrate it to Java, while making sure to preserve backward compatibility, which is risky in itself, let alone when introducing the new parameters needs for the proposed improvement
  2. There is also a small risk that XWiki.DocumentsMacro was customized, and that would be lost in the upgrade. Though it is possible to make the wiki macro higher priority, but in this case the instance wouldn’t benefit from the improvements
  3. Overall I feel like it would make the UI more complex as for instance the actions checkbox from the documents macro wouldn’t make much sense for the records macro.

So even it the documents and records macros are sharing some concepts, I think it is better to keep them separated, both in terms of development costs and in terms of UX.

Regarding the listing of one or many XObjects from the same page. It is a known limitation (Loading...) of the Live Table source (and of LT more generally).
The way we address this limitation is open to discussion but conceptually, I still find it interesting to list XObjects as independent entities help by pages.

Thanks for pointing this out. I aligned the PR vocabulary accordingly.

Agreed, objects are contained in pages, so exposing the concept of pages is useful to users.

While it is currently not part of the PR, to keep it somewhat at a reviewable size, this is definitely something I plan to add. Either as a separate commit with the same Jira ID if time allows, or as an improvement in a future version.

11

I’m not sure I understand this one. Do you mean that if you change a page from the standard distribution there is a risk that there are customizations of that page in XWiki deployments and so some sort of 3 way merge (at least at a logical level) will need to be done by the XWiki users that made such deployments ? (choosing between their customizations and the changes in the standard distribution). If this is indeed the concern, then I believe this is a non-problem, if we’re asking ourselves such questions then we never make any change in XWiki Standard anymore as this can be true of any page.

hmmm, there is a little contradiction between this statement and the agreement you expressed below about my answer to @pjeanjean where I explained that an application entry (record) is an embellished page. If a record is an embelished page (and this is what I believe it is / should be, maybe this is the part that was not clear enough), than all page actions make sense for records as well.

It’s ok that they stay separate, it’s not that bad. However, I would prefer that we do agree that records are documents / pages (“embelished” ones).

To me this is not a limitation, and issue XWIKI-23074 is an invalid bug.

Maybe it was not fully conceptualized as such in the livetable documentation (and livedata from livetable source), but, to me, the default source (results page) of that macro / method has always displayed a set of pages from the wiki - at most one entry per page of the wiki. The parameters (classname, etc.) are supposed to filter the displayed pages (“from all the pages of the wiki only list the ones with this object”) but they’re still displaying pages, not other types of entries.
If other types of entries are expected, custom source / results page needs to be built.

I should add this as a comment on the ticket.

So, to me there is not much that is open to discussion here, whenever some listing is promised by a feature we need to describe clearly what are the results. For “records” macro (the current one discussed here), as for livedata / livetable, the results should be pages (at most one entry in the table per page in the wiki). The class selection from the macro’s parameters fills 2 functions:

  • filters which pages would be displayed in the table (only those containing at least one object of that class) - together with the location filter and other filtering parameters,
  • describes how to interpret the names from the columns parameter: as properties of the named class. If more than one object of this class is present on such a page, the first one will be used.

Should listing of any other type of data be needed (there are, of course, valid usecases for other listings), the records macro is not an answer to those, a carefully crafted livetable or livedata is the solution for those cases.

Hope this helps,
Anca