Add a CSV export to Live Data in XWiki Standard

Development
#1

As already explained in a separate thread, I’m currently working on implementing a CSV export for Live Data (see there for technical details). At least the first version will be quite basic and most likely not include custom handling for fields of different types, in particular fields where the displayed data is HTML. Future versions might improve this and also introduce different export file formats.

What I’m asking here is: do we want this export to be in XWiki Standard as a feature that is always present and enabled in Live Data?

Having the export in XWiki Standard would mean that I could write a nice integration in the menu of Live Data and add an export panel similar to the advanced filters where the progress is displayed and the download link is available. Further, the feature would always be available and thus easily discoverable.

If the export feature should (at least initially) not be in XWiki-Platform, I would create a contrib extension. Unfortunately, we have currently no concept how Vue.js components can be extended in extensions and therefore, at least in a first version, there will be no integration in the Live Data UI. As I’m not familiar with Vue.js, I unfortunately also don’t see how I can create such an integration possibility for now. Providing such an extension possibility would also mean that the current Vue.js components become an API and an upgrade to Vue.js 3 will be more difficult if at all possible. Instead, the exporter I would develop would provide a separate macro that produces a button that triggers the export. From a short investigation, it seems possible to get the currently configured filters from LiveData by directly accessing the Vue.js component, so this won’t require any changes but will depend on APIs that should probably be considered internal.

New contrib extension - LiveData Exporter Application
#2

For me, we need a way to have extensions provide LD Export extensibility since it’s not the goal/role of XWiki Standard to offer any imaginable LD export. Thus anything we do that is not extensible is a technical debt added to the platform and will require double work since it’ll require refactoring in the future.

Regarding CSV export of LD as a core feature, I don’t think I would put it as a core feature because:

  1. you can correctly use xwiki without this feature and it seems like a feature not requested that much (at least I don’t recall seeing lots of user requests for it and for all the wikis I managed, I’ve never had the need so it seems a bit specific). Users who need this can use the EM to install it.

  2. we really need/want to have the smallest possible XWiki that can be used.

  3. it’s a way to ensure that extensibility is provided and is working for LD exports.

Thanks

PS: Re Vue.js, maybe it would be good to do a brainstorming with Manuel since he introduced it. Note that the extension point should probably be an xwiki one and not a Vue.js one so that it’s independent of Vue.js and users of it aren’t impacted if we changed the underlying js framework (or upgrade it).

#3

PS: my reply is a generic reply for the direction we should take. I don’t know:
a) the cost of each option
b) the time you have at hand

If we were to go for the shortcut (only at last resort IMO), we/you’d need to plan for some time later the refactoring of it.

#4

For me, CSV format is generic enough to be provided by XWiki Standard by default.

But I agree that we also ideally need to make as easy as possible for extensions to provide more export formats.

#5

Regarding the cost and the time, I assume that the contrib extension as outlined above (i.e., without proper integration) will be faster as it most likely won’t be as polished and requires less understanding of Vue.js. I fear I won’t have the time to develop a proper extension API for LiveData. I can imagine that it might be easier to just offer a way for extensions to provide a list of formats (in the form id/display string) and then provide a way via the lookup of a component with that identifier to export to that file format, and thus making the export itself extensible. The contrib extension would most likely duplicate some code of XWiki as I need to do the same as some of the internal APIs and thus probably introduce more technical debt.

I would be very much in favor of this but I fear this won’t be (at least easily) possible if we want to really use Vue.js. However, I don’t know what kind of possibilities Vue.js provides, so maybe it is possible.

#6

For me it’s not related to CSV format being generic enough but about the need to do a LD export without having to install any extension. I find that an advanced use case.

#7

I’m perfectly fine with the concept of export being a standard feature of the LD.

#8

ok then I think we need to have a brainstorming about what XWiki Standard is and what we consider “minimal”. We need to refine our definition. I see no problem to ask users to install an extension for such an edge case and not having it part of XS by default.

#9

Note that we could do as you’re doing for Macros to make the extensions discoverable by listing all available LD exporters available on e.x.o and propose to install them.

#10

yes me too but this is not about the concept of export but about an implementation of export by default.

#11

Exactly my opinion too.

For me being able to export your data from a wiki in CSV is not an advanced UC at all, and can be quite useful in many situations at the very least for interoperability operations. I find it more standard than allowing to export to office format for example.

Now it feels like one possibility for @MichaelHamann could be to work on that in xwiki-platform but not bundle it in XS by default, at least until it’s mature: that way it’s possible to have a first working version not polished to be installed as an extension, and then to improve it to bundle it.

#12

I think you guys are forgetting a bit that our goal is to make XS the smallest possible and use extensions to add features. So what we need to discuss is not so much whether we want to bundle the CSV export of not in XS but more, to agree again on what’s the scope of XS and define boundaries. We can also talk (again - as we listed some modules quite a while ago) about what we need to extract out of XS.

EDIT: Another way of viewing it would be to answer: “if we had to remove stuff from XS, what would be the ordered list from more obvious to remove to less obvious?” This allows comparing them together, for example LT CSV export vs XAR export, or LT CSV export vs Invitation Application or…

#13

This sounds logical at first glance, but unfortunately, it poses exactly the same extensibility problems for the UI as the contrib extension - how do we integrate this in the LD Vue.js UI? So from a technical point of view this doesn’t simplify much apart from the fact that it wouldn’t be so bad to let this depend on Vue.js 2.

#14

Hello all,

dunno how much this helps (or if you had a look at it) but there has been work in the past for the same feature for the livetables:

https://extensions.xwiki.org/xwiki/bin/view/Extension/Livetable%20Exporter%20Macro/

Things that may be interesting to learn:

  • injection methods for extensions (if there’s anything that could apply the same).
    Now, that extension is not the most ‘injected’ in the world, it kinda duplicates some standard code (less and less though)
  • from my experience with working on that feature (export of a livetable of data), there is a whole bunch of code that ends up in it, in the end, especially regarding 2 topics:
    • what to export, in terms of displayed data vs. stored data. Displayed data is interesting from a wysiwyg pov, but it needs special handling for html data. Stored data is interesting from a data transport pov (export from a wiki / application to import in another wiki / app)
    • performance & scalability: while the table itself is paginated, the export would not be and the UX expectation is that it works for any sizes of data. This means async job for the export which may need some extra code.

Note: the livetable export extension is still not complete from this pov, for example it doesn’t export stored data but it would be highly needed.

#15

Thank you for the suggestion, I had a short look already but I might have a closer look now again as while I originally thought the UI would be different (as I assumed it would be a Vue.js component), it might not be that different in the end. I’ve already started working on some backend code for the export which is based on a new REST API that a) creates the export job and b) provides the download once the job has finished. I’ve already implemented the export as a job that uses pagination to get all entries of the data source so I hope I’ve already covered most of the performance & scalability part.

What might be interesting is the job status UI part as I’m surprised to see that there seems to be no generic way to display the status of some job that is updated as the job is executed and calls some JavaScript callback/triggers an event once the job finishes.

Regarding the “what to export”, I fear this completely depends on the Live Data source. We could probably implement something custom for the Livetable source, but a generic source will just provide some values to display that might be HTML and there is - afaik - no generic way for the source to provide the stored data for export. We have some way to indicate that the displayed value is an XObject or document property and this could probably be used to load the original data from the document in the exporter. Otherwise, we could try to transform the HTML into some plain text that might be more suitable for the chosen data format (like provide just the displayed text). In my concrete use case, I’m developing a custom source where I would say the Live Data is more a preview for the export and a nice way to add filters and sorting options and thus - at least at the moment - I don’t see the need to support, for example, HTML. So from my point of view, this would be a second step that wouldn’t be in a first version of the export. Therefore, I can fully understand the point of view that this should be a contrib extension at least until we have better support for what to export in the case of more complex values.

#16

I think the job UI code from the livetable exporter macro was coded to be compatible with XWiki 9.11.x , so the fact that the livetable exporter is re-coding all that job UI manipulation thing may not be that worrying, maybe there’s better in more recent versions…

#17

Let me follow up on this with a couple of options how this Live Data export feature could be integrated into XWiki:

  1. Similar to the Livetable exporter, with a separate macro that provides the button. No changes necessary to XWiki.
  2. Introduce a JavaScript extension API in Live Data that allows to register a new menu item and a new panel. This would basically be similar to a UIXP but in JavaScript. The export feature wouldn’t use Vue.js.
  3. Introduce a Vue.js extension API in Live Data that allows to register new menu items and new panels. This is similar to option 2, but means that the extension needs to use Vue.js. Further, this complicates upgrading Vue.js to version 3. Having the exporter in the xwiki-platform repository could help with this, as we could make sure that we upgrade it at the same time.
  4. Integrate the export feature into Live Data, but let extensions contribute the actual export formats. The UI would only be shown if there are actually export formats available. We could use a mechanism similar to macro parameters or job questions to display a UI to configure the exporter - or let the exporter job use a question at the start to collect the parameters from the user.
  5. The same as 4, but also provide a CSV format by default.

I’m currently tending towards option 1 as a first version as it seems the easiest option where we don’t introduce new APIs that turn out to be badly designed and/or hard to maintain. If we implement a proper (automatic) integration, we could always deprecate the macro and/or disable its output.

I think it would be cool to have the extensibility features of 2 or 3 in general, as we seem to have very little structure how user interfaces can be extended on the client side but I fear this requires more thoughts and time than what I can provide here. I fear if we do something like this quickly now, we might get a ton of different ways the UI can be extended in the end.

Option 4 seems kind of easier at first, but getting the options API right might also be tricky, in particular if at first there won’t be any options, so this might be a good idea once we have something that actually requires options. Option 5 seems controversial and would have the same problems as option 4.

Any other opinions or options I’ve missed?

#18

Option 4 (and 5) is interesting but it’s more limited than a generic way to add a new “menu entry” in the LD UI (since it would be purely related to exporting while a menu entry can be about anything). I think I’d prefer option 3 (since the LD is written in Vue.js, it seems logical that extensions of it would require Vue.js too).

I don’t like option 1 too much, for the following reasons:

  • It doesn’t solve the full need. What about doing an export of say, the Page Index of the wiki? That would require the user to edit the content of this XS-provided page which is not good. So it would work only for user-provided LT.
  • I assume it would work by executing some JS that would insert some HTML in the LD-generated HTML and thus would plug onto some non-API that can be modified.

I’m really bad with JS and don’t know much but what would be the problem of having the LD JS code send an event when it’s ready for extensions to add menu entries and then have extensions simply listen to this event and then insert a menu entry under some HTML element id or class attribute (which would become an “API” in the sense that we wouldn’t break them)? Maybe the event can even send the id or class value as event data, idk. Also since events are part of standard JS, wouldn’t this work if the extension is written in pure JS or Vue.js?

Thanks

#19

No, the HTML code would be outside the Live Data UI, this would simply be a macro that inserts a button for the export in its own content, similar to how this is done by the Livetable exporter. It would only interact with the Live Data logic module to get the current state of the filter and sorting options, which is indeed a bit tricky to access due to XWIKI-19635.

Given your feedback, I’ll further investigate option 2 as to me it seems like the next best option. I was a bit worried about this option that it might be difficult to know when the HTML element is ready (or that the HTML element might be removed/added several times) as the rendering of the template is done by internal Vue.js code but there seem to be lifecycle hooks that can be used to be notified when a component is ready and in general it seems that the DOM isn’t completely replaced but only small parts are updated. You’re probably right that it would be possible to use Vue.js in the extension again, but this would then be completely independent of the Vue.js in XWiki itself I think.