Export pages in Markdown format

Dave.Jarvis · September 27, 2017, 5:40pm

What would it take to export a collection of pages in markdown format (e.g., CommonMark)? Imagine the following use case (see attached architecture):

User selects pages to export within XWiki.
User presses a button to generate a PDF (not an XWiki-generated PDF, see below).
XWiki exports selected pages in Markdown format.
System passes pages to pandoc.
Pandoc creates a TeX file.
ConTeXt reads the TeX file and applies styling to generate a high-quality (corporate- or government-branded) PDF.

What is involved in step 3? For example, if XWiki cannot export in Markdown format, then I’d have to: (1) write a Markdown exporter; (2) write an XSL template to transform XWiki XML into Markdown; or (3) first export to HTML and use pandoc to create a Markdown document (encumbered with unknown issues most likely) so as to re-use the Markdown process flow.

Many thanks,
Dave

P.S.
Exporting pages as PDF via XWiki is not an option for a number of technical and architectural requirements reasons.

P.P.S.
As XWiki uses flexmark, in theory it supports CommonMark and syntaxes not listed on:

vmassol · October 2, 2017, 12:10pm

So what exists currently is the ability to convert any page into “Markdown” syntax (CommonMark++).

For example: http://www.xwiki.org/xwiki/bin/get/Documentation/UserGuide/Features/PageEditing?outputSyntax=markdown

What’s lacking, is the UI to export a set of pages and the exporter to package that in a bundle (ZIP file for example).

We do have the architecture to support plugging-in an exporter through the Filter Stream Application: http://extensions.xwiki.org/xwiki/bin/view/Extension/Filter%20Application

See http://extensions.xwiki.org/xwiki/bin/view/Extension/Filter%20Module for the general architecture and a list of existing Filters.

As you can see, Filters can be input filters or output filters. We already have an input filters taking as input wiki pages (see http://extensions.xwiki.org/xwiki/bin/view/Extension/Filter%20XWiki%20Instance%20module#HInputFilterStream). I don’t know if it can take a discrete list of pages as input or all the pages of the wiki, but if not, it can certainly be easily extended to do that.

So you’d need to write an export Filter and in that filter you’d use the XWiki Rendering engine to convert any page’s content into Markdown. That’s the easy part, see http://rendering.xwiki.org/xwiki/bin/view/Main/GettingStarted#HExamples

You could check how those existing filters are written if you want.

Let me know if you’re up for this. It shouldn’t be too hard but there are some concepts to understand I’m willing to help you out on those. The best would be for you to jump on Matrix/IRC (see http://dev.xwiki.org/xwiki/bin/view/Community/Chat) and we can help you with it!

Indeed that’s a good point. I’ll update the list. The real list is at http://rendering.xwiki.org/xwiki/bin/view/Main/WebHome#HSupportedSyntaxes

Thanks!

vmassol · October 2, 2017, 12:34pm

Done, see Loading...

vmassol · October 2, 2017, 3:12pm

Note that I’ve checked and the wiki instance input filter does accept a list of pages (there’s an API for that). So if all you want is to write a script that converts a set of pages to markdown and save them as a zip, it’s quite easy to do.

Dave.Jarvis · October 2, 2017, 5:27pm

Not published on the wiki?

syntaxes

vmassol · October 2, 2017, 7:10pm

What I fixed is XWiki on master in git (ie it’ll be in the upcoming 9.9RC1 release), not xwiki.org!

xwiki.org is running XWiki 8.4.4, which is why you don’t see it.

Dave.Jarvis · January 4, 2018, 6:43pm

For example: http://www.xwiki.org/xwiki/bin/get/Documentation/UserGuide/Features/PageEditing?outputSyntax=markdown

I tried this, but in XWiki 9.10.1 the output remained the same. That is, no markdown appeared – it was as though the “outputSyntax=markdown” parameter was ignored.

That’s the easy part, see http://rendering.xwiki.org/xwiki/bin/view/Main/GettingStarted#HExamples

Thanks for this. Unfortunately, the entry point is not defined thus the example code is incomplete. Examples showing the entire class, including import statements, would be quite useful.

Also, I’m not clear on what I’m trying to write: a macro, a component, a filter, an XWiki application, or an WikiModel implementation?

I noticed that there is tight integration with Maven. Is this absolutely required, or is it possible to set up a plain vanilla Java project with the requisite libraries imported manually? If a vanilla Java project is possible, where are the instructions?

On the Getting Started page, consider making the reference to xwiki-commons-component-default an additional bullet so that it doesn’t get lost in the paragraph of text that precedes it.

vmassol · January 4, 2018, 7:08pm

You have installed the markdown extension right? (http://extensions.xwiki.org/xwiki/bin/view/Extension/Markdown%20Syntax%201.2 )

It’s actually there. If you go to http://rendering.xwiki.org/xwiki/bin/view/Main/GettingStarted#HExamples this first paragraph says "These examples should work with XWiki Rendering 4.1 and beyond (you can download them here). " and there’s a link on “here”…

Again if you read this page http://rendering.xwiki.org/xwiki/bin/view/Main/GettingStarted you’ll see that it starts by explaining how to make it work without Maven by showing the classpath you need

I don’t understand what you mean. Could you be more specific? Note that this is a wiki so you can edit the page to improve it if you think it’s missing something.

Nevertheless I’d fix it if I knew what you meant

Dave.Jarvis · January 4, 2018, 8:22pm

You have installed the markdown extension right?

Doesn’t look like it. How do you install extensions manually using an offline installation? (The wiki does not have Internet access.) I tried uploading the JAR via the “Global Administration: Extensions” page using Content/Import (XWiki 9.10.1), but that looks like it only accepts xar files for import.

My guess is that I have to copy the JAR file into /opt/payara/glassfish/domains/isd/applications/xwiki/WEB-INF/lib and restart the server. However, it would be fantastic to have this documented formally.

download them here). " and there’s a link on “here”…

Yes. That leads to: xwiki-rendering/xwiki-rendering-standalone/src/test/java/org/xwiki/rendering/example/ExampleTest.java at master · xwiki/xwiki-rendering · GitHub

However, there’s no entry point shown. All the methods are annotated as @Test, which implies they are for unit testing. The methods themselves look informative, but I can’t simply:

Create a class called MarkdownExporter.
Write an annotated @Test method in MarkdownExporter.
Build the class into a JAR or XAR file (not sure which).
Copy the JAR file into XWiki’s installation directory (or install the XAR via XWiki itself).
Use the UI to select some pages and pass them into MarkdownExporter.

There needs to be some kind of entry point into the class that XWiki calls when it receives input from the user interface to process a set of pages. It’s that entry point that isn’t shown anywhere. Where can I find the equivalent to public static void main( String args[] ) { ... }?

The http://extensions.xwiki.org/xwiki/bin/view/Extension/Component%20Module page clearly defines public List<Block> execute() as the entry point for a macro.

Again if you read this page http://rendering.xwiki.org/xwiki/bin/view/Main/GettingStarted you’ll see that it starts by explaining how to make it work without Maven by showing the classpath you need

Assuming I need to create a macro to implement the export pages as markdown feature, I need to know how to write a macro. This leads to: http://rendering.xwiki.org/xwiki/bin/view/Main/ExtendingMacro

The documentation states, “Another one that is the Macro itself. This class should implement the Macro interface. However we recommend extending AbstractMacro which does some heavy lifting for you.” Yet the page links to neither the Macro interface nor the abstract class. It would be handy to note that the package/class is org.xwiki.rendering.macro.Macro within http://repo2.maven.org/maven2/org/xwiki/rendering/xwiki-rendering-transformation-macro/ – or reference back to the setting up your classpath section on the getting started page.

All that said, I don’t think I want to create a macro, but extend the filter or filter application or write a component. Still not clear.

I don’t understand what you mean. Could you be more specific? Note that this is a wiki so you can edit the page to improve it if you think it’s missing something.

Since XWiki Rendering uses XWiki Components, you’ll also need the Component Manager API JAR (xwiki-commons-component-api and a Component Manager implementation. You could develop one that bridges to your own component system (Guice, Spring, etc) or you can use the xwiki-commons-component-default one that we provide.

Becomes (where represent links to the resource):

Since XWiki Rendering uses XWiki Components, you’ll also need:
- [xwiki-commons-component-api], the Component Manager API JAR; and
- [xwiki-commons-component-default], a Component Manager implementation (or you can develop [a renderer???] that bridges to your own component system using [Guice] or [Spring]).

I would have made the edits myself, but it seems I cannot use this forum account to edit those pages. If the forum and the wiki accepted the same credentials, then it’d be easier. (That is, I don’t want to create yet another XWiki-related account.)

What do I need to do to create:

A user interface (component?) that allows users to select multiple pages.
A filter (component?) that receives the list of pages and archives those along with all images referenced on the page.

Need I create a macro? A filter? An extension? A component? Where can I find examples that show the entry point (not a unit test…) for creating one of these items?

vmassol · January 4, 2018, 8:46pm

You can’t easily. You’d need a XIP download for the extension and we don’t provide one by default (you could create one though).

That’s not fully correct. You’d need to copy the JAR file + all the dependencies it requires.

The only info we have for offline is here: http://www.xwiki.org/xwiki/bin/view/Documentation/AdminGuide/Installation/#HInstallingwithoutinternetconnection

Dave.Jarvis · January 4, 2018, 8:57pm

You can’t easily. You’d need a XIP download for the extension and we don’t provide one by default (you could create one though).

Is it necessary if I’m writing my own component/filter/macro/application?

Also, what is the name for the thing I need to write so that I can select pages and export them to a Markdown archive (including images)? Am I writing a UI component? A filter? An application filter? A macro? Some combination?

I can wade through the documentation if I know what I’m writing. Right now, I don’t even know if all I have to do is write a subclass of Filter Extension, nor where to find the Filter Extension class (i.e., what JAR is it in).