New Filter Streams Extension: GitHub Importer

Requirements

1. Import from a GitHub Wiki repository

Instead of creating each page on XWiki Platform, user will be able to simply provide the GitHub repository with Wiki and let the Filter Application convert it to XWiki pages and save them inside XWiki according to the Sidebar (if provided).

2. Main Repository

A wiki repository is different from the main repository. It can be cloned via .wiki.git. Some users may provide the main <repository>.git, so the extension will automatically fix the link to improve UX.

3. Convert Syntax

Convert the pages written in the GitHub-flavored Markdown markup to XWiki Syntax 2.1. Also provide an option to decide to do this conversion or not (if users want to use them in MD in XWiki they should be able to do so).

4. Ability to pass an oauth token

Make the import process accept a oauth token to perform the fetching of the GitHub repo. This allows to support private repos.

5. Provide a simplified Import UI

The Filter Stream app provides a generic UI but it’s complex and not very user-friendly. This app should provide a simplified UI geared towards importing GitHub pages.

6. (Optional) Merge Wiki

In this use case, if the wiki is re-imported, it merges the changes inside the pages so the new page will be shown merged instead of overwritten.

7. (Optional) Two-way sync

If the project is done early and if there’s enough time, it would be interesting to support round-tripping and be able push changes done in XWiki on the imported pages back to GitHub (either by git pushing or by using a GitHub REST API). Specifically this means:

  • Remembering the list of imported pages and their content.
  • Writing an XWiki EventListener for auto-sync.
  • Having an XWiki UI to display changes since the last import with ability to see the diffs.

8. (Optional) Support offline importing

User will be able to use a .zip file containing the wiki repo, as an input to the GitHub Importer and get them into XWiki as XWiki pages.

Feel free to suggest more use cases.

…to be updated.

That’s a great idea. We shouldn’t duplicate jira ofc but it allows to organize all tasks that are not coding related. I’ve added some tasks there!

I’d recommend to all other mentors/students to do the same. I think it makes things more clear for everyone.

1 Like

Cool. Thanks for listing the use case on https://design.xwiki.org/xwiki/bin/view/Proposal/GitHubImporter. We can use this thread to discuss about them.

I’m wondering if we should support the use case of importing the pages from a zip for example. This could allow an offline use case: you download the pages from the github repo as a zip and later on you can provide this zip as input to the importer to import them inside xwiki.

Ofc we should also implement the direct use page where pages are fetched directly from github using the github API.

WDYT?

4. Convert Syntax

Convert the pages written in the GitHub-flavored Markdown markup to XWiki Syntax 2.1. Also provide an option to decide to do this conversion or not (if users want to use them in MD in XWiki they should be able to do so).

Note that for this you’ll need to improve the Markdown extension to make the MD flavor configurable, see https://jira.xwiki.org/browse/MARKDOWN-24 and especially the comments (since the issue needs to be repurposed as I’ve changed idea for it).

1 Like

@haxsen I’m putting ideas here but I’ll let you port them all to the design page (if you agree with them ofc!).

I’d rename this UC to “Import from a GitHub Pages repository”. WDYT?

5. Ability to pass an oauth token

Make the import process accept a oauth token to perform the fetching of the GitHub repo. This allows to support private repos.

Side note: we started calling these “use cases” but it’s probably better to call them requirements. WDYT?

6. (Optional) Two-way sync

If the project is done early and if there’s enough time, it would be interesting to support round-tripping and be able push changes done in XWiki on the imported pages back to GitHub (either by git pushing or by using a GitHub REST API). Specifically this means:

  • Remembering the list of imported pages and their content.
  • Writing an XWiki EventListener for auto-sync.
  • Having an XWiki UI to display changes since the last import with ability to see the diffs.

7. Provide a simplified Import UI

The Filter Stream app provides a generic UI but it’s complex and not very user-friendly. This app should provide a simplified UI geared towards importing GitHub pages.

I wonder if this will work, since the .git repo is different from .wiki.git repo. See: https://gist.github.com/subfuzion/0d3f19c4f780a7d75ba2
and https://github.com/godotengine/godot/issues/2019
Although the user can clone the wiki.git and zip it (extra work) and send it to GitHub Importer?
Let me know if I understood it wrong.

Yeah great, thanks. I will keep them in sync.

Yes I think that’s better.

Yep they’re actually requirements. I saw some designs on design.xwiki and they were using the Use case word so I thought I should I also do the same.
I am changing this to requirements, it’s better.

Yes that was the idea. Note that I consider this idea as not very important but it could be interesting to address the offline use case.

Yes both are used but they don’t mean the same things to me. For me a use case is from a usage point POV, and they are large, while requirements are not related to usage but more about characteristics that the implementation must implement. It’s pretty close I admit.

1 Like

We could make the “support offline importing” requirement as optional for now and push it to the end. But let’s list it.

1 Like

Hi @vmassol I hope you’re doing well.
I thought I should let you know that I wanted to request some time from you and Simon. You may have noticed I am not working very well lately (spending less time on XWiki) but it is mainly due to university issues.
I am trying to finish my semester project by this month so I can give full-time to GitHub Importer from 1st June. When I am not on XWiki, it’s usually because I am working on either semester project or subject assignments. University quizzes and assignments are not a big deal, but semester project is. The sooner I finish it, the better I can focus on GitHub Importer. It doesn’t mean I will vanish from community though, I will keep in contact and will do what I can when I find some time. Usually students are working on their university projects in this time (I asked my GSoC fellows). You can contact me any time and I will communicate with you.
So, please grant me some days to at least do my semester project up to 80% this month. I will be very grateful :slight_smile:
No problem if you can not. I will try to give minimum time on my semester project then and will try to give max time to XWiki.

Hi @haxsen. Thanks a lot for letting us know, that’s very important that we have some open communications, so I appreciate that a lot. Yes, sure, do what you have to do for your studies. Just don’t drop from the community and continue to do some small stuff.

Don’t start the GitHub importer then, but focus on small issues and the like. Also try to read https://dev.xwiki.org since it contains all the dev best practices that are going to be useful for the future (and https://contrib.xwiki.org).

Again thanks for your message, this allows to align everyone’s expectations.

Cheers