The future of Xwiki and possible 'paragraph' (block) entities for cross-linking and searching

Help / Discuss
#1

I like to invite the developers of Xwiki to share some thoughts about this, if possible.

Because of my work, over the years did research on applications that allow for the growth of collective intelligence in organizations with networked ideation and collaboration.
There currently is a strong trend in collaborative teams to make use of ‘block level linking’, which is often associated with the ‘outliner’ applications. The application that made this feature popular on large scale, was Roam Research. There were such applications already but not for multiple users. Since then, most well known applications starting to implement similar solutions.

You are probably very aware of apps like Obsidian and Roam research, which aimed to make the Zettelkasten (networked thought) possible as a personal app. Currently they are developing multi-user features for it also, and with them a number of other competing commercial and open source projects are following the trend.

I see value in this evolution, since the idea that ‘pages’ are the standard entity to make connections between, we inherited basically from the habit that paragaphs of paper pages cannot be used as separate building blocks in other pages.

We already have a feature being used in Wikis for years, which is transclusion. Transclusion is generally the inclusion of the content of a document into another document by reference. In Xwiki, it is the use of the Includ Macro and the Display Macro. However, transclusion is not the same as building up a page using paragraphs that are entities by themselves, which can be searchable, and can have properties assigned to them. A step further would be the ability similar to an outliner, in which the blocks can be easily embedded by dragging dropping and indenting.

I wonder if the developers have considered this outliner or text block level cross-linking, for the future of Xwiki?

#2

I’m a newbie here but as we’ve been dealing with this question a lot while still on MediaWiki, maybe I could share my thoughts.

Altready in MediaWiki, using the Semantic extension, we’ve been half-manually doing what you describe: Splitting up a document into subchapters, assigning metadata to make them searchable, and then transcluding it into one whole readable page. Now we assess XWiki far better in many aspects and here you can do just the same: Splitting up the document into subchapters, then adding property objects to them, and finally transcluding it all together. The separation of content and metadata is much better like this compared to Semantic MediaWiki (where you need to use inline annotations).

Still, the work of splitting up the document has to be done beforehand. We use a Python script for this and do this offline based on Markdown formats, thus creating a set of interlinked and transcluded subchapter pages which together make up for one whole readable document, whereas the subchapters are subpages and searchable on their own. The nested pages architecture of XWiki is very appropriate for that. We’re uploading the whole stuff using the REST API (after converting it to XWiki using Pandoc) and as such the process is automatic in principle (in principle because for any new doc we have to analyze and test if this will really work).

Is your idea that XWiki should provide a feature to automatically “explode” a given document with a heading block structure into such subpages? That would be interesting but also challenging I’d guess. If I misunderstood you then sorry for the interrupt.

1 Like
#3

Note that @caubin just started some development on an extension that might be related a bit to this subject: New contrib extension - Unified Documents Application

1 Like
#4

@Lukas Yes that comes close to what I thought indeed. I must admit that I’m not a core developer, although I have built Drupal sites for collaboration initiatives for many years, I mostly focused on user experience and conceptual engineering. As an NLP trainer I’m interested in belief systems and how we create meaning and perception.

I think what is happening in current trends is that the end users are becoming more dominant in the design of web experiences. Beforehand, tools were mostly dished out by developers and tech companies, that decided and recommended the general public what to use. End users are becoming more confident in web based tools, and less intimidated by technology. Some online tools to develop apps are even very accessible for non-techy people.
Because of this accessibility, a larger group of creative people are now contributing to open source initiatieves, or even come up with whole new approaches based on their own experiences and needs.

Taking myself as an example, I was frustrated for years because of the overload of notes and documents I was gathering, under which I drowned myself. Years ago common tools were mostly structured hierarchically or search based. There was no easy way to have insight in emerging meaning and relationships, apart from the indirect references using tags. So there was this tension in me of allowing the continuous stream of ideas to be useful, and the search for an app that can reflect the way one brain (and mutliple brains work) together. For human logic, this initially appears quite counter-intuitive because the brain (or more accurately, consciousness) works on multiple levels of awareness in a very organic way, what we would call ‘unorganized’ or ‘chaotic’. So currently there is more understanding about the benefit of building intuitive relatinships of meaning. Good writers already know this, that the best ideas and insights often ‘emerge’ and are less likely the result of logic thinking. That is because our brain processes most work in the background of our mental awareness.

Now why is the paragraphs based, atomic structure of a platform more useful? Because our imagination does not work with ‘pages’ but with scences, levels of identification and stories. We humans compose and combine stories as an internal movie continuously, simultaneously, in which we play a variety of roles, simultaneously. The more conscious we are of these roles and how we create our reality, the more artful and intelligent we can deal with our world and ourselves. Now researchers found evidence that neural systems actively remove memories, which suggests that forgetting may be the default mode of the brain, but its not the same as erasing. Hypnosis shows how accurately information is accessible still. It is still available in the background, and memories with similar themes can suddenly arise, be combined and enrich the understanding of a topic. So if we want to mimic that in our ways of being creative in our work, then it will indeed be quite similar to the Zettelkasten method, and even beyond that (the originator Niklas Luhmann only used paper notes).

So, my interest in a decentralized atomic information structure comes mostly out of frustration about the limitation of the page formats. Of course pages are still an important concept at the moment, and there needs to be a balance between creative chaos and structure. But I think that we collectively are moving towards new ways to learn and communicate in a more organic way.

Thank you, its always inspiring to find people actively participating in a open source community. And what you say about Xwiki its potential in comparison to Mediawiki I believe, although I got the impression that Xwiki is being actively developed mainly by just a handful of people, there is not such a big community contributing as I see. Very impressive though, what has been built over the years here.

#5

Thanks @surli that was a quick connection you made! Spot on. It looks very interesting, will try it out. Maybe @caubin is willing to chime in here in this topic and explain a bit more about the ideas behind his project? I’m for example interested to know whether the so-called ‘sections’ are technically normal xwiki pages, or that they are different entities, which can be accessed separatedly outside from pages.
In comparison, for example the well known outliners such as Roam Research and Logseq use the concepts of both pages and blocks, and blocks can be easily turned into pages, and blocks can contain child blocks, etc.

#6

Hi,

Thanks for the ping :slight_smile:

To give a bit more context, the Unified Documents Application was initially developed for a client of XWiki SAS. XWiki SAS is the company who employs most of the core devs of XWiki ; we also offer support, hosting, and custom services for some clients (see Sponsoring Companies (XWiki.org)).

The initial need behind this app was to help build very large documents (technical reports, contracts, … generally of 100+ pages), where :

  • each section of a document needs to be written independently (because it requires a specific expertise) ;
  • each section can be included in other unified documents (for example, a section “Introduction” may be shared among many unified documents if it contains a very generic content) ;
  • a document can written in different languages (translation support), while keeping the same section structure

The Unified Documents application comes with 3 key data structures (XWiki classes), which correspond to 3 “types” of document that can be created :

  • Sections : they define the title of a specific section in a document, for example “Introduction”, “Conclusion”, “Design analysis”, …
  • Section contents : a page of type “Section content” is linked to a section, and provides the content for this section in a specific language. For example, the section “Introduction” can have two section contents, corresponding to the English and the French version of an introduction.
  • Unified documents : this data structure is what links sections and section contents altogether to produce a unique result. In a unified document, a user can define a tree of sections to be displayed. The extension then calculates the numbering of each section in the document, and generate the aggregate of every section in the different languages of the document.

If you want to see how a unified document looks like (in view mode), you can head to Fairtrade Assurance - Rules and Guidelines (Public) - XWiki ; I’ll add screenshots to the extension documentation page once the application is fully contributed.

So this is the main use case behind which the Unified Documents app was initially created. The main stakeholders of this application agreed to make it Open Source, hence the new extension on the Contrib repositories.

Back on the topic of this thread, even though the app allows for content transclusion between section contents and the unified documents themselves ; I feel that this transclusion would be much more “heavy” to set-up compared to simply using an include macro in an XWiki document for your use-case. Working with paragraphs only in XWiki is indeed a bit difficult today, mainly because all the logic for accounting for changes, managing history, applying permissions is handled at the level of the document, which serves as a big bag for carrying its title, its content and other metadata through XWiki Objects.

While it would be very difficult to move away from this model, there are some XWiki applications that currently allow targeting only a specific part of a document contents, such as the Annotations Application (bundled in XWiki standard). Here, this application attaches a string (a comment) to a specific part of the text in a document. The main issue we face today with this approach is when it comes to the update of the document content, where annotations can be lost because the context information that was used to locate the annotation within the document is not available anymore.

1 Like