New application: ActivityPub

Hi everyone,

just to inform you that I’m starting to work on a new contrib application to allow XWiki to be part of the fediverse by implementing the ActivityPub protocol.

Note that this work is funded by the NGI Search & Discovery call.

I don’t have anything to show yet, but I’ll keep you informed here.

For full visibility and disclosure, here’s the project we submitted and that is sponsored by the NGI Search & Discovery call.

XWiki Federation

Abstract

Can you explain the whole project and its expected outcome(s).

Please be short and to the point in your answers; focus primarily on the what and how, not so much on the why. Add longer descriptions as attachments (see below). If English isn’t your first language, don’t worry - our reviewers don’t care about spelling errors, only about great ideas. We apologise for the inconvenience of having to submit in English. On the up side, you can be as technical as you need to be (but you don’t have to). Do stay concrete. Use plain text in your reply only, if you need any HTML to make your point please include this as attachment.

So far, XWiki has been focusing on providing the best collaboration experience and features to its users. It’s now the right time for XWiki to start collaborating externally and be part of the larger federation of collaboration and social software (a.k.a. fediverse). The proposal is to achieve this collaboration by embracing the W3C ActivityPub specification (https://www.w3.org/TR/activitypub/) and becoming compliant with it. More specifically to implement at least the server to server part of the specifications and be able to both view activity and content happening in external services (such as NextCloud, Discourse, Mastodon, PeerTube, Friendica, Plume, etc) inside XWiki itself and to make XWiki’s activity and content available from these other services too. Similary, we would also like to consider XWiki as an external service and work on the XWiki federation itself. This means considering the various XWiki instances around the world and be able to share activity and content between these instances.

Note: We use AP below this point to mean ActivityPub.

General information about AP:

Use Cases & Integration Points

  • Groundwork:

    • Integrate into XWiki’s profiles by exposing each profile as an AP URL returning data as JSON.
    • Integrate into XWiki’s authentication/permissions system so that only allowed external users are able to perform AP queries against XWiki
    • Implement discovery of user profiles, using WebFinger. The idea is to convert a user alias (in the form of name@domain) into an AP URL.
    • Integrate and test with other services. Since the integration between services requires some discussion and collaboration (the specifications leaves some points open), focus on integrating and testing with a few services to start with: XWiki instances, NextCloud, Discourse. The choice of NextCloud is because we think it makes business sense to get closer to them and because of the maturity of their implementation. The choice of Discourse is because we use it as the forum for the XWiki open source project at https://forum.xwiki.org and an integration with Discourse would thus be interesting.
  • UC1: Personal Following/Followers

    • Integrate into XWiki’s existing follower/following feature. Also allow each XWiki user to follow the activity of external users from external services. This means modifying the XWiki profile UI to be able to enter the external user handles.
    • Contribute the activity done by users you follow in your XWiki Activity Stream.
    • Support this use case for external XWiki instances + NextCloud + Discourse (provided their implementation is good and ready to be used)
  • UC2: Global Following & Content viewing

    • Add the ability for an XWiki admin to enter users to follow at the level of the whole XWiki and contribute their activity to the general Activity Stream of XWiki.
    • Be able to not only follow the activity but also view the related content from inside XWiki (created content, modified content, etc).
    • Support this use case for external XWiki instances + NextCloud + Discourse (provided their implementation is good and ready to be used)
    • Performance of content viewing. For performance reasons, the first time some content is fetched remotely, it would be stored locally on the XWiki server so that subsequent queries to view content would serve it locally until the content is modified again remotely.
  • UC3: Expose XWiki Content

    • Expose XWiki pages as AP endpoints so that they can be viewed from external services (or other XWiki instances).
  • UC4: Messaging

    • Integrate with XWiki messaging feature so that messages can be posted to external users and not just internal users as it’s currently the case.

For all use cases:

  • Support integration with external XWiki instances. Ideally NextCloud and Discourse would also be able to display XWiki info in their UI but it depends on their willingness to do so and we can’t commit on this.

Future use cases (not planned for this first grant. Candidates for future work):

  • UC5: Integration in XWiki’s notifications
    • Display external events and content inside XWiki’s notifications system.
  • UC6: Federated search
    • Integrate into XWiki’s search feature and return results from XWiki nodes and other software connected via ActivityPub.
  • UC7: External modifications
    • Be able to also edit XWiki content externally and not just view it. It’s more complex and requires handling conflicts and merge.
  • UC8: Cryptpad integration
    • Integrate and implement AP also in Cryptpad so that XWiki and CryptPad (and the rest of the world) can share activity and content.
  • UC9: Blog Integration
    • Integration of AP in the XWiki Blog Extension to list blog posts from followed users in the blog view. When creating new blog posts, also publish them with AP so that they can be viewed in other platforms, such as WordPress. Add extension points inside XWiki for implementing this cleanly.
  • UC10: More Integrations
    • Increase the number of external services XWiki integrates with by testing and adjusting the XWiki AP implementation depending on how these services have implemented the AP specification.
  • UC11: Advanced messaging
    • Support of message threads and support the sharing of partial conversations.

Advantages

  • Increase search and discovery on the internet by increasing collaboration on content and activity from the growing list of services implementing AP. XWiki’s content/activity become available to all services implement AP and the content/activity from these external services become available from within XWiki.
  • Implement the XWiki federation by allowing decentralization for XWiki and the ability to achieve synchronization of different instances of XWiki using AP.
  • Make XWiki a nicer web citizen by integrating with other services. It also provides more integrations of services for XWiki and thus makes it simpler for users to cross software boundaries and not be locked to a single software.
  • Generally speaking, promote decentralization and federation in the open source at large.

It should also be noted that while there are a number of solutions already implementing at least partly ActivityPub, there is no wiki engine currently implementing it (the closest to the wiki approach might be Dokieli, but it has only basic wiki features). This would make XWiki a pioneer and possibly inspire others. The recent Wikipedia outage shows that a new generation of solutions is needed for enhanced resilience. Decentralized wikis can be a good solution.

Past Experience

Have you been involved with projects or organisations relevant to this project before? And if so, can you tell us a bit about your contributions?

XWiki SAS (the main company sponsoring the development of the XWiki open source software and of Cryptpad) has participated in several research projects in the past that are are relevant to this domain:

The outcomes of the project will be published as an XWiki Extension and thus installable and usable immediately (through XWiki’s Extension Manager) by the 7000+ current active installations of XWiki over the world.

Comparison

Compare your own project with existing or historical efforts.

Compared to the experiments done in the “Past Experience” section above we believe that AP is more interesting as it’s a standard specification that applies to all content/activity software. Once we implement the specifications in XWiki, this makes XWiki interoperable with all other software implementing AP too (In practice this won’t be completely true since the specifications leaves some areas underspecified and open but it makes partially true and the remaining work to make it work with a specific software is marginal). If AP catches on and more and more software/services implement it, it would be a game changer for interoperability over the internet and a win for open standards and for XWiki.

Challenges

What are significant technical challenges you expect to solve during the project, if any?

The main challenge we foresee are differences in the various actor’s AP implementations. This can happen for 2 reasons: the AP specification is underspecified in some areas and thus leaves it open for interpretation (such as authentication) and it’s also extensible and thus if we want to benefit from the extension inside XWiki we would need to discuss and interact strongly with the actors to understand how they extended the AP formats.

Ecosystem

Describe the ecosystem of the project, and how you will engage with relevant actors and promote the outcomes?

We propose to interact and integrate with at least 2 other actors (NextCloud and Discourse) to make sure XWiki’s AP implementation and these actor’s AP implementation work well together.

We will also promote search and discovery and AP virally through the fact that XWiki is open source and well known and used over the world but also through XWiki SAS’s regular technical conferences attended every year around XWiki’s technologies (XWiki SAS does between 10 to 20 events annually at various conferences).

Attachments

Attachments: add any additional information about the project that may help us to gain more insight into the proposed effort, for instance a more detailed task description, a justification of costs or relevant endorsements. Attachments should only contain background information, please make sure that the proposal without attachments is self-contained and concise. Don’t waste too much time on this. Really.

And here’s the research I had done on this topic. Maybe it can be useful.

General Research

2 parts:

    1. a client/server API for creating, updating and deleting content,
    1. a federated server-to-server API for delivering notifications and content.

Implementations:

Most implement part 2 only.

part 1: decentralized websites can share information
part 2: users, including real-world users, bots, and other automated processes, can communicate with ActivityPub using their accounts on servers, from a phone or desktop or web application or whatever

“ActivityPub implementations can implement just one of these things or both of them. However, once you’ve implemented one, it isn’t too many steps to implement the other, and there are a lot of benefits to both (making your website part of the decentralized social web, and being able to use clients and client libraries that work across a wide variety of social websites).”

Activity Stream 2.0:

  • Actors: Application, Group, Organization, Person, Service.
  • Activity types: Accept, Add, Announce, Arrive, Block, Create, Delete, Dislike, Flag, Follow, Ignore, Invite, Join, Leave, Like, Listen, Move, Offer, Question, Read, Reject, Remove, TentativeAccept, TentativeReject, Travel, Undo, Update, View.
  • Objects: Article, Audio, Document, Event, Image, Note, Page, Place, Profile, Relationship, Tombstone, Video.

“To build a valid Activity Streams activity, you pick one of each category and add some metadata to it. You describe that something did something to or with something, and you explain those things in more detail.”

Haven’t found any proper java library to help implement ActivityPub. Some projects that can help:

Good links to understand it:

Implementations seem to be quite specific and under-specified. For example for Mastodon: “Applications interoperating with Mastodon over ActivityPub must support Webfinger resolution, with a unique preferredUsername per host, so that they can be interacted with by other users. Posts to the inbox must be signed using the HTTP Signatures draft spec with a public key that can be fetched from the actor, using the publicKey object format from the WebPayments JSON-LD Security Vocabulary.”

“If you are not too familiar with web terminology, think of ActivityPub as the postal service. The postal service drives around and delivers letters into inboxes, but it does not care about what actually is inside those letters. This is not a perfect analogy, but close enough.”: this shows that interoperability with ActivirtyPub is going to be messy…

Chat:

There’s a “source” property for defining the format of the content, see https://www.w3.org/TR/activitypub/#source-property. Check the note that is very interesting to understand how client should allow editing mediatype they don’t understand.

Implementation notes:

Use Cases for XWiki

  • Integrate into XWiki’s Notifications/Activity Stream with events coming from other servers (e.g. NextCloud)
  • Allow your XWiki activity and page content to be viewed in other servers (Mastodon for example)
  • If Cryptpad also implements AP then we could see the pad’s activity and content from XWiki and vice-versa
  • General objective: make XWiki more interoperable with other software.
  • Add a new tab in the user’s profile for external content/for viewing the inboxes of followed actors. Integrate the XWiki following action with AP. Add a field in the user profile to specify aliases (of the form @) for a user on other servers. When following a user, use these aliases for AP. Also add the ability to explicitly follow a new external person in the UI
  • Integration of AP in the Blog Extension to list blog posts from followed users in the blog view. When creating new blog posts, also publish them with AP so that they can be viewed in other platforms, such as WordPress. Add extension points for implementing this cleanly.
  • XWiki federation: ability to view (possibly edit although that’s a lot more complex) content and AS from other XWiki instances. Example use case: RR where the content has to be stored in specific countries and there are export law permissions applying.

Notes about UC

  • UC7: We need to use virtual users to be able to manage rights. It only applies from one xwiki to another, and if the syntax is supported. If not supported, we should either just present the page in view mode or try with an html edition (?).
  • UC3: Idea, to get an entire arborescence from a given reference to “mount” an external wiki