Backend-agnostic storage of user-defined configurations

Cristal
1

Hi everyone,

An important feature currently missing in Cristal is the option to let users define their own configurations, and connect to backends they want without requiring for the instance administrator to add them globally for every user (e.g., their own personal XWiki instance, or GitHub repository).

These custom configurations will need to be persisted between browser sessions, and ideally between Cristal instances as well (if we want to let users share their configurations between a browser instance and the electron client, for example).

From a technical point of view, a generic way to manage that would be to add the concept of “Personal Storages” to Cristal, that could be used to not only store configurations but also user settings that we may need to handle in the future.
A first Personal Storage implementation could use the browser’s local storage, and its content could be later synchronized with other remote implementations.

The point of this proposal is two-folds:

  • First, decide if it’s the right approach.
  • Second, choose the implementations we should focus on initially.

There will be a second proposal for the UI later.

Possible backends for Personal Storage:

XWiki

Storing personal user settings in XWiki is a logical choice. For XWiki admins that also deploy a Cristal instance, this makes a remote storage solution available out of the box. The data could easily be stored as XObjects on the user profile page. My only concern in making this our first target is that it forces users of other backends to also make an account on an XWiki instance just to synchronize their configurations, even if they don’t plan on using it as a backend.

Solid pods

Solid pods are a way of offering a decentralized storage solution for web applications. Anyone can make an account on a pod provider service, or self-host their own provider service, and get a pod that an application can then use to read/store data. Users can also easily migrate their pod from a provider to another. There is, however, a concern in asking users to make an account on an external service in order to sync their data, which could be mitigated if an administrator also offers a pod provider service but it’s still extra steps.

remoteStorage protocol

The remoteStorage protocol offers basically the same feature as Solid pods. An added benefit is that it supports data scoping. Their library also supports several backends, such as Google Drive and Dropbox, and should include Solid in the end (they have an active PR for that). In theory, this should be the most flexible way to handle personal data storage, but at this point I have no idea how hard it would be for an administrator to set-up support for the other backends.

Personally, I would lean into remoteStorage as a first target since I believe it gives us the most flexibility with minimal work, but WDYT?

2

I’m not sure about this project. The project seems really slow since 2018, and almost dead since 2022.

Same thing for solid, I might be wrong, but my feeling is that solid is currently not a stable enough technology to be used for projects that are not directly aiming at experimenting with solid itself.

By elimination, I’m +1 for introducing a “Cristal extension storage” extension.
The limitations regarding authentication could be addressed later by plugging an SSO on the XWiki instance, avoiding the need for separate authentication.

3

From what I understand, development has slowed down because the project is considered stable. Dependency upgrades are still performed properly on the library, and issues and PRs still seem to be addressed.

Do we actually need an extension here? It’s not clear for me what the benefits would be.

4

I guess we could/should have a generic Cristal typescript api for data requiring persistence and then there can be different implementations. One of them could be XWiki, another one could be a web database like firebase or other, and the default one could be the browser’s local storage for the Cristal web distrib and the local filesystem for the Cristal Electron distrib.

I’m probably repeating what you said (but using my own words ;)).

Thanks

5

That’s a good summary :wink:
The only point missing is that I don’t think the implementations should be exclusive, but the UI should offer a synchronization mechanism between them.
A basic process I can imagine is a user working exclusively with their local storage at first on the electron client (or a browser session, but in this case we would display a warning about potential data loss), until they actually have a use-case where they need to create an account on a remote storage.

6

I’m not sure about the technical solution, but I think that the backend storage should also be able to store user accounts.
We might want to have accounts stored, be able to have several backends related.
For example, I’m first creating an account through Github SSO, but at my next connexion I want to link my Nextcloud account.
Would it have an impact on the possible solutions proposed?

7

Unless you plan to reuse pre-existing concepts provided by XWiki, I don’t see a way around defining a new “Cristal config storage extension” that would provide the concepts and APIs to allow users to save and access their configurations.

Note that this extension could actually be a remoteStorage provider if we agree to go with it.

This is quickly going to be needed to synchronize configurations across devices and browsers. But this is not mandatory in a first step.
We could start with only local storage (possibly with a way to export in JSON).

That’s indeed a point to be discussed separately. I’m not sure about the security implications.

  1. storing authentication tokens seems dangerous if done wrong
  2. another option is to allow services to all be linked to the same SSO, that way a single login would be enough to get access to everything

Feels like it’s more about what we choose to store than the solution itself.