Hello all,
For the file system and NextCloud storage, Cristal currently stores files at the root of a directory:
~/.cristalfor the file system- a
.cristaldirectory at the root of the user’s storage for Nextcloud
In both cases, the content of the directory is the same:
- one directory per page
- a
page.jsonfile containing the content of the page as well as all the metadata (e.g., syntax, title, author…) - sub-pages are stored in subdirectories
attachmentsis a special keyword, attachments of page X are stored inX/attachments/
The current storage suffers from several limitations:
- if a user creates an
attachmentspage, there is a potential collision - users accessing the storage externally can’t easily read the page content
Note: synchronization of content edited externally is currently out of the scope.
Proposal
To address those limitations, I propose the following.
File and pages names
Option A
- page and sub-pages are stored as directories and subdirectories, the page content is stored in a
page.mdfile (e.g.,mypage/page.md, orsome/path/page.md) - attachment directories are now named
.attachments - a page name can’t start with a dot
Option B
- spaces are directories, but pages are stored as
${pageName}.md, in this case attachments should be stored in.attachments_${pageName}subdirectories. - a page can’t start with a dot (to avoid conflict if someone names a space
.attachments_existingPageName)
Page format
This section discusses specifically how we store the page content and its metadata.
Option 1 — Embedded metadata
In this case the metadata are stored directly in the markdown file.
A common format is to have a header with json or yaml content.
---
title: My Page
creation-date: YYYY-MM-DD hh:mm
---
# Page H1
Some content....
This is quite common, this is for instance what Hugo or Jekyll
Cons:
- if a backend support markdown but not header metadata, they will be displayed, breaking the rendering
- based on how complex are the data we want to store, having them in the header might not scale
Option 2 — Embedded metadata
A variation of option 2 is to store the metadata at the bottom of the page.
In case of lack of support from the backend, they metadata would still pollute the rendered page, but only after the main content.
The cons being that accessing the metadata required to read the whole page content which might not be optimal.
Option 3 — Side file
In this case, a pageName.json file would be stored aside the pageName.md file, storing the page metadata.
Pro:
- This solves the support from markdown header by the backend
Cons: - This makes the move/rename page logic more complex and increases the changes of users breaking the storage by manually moving the markdown file without its json metadata file
Option 4 — Side directory
This is following the same logic as Option 3, but with a directory instead of a file.
In this case a .pageName_metadata directory would be stored aside the markdown file (same logic as Option B for the attachments).
This directory would itself store metadata in files.
The pro being that each extension could choose to store its own files (e.g., .pageName_metadata/extensionId/data.json), making the storage design more extensible.
Note: if we consider the attachment storage as an extension, attachments could even be stored in .pageName_metadata/attachments/.... But we can consider attachments are a primitive concept with it own rules, to make them easier to access in the architecture.
Do you see something missing?
What is are preferred options?
My choice, before discussion is:
- Option B
- Option 4 with a special handling for attachments
I feel like this is the combination leading to the most robust storage architecture while allowing users accessing the storage externally to easily understand the files structure.
Thanks for reading this far.
PS: Given the current state of the project, I’m -1 to introduce a migration tool from the old format to a new one, unless somebody explicitly ask for it.