Increase the size of all the document reference related database columns

Development
1

Hi devs,

A first safe (from retro compatibility point of view) step for improving the limit regarding the maximum size of the references in XWiki (currently 255 characters) could be to increase the cause of this limitation: the size of the columns in the database which store document references.

Long story short with the currently supported databases we can increase it to 768 characters without impacting noticeably the performances (of course bigger references will take more disk space but for example in supported versions of MySQL the storage is variable, and you don’t get the maximum size allocated all the time).

See https://design.xwiki.org/xwiki/bin/view/Proposal/Increasethe255characterslimitforreferences#HIncreasethesizeofthereferencerelatedfields for more technical details that leaded to 768. It also contains very first thoughts on other ideas to help workaround this limitation.

So I propose to do the following:

Of course 768 is still not huge, and I don’t see it as the final solution to close this subject for good, but it’s already a nice improvement that we can have in the next version of XWiki without taking much risk.

WDYT ?

2

+1

Thanks,
Marius

3

+1

4

+1, Thanks

5

+0 for 2 reasons:

  • 768 is still low and we’ll still have some problems
  • a migration is never simple and there are often edge cases or it takes long for large XWiki instances to migrate. In short, it’s not seamless and there are lots and lots of columns to modify here. So I would feel more comfortable if we decide what we want to do in the future first to have a target direction. Imagine if we find a different solution in not long and we have to do again another migration. It’s a bit painful and risky.

So I don’t want to block anything but a proposal/vote about the future would be nice.

2 notes:

6

FTR I couldn’t find an explanation on https://design.xwiki.org/xwiki/bin/view/Proposal/Increasethe255characterslimitforreferences#HIncreasethesizeofthereferencerelatedfields as to why we chose 768 chars for all DBs. Since we have different hbm files for various DBs, we coud use different limits depending on the DB. For ex for postgreSQL, it seems to be 8000 chars (Data types in Postgres) and thus we could use a higher value in xwiki.postgresql.hbm.xml.

@tmortagne What am I missing?

If the problem is that users who want to migrate from one DB to another would have issues with different max sizes, then I don’t think it’s a real issue (or at least it should be a choice of the admins, since this use case is quite unlikely for most instances).

Thx

EDIT: Also it seems that MariaDB allows for larger limits with higher page sizes, see InnoDB Limitations | Server | MariaDB Documentation

7

It seems you are mixing the maximum size of an index (which is the subject here) and the maximum size of a VARCHAR (which is not the subject, and you can have a much bigger VACHAR than 768 characters in MySQL/MariaDB).

8

Indeed. So 768 is the max index size for all DBs we support? (my question still stands otherwise).

9

No, but it’s a lot simpler to choose a number which works for sure in all supported DBs as it’s not always simple to really know what is the maximum (for example with postgres, as explained on https://design.xwiki.org/xwiki/bin/view/Proposal/Increasethe255characterslimitforreferences#HIncreasethesizeofthereferencerelatedfields, you cannot really know what is the maximum size as it’s indexing the compressed version of the value).