LiveData without pagination

Development
#1

Hi everyone,

so I’m writing this topic as a follow up of a brainstorming started on the chat with @mleduc. This brainstorming started because I have a problem with a LiveData in Change Request extension: my UC is that I want to display a LD containing specific xobjects (with the usual filters on xobjects), but that also respects some specific dynamic criteria (in my case I have to check a combination of critieria such as right, configuration option, etc).
So right now I have a custom source that is calling the default LiveTable macros and then iterate over the entries to filter them out if some APIs check is not met.

And the problem with that is the pagination.

Currently, our way to deal with that issue is by obfuscating the entries: that’s why we have the N/A entries in index of documents, and we even have this hardcoded hint below the table to inform why some entries are not displayed.
The problem is that for me this solution is very focused on the UC of rights: that’s even the reason why the hint is currently hardcoded and mention the right, see:
image

So for me that solution works to fix a pagination problem, but provides a very bad UX, see an example of the LD I obtain for CR:
Capture d’écran du 2023-03-28 09-25-39

Note that in this screenshot all the entries I’m getting are actually viewable by the user, it’s just that the one with N/A cannot be reviewed by that user. So the hint is not good and the table is tedious to use as we get many pages with obfuscated entries.

Ideally the solution would be to be able to compute properly the total number of entries and to iterate over them: the problem is that since the filter is very dynamic it’s very expensive to do that. I would basically need to load all the info in memory, and it’s very possible that with time the number of CRs grow enough so that it doesn’t scale anymore.

For all that reason, @mleduc proposed a solution in the chat that could fit properly this UC: to just not display any number of pages, but have a button “load more”.
The idea would be to have a way for LD to specify that a source is not returning the total number of results, and to adapt the UI so that it would only propose to load new entries.
Obviously this would have some impact: the sort of columns would be limited to the already loaded entries in such case.

But I think it would be an elegant solution to avoid this kind of problem in the future, when it’s really difficult to provide proper computation of the total number of entries.

wdyt?

1 Like
#2

We always have this problem, even with the default LiveTable source. It is rare to have a source that doesn’t have this problem. So +1 for implementing this, and also +1 for implementing this for the default source.

I think, however, that we shouldn’t just provide a way to load more entries but instead to provide pagination with next/previous links. Also, it might be nice to display some approximation for the number of entries. The next/previous links could use cursor-based pagination to allow efficient calculation of the next/previous entries. I expect you’ll need this, anyways, to compute the next entries so I think it shouldn’t be much more complicated to compute the previous entries. The exact cursor value should be determined by the Live Data source and should be treated as opaque string values outside of it. The cursor could also be the offset before filtering in your case, but it could also be a serialization of the first displayed entry. It probably makes sense to also send the cursor value of the currently displayed data to the client and to send it back to the server when the next/previous page is requested so the source doesn’t need to compute it again in case the computation is expensive.

I would also suggest to not to implement this for LiveTable but just for Live Data (if that was not already the plan) to avoid the duplicate work.

1 Like
#3

+1 for the general idea

Why ? Isn’t sort supposed to ask the source for new results, but with this sorting constraint this time ?

#4

I think sort is possible in this proposal, but the results are reset the first page.

Also, +1 for the general idea

#5

Yes, but I think that’s not really a problem. At least when I change sorting settings, I usually want to go back to the first page, anyways - at least if the sorting isn’t just adding a sub-criteria, and that’s already a very special case. Depending on the used cursor values, the source might support displaying all entries starting with the first currently displayed entry in the new sort order but imho that’s not really intuitive.

#6

Yes, I would also be expecting the reset, in case of sort update but also in case of filtering update.

#7

Sure, but that’s not a problem, and it’s definitely far from being “limited to the already loaded entries” :slight_smile:

#8

+1 to add support for cursor based pagination. At the API level we need to:

  • return the start and end cursor with the live data:
    • LiveData#getStartCursor() : String
    • LiveData#getEndCursor() : String
    • LiveData#getCount() would return -1 when cursor based pagination is used
  • add support for specifying the cursor and direction in the live data query:
    • LiveDataQuery#setCursor(new Cursor(value : String, direction : boolean))
  • add a live data macro (boolean) parameter (cursor="true") to enable the cursor based pagination; I think boolean is enough because I’m not sure storing the cursor in the wiki syntax is a good idea

At the UI level the live data widget needs to detect (from the live data configuration) when cursor pagination is active and update the UI accordingly. A query like this could be the trigger:

"query": {
  "properties": ["title", "year", ...],
  "source": {
    "id": "..."
  },
  "cursor": {
    "value": "", // empty meaning the start
    "next": true
  }
}

Thanks,
Marius

2 Likes
#9

So it’s actually out of the scope of this proposal. I was thinking about my specific UC where I also inject a new column data during the post-filtering: right now I don’t even allow to sort on this column, but I was thinking about allowing it only for data already loaded.

#10

So I opened a JIRA ticket for implementing the cursor based pagination: Loading...

1 Like