Solr-based LiveData Source

Development
1

Hi devs,

FYI, I’m trying to tackle Loading... with the goal of not doing a 100% full scale generic solution on the first go but enough (and a little beyond) to be usable for implementing Loading....

I want to see how good (or bad) claude code can be for this design work.

FYI the first pass design doc (generated by Claude Code):

Design: Solr-based Live Data Source (XWIKI-23940)

Context

  • Issue: XWIKI-23940 — provide a Live Data source backed by Solr to find and list documents in a Live Data table.
  • Primary driver: XWIKI-22733 — the Backlinks / “Information” section doesn’t scale, can’t filter/sort. Existing Live Data sources cannot display backlinks, so we need a new source first.
  • Scope agreed in XWIKI-22733 (Michael Hamann, confirmed by Vincent): a generic Solr search Live Data source. A first version only needs an arbitrary Solr query as input plus title and link columns. More fields added later as needed.
  • Open question from the issue (now resolved for v1): “use Solr only for query, or also fetch display data from Solr?” → Fetch display data directly from Solr (faster; avoids loading documents). Loading documents to reuse LiveTable column metadata is explicitly a later option.

How Live Data sources work (recap of the contract)

A source is three cooperating components, all looked up by the same hint and all PER_LOOKUP, extending WithParameters (so source parameters from the macro flow into them):

  • LiveDataSource (@Named("solr")) → exposes getEntries() and getProperties().
  • LiveDataEntryStore.get(LiveDataQuery) → returns a LiveData (a count + a list of entries, each entry a Map<String,Object>).
  • LiveDataPropertyDescriptorStore.get() → returns the column/property descriptors (id, name, type, sortable, filterable, displayer…).

LiveDataQuery is essentially an SQL-shaped object:

  • properties = select (which columns to return)
  • source = Source(id, parameters) (the from clause — our solr source + its params)
  • filters = list of Filter{property, matchAll, constraints[{operator, value}]} (the where clause)
  • sort = list of SortEntry{property, descending}
  • offset (Long), limit (Integer)

Reference implementation to mirror: xwiki-platform-livedata-livetable (LiveTableLiveDataSource / LiveTableLiveDataEntryStore / LiveTableLiveDataPropertyStore).

Existing Solr plumbing we reuse

  • QueryManager.createQuery(queryString, SolrQueryExecutor.SOLR) builds a Solr Query.
  • ((SecureQuery) query).checkCurrentUser(true) → SolrQueryExecutor filters out documents the current user cannot view. We get view-rights enforcement for free.
  • query.bindValue("fq", List<String>), bindValue("sort", "field asc|desc"), setLimit(n), setOffset(start) — extra Solr params are passed through getNamedParameters().
  • execute() returns a single-element List<QueryResponse>; from QueryResponse we read getResults() → SolrDocumentList (getNumFound() + iterate SolrDocument).
  • FieldUtils holds the indexed field names: TITLE, FULLNAME, REFERENCE, NAME, SPACE/SPACES, WIKI, TYPE, LINKS (outgoing links), HIDDEN, DATE, AUTHOR_DISPLAY, SCORE, *_sort variants for sorting, etc. Localized fields (e.g. title) are auto-expanded across supported locales by the executor.

Prior art for building/running a Solr query string and reading results: SOLRSearchSource (xwiki-platform-search-solr-rest).

Proposed module

New sibling module under xwiki-platform-livedata:

xwiki-platform-core/xwiki-platform-livedata/xwiki-platform-livedata-solr/
  pom.xml                         (artifactId: xwiki-platform-livedata-solr)
  src/main/java/org/xwiki/livedata/internal/solr/
    SolrLiveDataSource.java               @Named("solr")
    SolrLiveDataEntryStore.java           @Named("solr")
    SolrLiveDataPropertyStore.java        @Named("solr")
    SolrLiveDataConfigurationResolver.java (optional, see below)
  src/main/resources/META-INF/components.txt
  src/test/java/...

Dependencies: xwiki-platform-livedata-api, xwiki-platform-query-manager (Query/QueryManager/SecureQuery), xwiki-platform-search-solr-api (for FieldUtils and Solr field constants), SolrJ (transitively), xwiki-platform-model-api. Register the module in xwiki-platform-livedata/pom.xml <modules> and add it to the WAR/flavor as the LiveTable connector is.

@since should be the current SNAPSHOT version from the root pom (currently 18.5.0-SNAPSHOT → @since 18.5.0RC1), not the value in CLAUDE.md.

Source parameters (macro-facing)

Parameter Meaning v1
query Arbitrary Solr query string (the “search query”), e.g. links:"xwiki:Sandbox.WebHome" for backlinks yes
fq Optional extra Solr filter queries nice-to-have
wiki Restrict to a wiki (else current wiki) nice-to-have
(future) cql CQL query support later

Filtering to type:DOCUMENT and hidden:false (for users not showing hidden docs) is applied internally as fq, mirroring SOLRSearchSource.

Entry store: get(LiveDataQuery) algorithm

  1. Read query param from merged source parameters; if absent default to *:*.
  2. Query q = queryManager.createQuery(queryParam, SolrQueryExecutor.SOLR).
  3. ((SecureQuery) q).checkCurrentUser(true) → enforce view rights.
  4. Build fq: type:("DOCUMENT"); add hidden:(false) unless the user has programming rights; add any caller fq/wiki.
  5. Translate LiveDataQuery.filters → Solr fq clauses. Each Filter maps to one fq; constraints within a filter are joined by OR/AND per matchAll. Operator mapping (v1 minimal):
    • equals → field:"value"
    • contains → field:*value*
    • startsWith → field:value*
    • Values are escaped with Solr ClientUtils.escapeQueryChars to prevent query injection.
  6. Translate LiveDataQuery.sort → Solr sort, mapping the LD property id to the sortable Solr field (e.g. title → title_sort). Default sort configurable; for backlinks, by title.
  7. q.setOffset(query.getOffset()), q.setLimit(query.getLimit()).
  8. Execute; take response.getResults().
  9. For each SolrDocument, build an entry Map<String,Object> containing only the requested query.getProperties(), e.g.:
    • doc.title ← title (fallback to name/fullname if blank)
    • doc.url / link ← from REFERENCE resolved to a DocumentReference (reuse the solr DocumentReferenceResolver<SolrDocument>) → built into a view URL
    • additional fields as configured
  10. LiveData ld = new LiveData(); ld.setCount(results.getNumFound()); ld.getEntries().addAll(entries);

Known caveat — count vs. security filtering

SolrQueryExecutor filters results after Solr returns them, but getNumFound() reflects the pre-filter total. So count can be slightly higher than the number of viewable rows (same limitation the REST search has). Acceptable for v1; document it. A proper fix is a Solr PostFilter (out of scope, noted in SOLRSearchSource too).

Property store: get() (columns) — configurable via source params

Decision: columns are configurable through source parameters, defaulting to title + link.

  • Default descriptors when no column config is given:
    • doc.title — type String, displayer link (links to the page), sortable (title_sort), filterable (contains).
  • A source parameter (e.g. fields / columns) lets callers declare additional columns by Solr field name. For each declared field the property store emits a descriptor (id = the LD property, mapped to its Solr field + *_sort variant for sorting where one exists), and the entry store includes that field in the result mapping.
  • This keeps the generic source minimal while letting specializations (e.g. backlinks, or any future table) tailor the visible columns without new Java. Later iterations can derive descriptors dynamically from the Solr schema / FieldUtils (types, sortability) instead of relying on the caller.

Backlinks specialization (the XWIKI-22733 payoff)

No new Java needed beyond the generic source. The backlinks UI (e.g. a panel or the page “Information” area) declares a Live Data using:

  • source = solr
  • query = links:"<serialized reference of the current document>"
  • columns = title (as link)

This replaces the non-scaling backlinks query with a paginated, sortable, filterable, rights-aware Solr-backed table. Optionally wrap it in a small macro/template that injects the current doc reference into the query parameter.

Testing

  • Unit tests for the entry store (mock QueryManager/Query/QueryResponse, assert the generated query string, fq, sort, offset/limit, and entry mapping) — mirror LiveTableLiveDataEntryStoreTest and SolrKeywordSearchSourceTest.
  • Unit test the filter/operator → Solr translation and escaping.
  • A functional @UITest (later) verifying a backlinks Live Data renders, paginates and sorts against a seeded Solr index.

Phasing

  1. v1: generic solr source — arbitrary query param, title + link columns, filter/sort translation, view-rights via SecureQuery, count from numFound. Wire up backlinks for XWIKI-22733.
  2. v2: richer/configurable columns, dynamic descriptors from the Solr schema, more operators, multi-wiki, CQL input.
  3. v3 (if needed): PostFilter-based accurate counts; option to load documents for column metadata reuse.

Decisions (confirmed)

  • Module: xwiki-platform-livedata-solr, placed under xwiki-platform-livedata (sibling of -livetable).
  • v1 columns: configurable via source parameters, defaulting to title + link.
  • Backlinks: this issue (XWIKI-23940) delivers the generic Solr source only; the backlinks UI wiring is done separately under XWIKI-22733.

Let me know if you see something wrong. Main issue I see ATM is the count (see above).

I’ll soon have a PR to show.

Thanks

2

We have org.xwiki.search.solr.SolrUtils which should be used instead of ClientUtils.

Filtering to type:DOCUMENT and hidden:false (for non-PR users) is applied internally as fq, mirroring SOLRSearchSource.

Have you reviewed this? The hidden status should of course be based on the user preference, not having PR. We have some old, bad code that uses PR for this, but it is clearly wrong.

Using wildcards in Solr is usually not a good idea as it completely bypasses the tokenizer, leading to weird bugs where queries don’t return what you would expect. I’ve tried doing something more clever for a wildcard at the end in SolrKeywordSearchSource, but it doesn’t respect per-language tokenizers currently, which is why I’m applying the filter to title_ (which also uses the generic tokenizer).

9. For each `SolrDocument`, build an entry `Map<String,Object>` containing only the requested `query.getProperties()`, e.g.:
   - `doc.title` ← `title` (fallback to `name`/`fullname` if blank)

Solr indexes the rendered title, so unless the title of a document is intentionally blank, there should be no fallback needed. Also, name/fullName is a bad fallback, you don’t want WebHome to be displayed as title.

This should use links_extended like the link store.

I don’t think it is a good design to let the caller guess the Solr field name - it will make it very hard to match this, e.g., to actual object properties to allow inline editing. I think this should rather use a syntax matching or inspired by property references that it then maps to Solr fields internally based on the same logic as the Solr indexer. I would rather not support object properties in a first version and then add proper support later.

3

What I’m missing in this design is how to support entities other than full pages. I think it would be very nice to be able to use this also for displaying lists of attachments or objects.

4

Thanks @MichaelHamann .

For everyone, these points were hopefully handled in the PR and the discussions has continued there.

Thanks