Brainstorming: keeping files identical across our repos (.gitignore and friends)

Hello,

Context

Some files must be identical across several of our repos: .gitignore is the obvious one (xwiki-platform, xwiki-commons, xwiki-rendering, and most of xwiki-contrib), but the same need exists for things like .editorconfig, SECURITY.md or any future shared config. Today each repo owns its own copy and they drift silently.

Note that the .github special repo does not help here: it only shares a fixed list of GitHub community-health files (issue/PR templates, CODE_OF_CONDUCT, CONTRIBUTING, SECURITY, SUPPORT, FUNDING, profile README). .gitignore is a Git client file, it must physically exist in every clone, and gitignore syntax has no include directive. So there is no “inheritance” available, only synchronization.

How bad is it today?

I sampled xwiki-contrib (369 public repos). Out of 80 repos checked, 73 have a .gitignore that differs from the canonical one and 7 match. And the drift is not legitimate local need, it is staleness and copy/paste:

  • application-changerequest carries xwiki-platform’s Solr zip and replication template paths, which mean nothing there.
  • macro-pdfviewer has a hand-written variant without the license header.

Good news: a canonical version already exists de facto. xwiki-commons, xwiki-rendering and xwiki-contrib/github-template-repository all ship the exact same file (blob 3d78cf1e). Only xwiki-platform adds a few extra blocks, of which 2 are path-specific to it.

Proposal (to discuss)

1. One canonical copy in xwiki/.github, e.g. under shared/, described by a manifest that says what is synced where:

files:
  - source: shared/gitignore
    target: .gitignore
    orgs: [xwiki, xwiki-contrib]
    select: has-file(pom.xml)
    mode: overwrite
  - source: shared/mvn/extensions.xml
    target: .mvn/extensions.xml
    repos: [xwiki/xwiki-commons, xwiki/xwiki-rendering, xwiki/xwiki-platform]
    mode: overwrite

Note that the scope is per file, not global: some files concern everything with a pom.xml, others only a handful of repos (see .mvn below).

2. Two propagation points, because a single one is never enough:

  • Creation time: the repo template. xwiki-contrib/github-template-repository already exists and already carries the right .gitignore, so repos created from it start correct.
  • Steady state: a scheduled job in xwiki/.github that enumerates the org through the API and fixes drift. This is what covers repos not created from the template, and also covers brand new repos the next night without anyone maintaining a list of repo names. Authentication through a GitHub App installed on the org with “All repositories” means repos created later are covered automatically.

3. Not all repos should be synced. This is the part I would most like feedback on. Three possible targeting strategies:

  • Explicit allowlist in xwiki/.github: precise, but new repos are never covered unless someone remembers to add them, which defeats the main purpose.
  • All repos by default + a repo-local opt-out marker (a file, or a GitHub topic): new repos are covered for free, and the repo owner stays in control without having to touch xwiki/.github.
  • Rule-based: sync only repos matching a criteria (not archived, not a fork, has a pom.xml, has a given topic…).

My preference is rule-based selection to pick the candidates, plus a repo-local opt-out marker as the override.

4. Write mode. At 369 repos, opening pull requests would create hundreds of PRs on repos with no active maintainer. I would rather have the job commit directly to the default branch, which is acceptable for this class of low-risk file, and keep PRs for anything riskier.

5. Escape hatch, otherwise people will fight the sync. Repo-specific ignores should go in a nested .gitignore in a subdirectory (this is how xwiki-platform’s Solr entries should live), and personal ones in .git/info/exclude. The sync then only ever owns the root file.

How do we prevent someone editing their local copy instead of the central one?

We cannot technically prevent it, Git has no server-side per-file ownership. But we can make it harmless:

  • A header comment in the synced file: “Managed centrally in xwiki/.github, edits here are overwritten, change it there”.
  • The sweep is self-healing: local drift is rewritten on the next run. Given that the current drift is stale copy/paste and not deliberate, overwriting is the right default here.
  • For the 3 core repos only, where branch protection already exists, we could additionally make it a blocking PR check (a reusable workflow that fails when the root .gitignore differs from the canonical one), so the change has to go through xwiki/.github first. That is not realistic for contrib, where a lot of repos are pushed to directly.

Another good candidate: .mvn

.gitignore is not the only case. The .mvn directory of the 3 core repos is already almost shared:

  • .mvn/extensions.xml and .mvn/develocity.xml are byte-identical in xwiki-commons, xwiki-rendering and xwiki-platform.
  • .mvn/develocity-custom-user-data.groovy is identical except that xwiki-platform appends ~20 lines that tag the build scan with the Docker functional test environment (servlet engine, database, browser). Those lines are a no-op when the corresponding system properties are not set, so the file could very likely be unified into a single shared version.

This is also a good illustration of point 3: xwiki-contrib repos have no .mvn directory at all, so this particular file set must be scoped to the 3 core repos only, while .gitignore is scoped to nearly everything. The mechanism therefore needs per-file targeting, not one global repo list.

Scope

I would keep this mechanism as a fallback for what no native mechanism already covers. Renovate presets already handle Renovate config, reusable workflows already handle CI, and .github inheritance already handles the community-health files. The sync is only for the leftovers, .gitignore being the first one.

Questions

  • Do you agree with the general direction, or do you see a simpler option I missed?
  • Which targeting strategy do you prefer for choosing which repos get synced?
  • Direct commit or pull request?
  • Beyond .gitignore and .mvn, which files would you want in the shared set?

Thanks!

I’m not too keen on manifests. Although they’re handy to automate things, they also add complexity (to a codebase that is already complex, furthermore). And sharing those files synced is not critical enough to warrant such a mechanism IMHO.

I think .gitignore could be shared with a common based in every repo, and then we can sub-.gitignore in relevant folders when needed.

e.g. some repos will have some JS content with one (or more) node_modules to exclude, while pure-Java repositories won’t. It would make sense to keep these exclusions in subfolders.

Can you explain in more details how it would work? I don’t understand what you mean. It seems like this is what exists today (except that we have a single .gitignore because we don’t need more).

I don’t think it makes sense to have several .gitignore. A single one is enough. It doesn’t matter if it contains ignores that are not used in a given project (actually you can never know if they are used or not since it depends what the user has lying around in their workspace). It lists what we don’t want committed and that can true whatever the repo.

If we want to have the exact same .gitignore files across all repositories and accept that these files will reference items that do not exist (such as node_modules in pure Java projects) then I have the opposite opinion and am in favor of using a programmatic synchronization mechanism.

I was thinking it would be weird to put some non-existing items in the gitignore, but after more thought I don’t think it would be that bad given it would enforce consistency across all codebases.