Replace LGTM by CodeQL analysis on GitHub?

Development
,
1

Hi Devs,

We’ve received a new PR (Add CodeQL workflow for GitHub code scanning by lgtm-com[bot] · Pull Request #1978 · xwiki/xwiki-platform · GitHub) to set up a GitHub Action workflow to set up GH code scanning using CodeQL analysis.

My understanding is that we would need to remove the configured LGTM GitHub app (on all 3 xwiki repos) and then configure “code scanning” on the 3 repos and use CodeQL. I guess we will need to check how to port our xwiki-platform/.lgtm.yml at master · xwiki/xwiki-platform · GitHub (and remove it). The config for codeql scanning is at Customizing your advanced setup for code scanning - GitHub Docs

We need to decide if that’s what we want but it seems like a good idea to me and the way forward (since it’s LGTM that integrated with GitHub).

We could try to set it up and see what we get before removing anything related to LGTM.

WDYT?

Thanks

2

+1

Note that according to this announcement, LGTM.com will be shut down on the 16th of December, so there is not much of an alternative to the new code scanning setup.

3

+1

thanks

4

+1

5

So, we also have another option which is to not use LGTM and instead rely on SonarQube:

  • We started LGTM as an experiment to see what it would bring us. I’m not sure it has brought us a lot.
  • SonarQube is what we are standardizing more on and we worked to specify the rules we want to apply and we maintain it.

SonarQube also has a GH integration (see GitHub Code Quality & Security CI/CD Integration Pipeline Workflow for SonarCloud and SonarCloud Scan · Actions · GitHub Marketplace · GitHub)

Right now we use SonarQube through our build from all the branches we build using Jenkins. We currently don’t build PR branches.

We have 2 choices if we want to get SonarQube validation for PRs:

  1. Enable building PR branches on Jenkins. We could decide that for platform we wouldn’t run the docker tests on multiple environments for example, so that the full build takes less time. We could even decide that we don’t run integration-tests to win even more time.

  2. Enable building PR branches with the SonarQube GH integration. This means adding a GH Action like sonarcloud-github-action-samples/.github/workflows/build-workflow.yml at maven · SonarSource/sonarcloud-github-action-samples · GitHub to our repos. We could configured it to only build for PRs and not for pushes on master/supported branches.

Pros for 1)

  • Same build infra for everything. Simpler for maintenance and nicer architecture (more homogeneous)
    Cons for 1)
  • The quality of the build is better since we run everything that is in our Jenkins pipeline (quite a lot of lines). It’s a real build.
  • No close GH integration, i.e. no message directly in the PR to say what problems were found. Now, maybe there’s a Jenkins integration with GH that exists somewhere and that could report the status of the build in GH. I would be surprised if that didn’t exist.

Pros for 2)

  • Close GH integration
  • Uses GH infrastructure and thus more time available for our agents. Cons: maybe GH will stop the build after N minutes and that could be an issue for building platform. Same problem for memory requirements. If that’s a problem we could use custom runners with GH but then we don’t win time for our agents.
    Cons for 2)
  • Another build to support (the GH Action yml) and when we change something in our CI build, we need to also backport it to the GH Action.
  • Less good quality build results. It’s a partial build that has the minimum to execute SonarQube.
  • More complex architecture/setup.

So we have 2 questions to answer here:
A) Use SonarQube instead of LGTM
B) Run SonarQube for PRs on Jenkins (option 1) or with GH Actions (option 2)

I think my preferences are:
A) Continue standardizing on Sonarqube and drop LGTM.
B) option 2 (even if it’s a less good build) because our agents are quite busy and this offloads work. However, it’s very possible we’ll hit free limitations and in this case, we’ll need to move to option 1 if that happens.

Note: I guess I wouldn’t be opposed to continue using both SonarQube and LGTM but only with option B2 since otherwise it’s too much for our agents.

WDYT?