[JENKINS-75902] Don't save incomplete pipeline configs on GitLab API errors while fetching project info

[Pitxyoki]

Please see JENKINS-75902 for the original issue description.

Several safety assurances are added to the GitLabSCMSource class:

1. GitLabApiExceptions are allowed to be thrown instead of hidden by IllegalStateExceptions on the getGitlabProject() methods.
2. The call to getGitlabProject(gitLabApi), at the beginning of the retrieve() method, is enclosed in a dedicated try-catch. This prevents the config.xml from being saved with incomplete data due to any communication error with GitLab. This should avoid the "loss of pipeline triggering" as described in JENKINS-75902.
3. Other methods (retrieveActions() and createProbe()) that communicate with the GitLab API in a similar way were also changed, but just to throw IOExceptions in a non-breaking way, compatible with their inherited declarations.
4. Additionally, I add Javadoc to the functions I touched, helping to guide future developers looking at this code.

There are several line changes where only whitespace/indentation is changed. Unfortunately, GitHub highlights the whole lines. You may be able to see the diff more properly in a better diff editor or your IDE, setting it to ignore whitespace changes.

Please note that these changes were the result of some careful investigation on our part (I work for Feedzai) and we think we have headed in the right direction. We would appreciate any guidance if you find something is not right in our reasoning and/or the patch is inappropriate.

This seems to be related to:

• [JENKINS-60304] Branches and merge requests disappear when calls to GitLab fail
• [ENKINS-60304] Error handling corrected, retry added on all API calls #63
• [JENKINS-60338] Fix the problem that the project id cannot be saved #65
• [JENKINS-60304] Forward GitLabApiException as IOException #88
• I suspect this also fixes this: Branches being disabled in multi branch pipelines since update to Gitlab 13.12.4 #147

Testing done

Testing this involved deploying the patched plugin onto our running Jenkins and leaving it there for one month.

We have not seen the original issue occur again ever since (we have, however, seen a similar occurrence associated to another plugin [cloudbees-folder] — we will open a respective issue+pull request for that as well).

Since we deployed the patched plugin, we have had GitLab become unreachable, but the problem of the pipelines not being triggered (and their config.xml files inconsistent) did not manifest.

Given the complexity of simulating GitLab failures when this plugin communicates with it, I did not implement automated tests. If there is a straightforward way to implement that, I'll appreciate it if you can guide me on how to do it.

Submitter checklist

• Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
• Ensure that the pull request title represents the desired changelog entry
• Please describe what you did
• Link to relevant issues in GitHub or Jira
• Link to relevant pull requests, esp. upstream and downstream changes
• Ensure you have provided tests that demonstrate the feature works or the issue is fixed

[Pitxyoki]

Hello @mifitous, @basil, @gounthar. I saw in #479 that this repository does not have active reviewers.
Is there any chance you may still take a look and provide any inputs about this?

Thank you!

[basil]

Hi @Pitxyoki, unfortunately we don't have any developers actively reviewing incoming pull requests for this plugin. We don't want your pull request to be in limbo forever; you can step forward to become a maintainer of this plugin following the instructions here:

https://www.jenkins.io/doc/developer/plugin-governance/adopt-a-plugin/

[Pitxyoki]

Note that this change was not enough - we saw re-occurrences of the original issue.

We are now testing with the extra changes added in Pitxyoki#1.

[Pitxyoki]

Hi @Pitxyoki, unfortunately we don't have any developers actively reviewing incoming pull requests for this plugin. We don't want your pull request to be in limbo forever; you can step forward to become a maintainer of this plugin following the instructions here:

https://www.jenkins.io/doc/developer/plugin-governance/adopt-a-plugin/

Hello @basil,

I might very well step forward and do that. My main motivation is to get this and related changes into the plugin and release a fixed version, but I may as well help with approving minor changes such as non-functional code quality or very small improvements.

I foresee, however, that I will need:

• Some guidance on the architecture of this plugin and how it integrates with others. I did quite some investigative work, but I saw there is also a quite sizeable amount of more information available. An introduction to how this integrates with the job-dsl-plugin, cloudbees-folder, multibranch-pipeline, and other plugins might be useful. This ties with my initial request that someone checks the validity of my proposed change in this PR.
• Guidance on how releases are performed:
  • Timing: when can a version come out? Is this scheduled? Synchronized with some other plugins also releasing? Or is it fully controlled by the maintainer?
  • Pipelines/validations performed before a release: what are they, and how should I navigate through them?

How can we coordinate this more efficiently?

Thank you!

[basil]

I might very well step forward and do that. My main motivation is to get this and related changes into the plugin and release a fixed version, but I may as well help with approving minor changes such as non-functional code quality or very small improvements.

Hi @Pitxyoki, thanks for your interest in adopting this plugin. That motivation is well aligned with that of other new plugin maintainers.

Some guidance on the architecture of this plugin and how it integrates with others. I did quite some investigative work, but I saw there is also a quite sizeable amount of more information available. An introduction to how this integrates with the job-dsl-plugin, cloudbees-folder, multibranch-pipeline, and other plugins might be useful.

As far as integration with Job DSL's dynamic DSL and Pipeline jobs, I think this is the best documentation we have:

• https://www.jenkins.io/doc/developer/plugin-development/pipeline-integration/
• https://www.jenkins.io/doc/developer/forms/structured-form-submission/

I don't think we have any explicit documentation about integration with the Folders plugin. The GitHub Branch Source plugin should be considered a role model here. Probably many questions can be answered by checking to see how something is implemented there.

This ties with my initial request that someone checks the validity of my proposed change in this PR.

Our main concern is not introducing regressions and maintaining backwards compatibility with existing data. For the first, running it on your own Jenkins controller is the best way to make sure there are no regressions. For the second, documentation is here:

• https://www.jenkins.io/doc/developer/persistence/backward-compatibility/

If you aren't introducing any changes to on-disk saved state, then you don't need to worry about XStream compatibility.

Guidance on how releases are performed:
Timing: when can a version come out? Is this scheduled? Synchronized with some other plugins also releasing? Or is it fully controlled by the maintainer?
Pipelines/validations performed before a release: what are they, and how should I navigate through them?

This already has automated plugin release enabled as documented here:

• https://www.jenkins.io/doc/developer/publishing/releasing-cd/

So you can simply apply a label to the PR, merge it, and a new release will be created automatically.

Let me know if I can help with anything else. You can also join the jenkinsci-dev mailing list here:

• https://www.jenkins.io/mailing-lists/