Pipelines and the UI don't reflect reality when git push --force is used

[mathrick]

I've done fairly extensive research and failed to find any definitive answer for how to deal with a not uncommon scenario of a resource producing a version which has already been seen, but which nevertheless represents a change from its previous state. In particular, this happens in the following situation:

• git resource is set up to monitor branch somebranch, and job tests is triggered by it
• somebranch initially points to commit 01abcdef. tests runs for 01abcdef and passes in build 1
• somebranch is pushed to, and now points to deadbeef. tests runs for deadbeef and fails in build 2
• somebranch is rolled back to 01abcdef (last known good revision) via git push --force
• because the git resource produces ref: 01abcdef which has been seen previously, tests is not triggered again, and the UI continues to show tests as having failed, with no way for the user to correct it
  • using "re-run with the same input" option in the UI on build 2 will trigger build 2.1 for deadbeef, not 01abcdef
  • manually triggering a new build 3 will likewise use deadbeef
  • even re-running build 1 doesn't work, as it will produce a passing build 1.1, but the UI will continue showing failure for build 2.1 and/or build 3
  • and shockingly, even changing the definition of tests and triggering a new build still won't re-run it with the current state of somebranch but will stubbornly trigger on deadbeef, despite the fact that the current version of tests has never run on 01abcdef, so there's no conceptual reason to skip it

There seems to be very little guidance on how to resolve this. concourse/git-resource#174 simply said in 2019 this "should be" resolved by concourse/rfcs#1 and #1707, but in 2026 this is very much an issue when running Concourse v8.1.0. There are some other tangentially related issues and discussions, but none of them actually address this situation or give anything in the way of concrete steps that would fix things.

[taylorsilva]

How do you "rollback" the branch to the last good revision? It sounds like you're dropping the bad commit from your branch history (git reset or similar set of commands?).

Concourse doesn't have an automatic way to handle "the version disappeared". You can workaround this by doing fly clear-versions, which will delete all versions. You could also disable the version with fly disable-resource-version, or in the web UI:

Neither of those would cause the job to re-trigger though.

Another way to handle this, instead of dropping the commit from the branch, could be to revert the bad commit(s). This way you continue to make new commits on the branch (revert commits) which the git resource will discover and trigger downstream jobs as expected. Also creates a nice history of everything happening on the branch, like an audit trail.

---

using "re-run with the same input" option in the UI on build 2 will trigger build 2.1 for deadbeef, not 01abcdef

This is as expected. You should click on build 1 and then press the re-run button. I see that you figure that out farther down and find out that re-run builds don't appear as the "latest" build if newer versions have been discovered.

and shockingly, even changing the definition of tests and triggering a new build still won't re-run it with the current state of somebranch but will stubbornly trigger on deadbeef, despite the fact that the current version of tests has never run on 01abcdef, so there's no conceptual reason to skip it

Depends what you changed in the tests job, but if the input resources haven't changed then it makes sense that you'll continue to see the same triggering behaviour. It sounds like you're stuck with deadbeef, and should probably disable the version as I previously showed.

You could also rename the job if you want to reset the job's history. Job history is tracked by the name of the job. That feels like overkill though and I don't recommend it.

[mathrick]

How do you "rollback" the branch to the last good revision? It sounds like you're dropping the bad commit from your branch history (git reset or similar set of commands?).

Yes, that's what's happening.

You can workaround this by doing fly clear-versions, which will delete all versions. You could also disable the version with fly disable-resource-version, or in the web UI

This is very manual though, in addition to being potentially incorrect (the version in question might still exist on other branches).

Another way to handle this, instead of dropping the commit from the branch, could be to revert the bad commit(s). This way you continue to make new commits on the branch (revert commits) which the git resource will discover and trigger downstream jobs as expected. Also creates a nice history of everything happening on the branch, like an audit trail.

Sure, but that is a completely different set of operations. There are absolutely situations in which it's the correct thing to do, but that should be dictated by the version control semantics that are desired in the particular case, not by the CI system just deciding it will forever display falsehoods for pretty arbitrary reasons that have nothing to do with version control operations as such.

This is as expected. You should click on build 1 and then press the re-run button. I see that you figure that out farther down and find out that re-run builds don't appear as the "latest" build if newer versions have been discovered.

Why is this expected though? I get that Concourse wants to be idempotent, but "avoid redoing operations with identical inputs" is not the same as "claim things that are at odds with reality with no recourse for the users". I could understand if the outcome was Concourse reverting to showing the results of build 1 without actually re-running it. A little surprising maybe, but logically consistent. But why is showing outdated information that was arbitrarily decided to be "more latest" than the actual latest state expected? Especially since it actively conflicts with what I understand is the other motivator behind Concourses design, i.e. statelessness. It's not like git commit SHA1s have any inherent ordering, it's just an accident of state of which one happened to be returned first. So time and state are absolutely a dimension, but a hidden one so the users just lose the ability to influence it.

Depends what you changed in the tests job, but if the input resources haven't changed then it makes sense that you'll continue to see the same triggering behaviour. It sounds like you're stuck with deadbeef, and should probably disable the version as I previously showed.

But that's not correct! The current version of tests never actually ran with the 01abcdef, so skipping the execution makes no sense, regardless of how idempotent Concourse is trying to be. "What definition of the job are we using?" is an input and I'd argue that Concourse should treat it as such.

[mathrick]

Apologies, I fat-fingered the response so it was sent prematurely. Here's the missing part that I intended to include:

When you say "this is expected", do you mean "this is how I envision Concourse working according to its principles, so even the platonic ideal version would behave that way", or do you mean "this is how the current code happens to behave, and I'm very familiar with it so it didn't surprise me"? If the latter, it still feels like a bug (or at least suboptimal behaviour), since I don't understand what set of principles would make it actually desirable.

I do understand the general idea behind wanting the system to be idempotent and stateless, but to my novice eyes, it doesn't seem especially useful to be idempotent and stateless in this specific way, especially since it's not actually either, as I mentioned above. And it's certainly oddly limiting and prevents Concourse from being practically useful in what to me feels like a very run of the mill scenario. And that's a shame; I was really looking forward to learning and using properly, as I really love a lot of the design principles (I especially appreciate that you understand that we need to debug pipeline failures just as much as any other kind of code, and give us actual tools to reproduce and debug failure locally).

Forgive me if this is something you've already considered and discussed, but wouldn't it make more sense to track state(lessness) and purity as explicit properties of jobs and pipelines and give the users the tools to influence those, so that "what happens if things change?" becomes a thing that users need to reason about in terms of correctness rather than being caught off guard by (à la type checking in Haskell)? It would actually be pretty cool to have a system that understands the distinction between the different kinds of inputs and results (e.g. building a binary is not actually idempotent because most compilers embed date information in the result in some form, but I want to treat it that way because binaries get signed and distributed and we don't want them to change).

[taylorsilva]

So much to read! Honestly, having a little trouble fully understanding some parts of your response. I don't want us to talk past each other though, so doing my best to fully engage here :)

It does sound like you're trying to mesh how you expect things to work in Concourse vs how it actually works. It feels like it's really frustrating you; my answer was focused on trying to help you work past the current problem you were facing and I didn't spend any time explaining the why of parts of the system. Happy to get into that a bit more now.

The main thing that's causing friction for you appears to be resources and how they work. You can read about the resource interface here: https://concourse-ci.org/docs/resource-types/implementing/

The discovery of "versions" is what you seem to have a problem with. I'll explain how this works.

When a resource check runs to find new "versions", it returns a list of versions from oldest to newest. It is up to the resource how many "versions" it returns and what each "version" represents. We're using "version" in an extremely generic way; a version can represent anything and is technically a map of key-value strings.

So only the resource itself knows what each "version" represents. Concourse (the code in concourse/concourse) doesn't know anything about what a "version" represents. Concourse doesn't know that a resource is tracking a git repo, a container image, files in an S3 bucket, etc.

Concourse takes that list of versions and merges it with the existing version history. The resource interface doesn't provide a way for resources to delete old versions. This is where Concourse assumes immutability in version history.

To make things a bit more concrete, let's walk through an example of what probably happened with your git resource.

---

Your somebranch has commit 01abcdef as HEAD, so the git resource returns:

[
  {
    "ref": "01abcdef"
  }
]

In the web UI, version history looks like:

• ref: 01abcdef

Build 1 runs with that commit. The next commit is pushed and git resource returns:

[
  {
    "ref": "01abcdef"
  },
  {
    "ref": "deadbeef"
  }
]

Concourse merges that with the existing version history, adding deadbeef and ignoring the previously found version 01abcdef.

In the web UI, version history looks like:

• ref: deadbeef
• ref: 01abcdef

Build 2 runs. It fails. You do git reset to drop deadbeef from somebranch. The git resource runs its check again and returns:

[
  {
    "ref": "01abcdef"
  }
]

Concourse merges that with the existing version history and does nothing because 01abcdef already exists.

In the web UI, version history still looks like:

• ref: deadbeef
• ref: 01abcdef

Now you are at the point where the version history does not match reality anymore.

---

As you've noted, Concourse expects "version" history to behave in an immutable way. We give you some escape hatches for this, like disabling versions or clearing the version history. You're right, these aren't perfect solutions, but it's what we have.

We all work in different ways, but Concourse does try to push you towards working in an idempotent and immutable way. It's definitely the kind of workflows that Concourse works best with and what we think leads to easier software project maintenance (this is an OPINION, I'm not saying it's an absolute truth! There are 100% valid reasons to NOT work in an immutable way.).

So when I said "this is expected", it was in the sense that the code is aligned with Concourse's principles.

I hope all that helps you understand the "why" of Concourse a bit better. Your post did make me think "how could we make this easier?" for use-cases like this. The only idea that immediately came to me was to add an option to resources that would reset version history on every check of the resource. That comes with its own set of cons though, so I wouldn't go implementing that.

Good luck on your CI journey and happy to continue discussing or answer any other questions you have.

[mathrick]

@taylorsilva: thank you very much for the detailed answer, I really appreciate your time!

The model you describe is pretty much what I thought it was, so I'm glad to know I wasn't too far off in my assumptions.

Concourse takes that list of versions and merges it with the existing version history. The resource interface doesn't provide a way for resources to delete old versions. This is where Concourse assumes immutability in version history.

Right, and that I think is the issue at hand. Concourse assumes a meaningful, ordered, and unique history, and assigns importance to the implicit and arbitrary timestamps at which these version were first returned by the associated resource. But that is not, in general, a valid assumption -- unlike say svn, which actually provides a well-defined and linear history where each element will occur exactly once, there is no reason to expect the random collection of "all the heads all the refs have ever been at in a particular git repo" to form a meaningful history. They need neither be ordered nor unique (in other words, it's a multiset).

The underlying problem seems to have manifested before, for example in concourse/git-resource#174, and from what I've been able to figure out, has not been resolved since, given that both concourse/rfcs#1 and concourse/rfcs#24 were closed without being adopted. I haven't tested it -- will Concourse still silently skip the newest git ref on a branch if the same ref has previously been seen on a different branch?

I might of course be missing some complications here, but it seems that a relatively simple change that would resolve a lot of these problems would be to make Concourse accept a new version every time there is a change from the previous most recent version, even if that specific version has been seen before. I think this could be done without touching the other assumptions Concourse makes, including idempotency -- IMHO even if it reused previous results without actually re-running the build, it would still be a significant improvement over the current situation (especially if the UI also makes it clear that the results were reused).

Additionally, as I mentioned above, it does feel to me that Concourse should not actually assume idempotency when the version to be reused was built with a different job config than the one currently in effect, since it's not actually correct to say that it has previously been built by the current job. There is a previous build result, yes, but by a different job, albeit under the same name. But that would be a much more invasive change to how Concourse works and I can see it breaking real-world scenarios if users rely on the current behaviour (e.g. when the job is only "pseudo-idempotent" and has some significant side-effects such as publishing a new version), necessitating a much more careful approach to avoid unintentionally breaking compatibility.

We all work in different ways, but Concourse does try to push you towards working in an idempotent and immutable way. It's definitely the kind of workflows that Concourse works best with and what we think leads to easier software project maintenance (this is an OPINION, I'm not saying it's an absolute truth! There are 100% valid reasons to NOT work in an immutable way.).

Absolutely, I'm a big fan of making things immutable and idempotent whenever possible. But I do feel that it is not really possible to simply assume that in a CI system, at least not without running into some hard walls in scenarios which would otherwise be reasonably expected to work (this is of course my opinion that can be disagreed with).

That's why I mentioned some system for the user to communicate the level of idempotency that is expected from a particular step. This would be conceptually similar to how PostgreSQL's user-defined functions can be marked PARALLEL SAFE, PARALLEL RESTRICTED, or PARALLEL UNSAFE. Obviously any such system would be a major change (and also something of a research project in its own right), so I'm not exactly expecting anyone to adopt it here and now, it's meant more as a potential start of a discussion.

[taylorsilva]

I might of course be missing some complications here, but it seems that a relatively simple change that would resolve a lot of these problems would be to make Concourse accept a new version every time there is a change from the previous most recent version, even if that specific version has been seen before. I think this could be done without touching the other assumptions Concourse makes, including idempotency -- IMHO even if it reused previous results without actually re-running the build, it would still be a significant improvement over the current situation (especially if the UI also makes it clear that the results were reused).

I'm not able to see how a change in Concourse could allow for this because I don't see how a resource signals to Concourse that there was some change to the version history.

That said, I think there might be a way to fix this in the git resource. I think having the resource check if the previous version still exists in the version history, and if it doesn't, emit the current HEAD ref with a timestamp, which would force it as the new version. I think something like that would work and resolve the issue you've raised here.

---

I have nothing else to add here. I liked reading your thoughts and opinions here. Really nice feedback to read. I'll try and keep some of your thoughts here in mind. TY for the thoughtful discussion!