Fail mirroring more gracefully: #34002
Open
+73
−11
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
GiteaBot
added
the
lgtm/need 2
pull-request-size
bot
added
the
size/M
rremer
force-pushed
the
missing-lfs-fixes
branch
from
39f376f to
c6337ab
Compare
wxiaoguang
reviewed
Mar 25, 2025
services/mirror/mirror_pull.go
Outdated
Comment on lines
241
to
250
| case strings.Contains(stderrMessage, "unable to resolve reference") && strings.Contains(stderrMessage, "reference broken"): | ||
| case strings.Contains(stderrMessage, "remote error") && strings.Contains(stderrMessage, "not our ref"): | ||
| case strings.Contains(stderrMessage, "cannot lock ref") && strings.Contains(stderrMessage, "but expected"): | ||
| case strings.Contains(stderrMessage, "Unable to create") && strings.Contains(stderrMessage, ".lock"): | ||
| default: | ||
| return true |
There was a problem hiding this comment.
It doesn't seem right. Golang switch doesn't "fallthrough" by default.
It's better to create a test for this function, and add comments for each case: why it would happen and why it could be ignored.
There was a problem hiding this comment.
thanks for the feedback, I documented the existing logic in a test, along with the new cases of it I've found helpful at my company.
rremer
force-pushed
the
missing-lfs-fixes
branch
from
c6337ab to
62a85f3
Compare
* reuse recoverable error checks across mirror_pull * add new cases for 'cannot lock ref/not our ref' (race condition in fetch) and 'Unable to create/lock" * move lfs sync right after commit graph write, and before other maintenance which may fail * try a prune for 'broken reference' as well as 'not our ref' * always sync LFS right after commit graph write, and before other maintenance which may fail
rremer
force-pushed
the
missing-lfs-fixes
branch
from
62a85f3 to
1871276
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This handles a few cases where our very large and very active repositories could serve mirrored git refs, but be missing lfs files:
Case 1 (multiple variants): Race condition in git fetch
There was already a check for 'unable to resolve reference' on a failed git fetch, after which a git prune and then subsequent fetch are performed. This is to work around a race condition where the git remote tells Gitea about a ref for some HEAD of a branch, then fails a few seconds later because the remote branch was deleted, or the ref was updated (force push).
There are two more variants to the error message you can get, but for the same kind of race condition. These may be related to the git binary version Gitea has access to (in my case, it was 2.48.1).
Case 2: githttp.go can serve updated git refs before it's synced lfs oids
There is probably a more aggressive refactor we could do here to have the cat-file loop use FETCH_HEAD instead of relying on the commit graphs to be committed locally (and thus serveable to clients of Gitea), but a simple reduction in the occurrences of this for me was to move the lfs sync block immediately after the commit-graph write and before any other time-consuming (or potentially erroring/exiting) blocks.