Need help with 429 errors for GitHub links

[wolf99]

We have the lychee action on our repo for a long time but have deactivated it as we started getting too many false positives.
All Github links give 429, all gnu.org links give a TIMEOUT
Probably I've messed up the config somewhere but I cant see what.
Would love some help.

• Config is directly in the workflow here, see the "Check links" step: https://github.com/exercism/c/blob/retry-link-checking/.github/workflows/checks.yml
• The related "Checks" job is failing on this PR: WIP: Retry link checking exercism/c#1154

[thomas-zahner]

Thanks for reporting the issue. It looks like you probably have seen the documentation page about rate limits already. Unrelated to this issue, I personally can recommend that you create a lychee.toml file config, which might simplify the configuration of lychee.

I notice you use --max-retries 0 and the documentation also mentions that. But setting it to zero also means that any rate limits are immediately reported instead of retried. Can you try to set it to something like 3? Additionally, increasing --retry-wait-time to something like 5 might help. I've also encountered problems with gnu.org in the past, which is why I've excluded it in the zed project.

Also @mre is working on #1844 which should improve the current rate limiting situation by a lot.

[wolf99]

Thanks for the suggestions @thomas-zahner. I've aded your suggestions which has reduced the issue, but there are still some 429s for Github links remaining.
Im guessing that this is where the checking ticks over some rate limit? But that seems odd because the repo does not have a massive number of links in total (121 reported by lychee)? I know lychee is used on repos with many more links without issue.

[wolf99]

Also quite puzzled why Github links are seeing rate limiting when we have the token configured... is that config somehow incorrect or not taking effect?
(I have confirmed the name of the token is correct for the repo in question).

[wolf99]

We have lychee also used in same way on another repo in the same org here with no problems and about 2x the number of links 🤷

https://github.com/exercism/problem-specifications/blob/main/.github/workflows/ci.yml

[thomas-zahner]

Yes, that seems quite weird. Have you tried to use the environment variable approach instead? That's the only apparent difference I can spot between the two repositories.

        env:
          GITHUB_TOKEN: ${{secrets.GITHUB_TOKEN}}

[wolf99]

Thanks for the suggestion @thomas-zahner unfortuantly, that approach was what was in place at the start of my PR.

[katrinafyi]

I think the github token isn't changing the outcome because the token's applicability is fairly limited. It's used to create a Github API client which is then used to check URLs of the form https://github.com/owner/repo. In particular, this can't check file existence because the REST API doesn't have an endpoint for files within a repository. The code is here.

I can come up with some workarounds that seem to work from my computer.

• Adding a --header 'Authorization: anything' argument seems to relax the Github rate limit, even if the authorization is bogus. However, this applies to all URLs and some other websites will reject the invalid token (namely, wikipedia).
• Since you have a local checkout of the repo, you could remap links to repo files to use the local files on disk: --remap "^https://github\.com/exercism/c/blob/master file://$(pwd)". This will work but it's limited to URLs in the current repo (so you might still exceed the rate limit with other URLs) and it could make the errors confusing, as the errors display the file:// URL.
• You could remap files to use the raw.githubusercontent domain. This works for all repos on Github, but it also changes the URL which appears in error messages: --remap "^https://github\.com/exercism/c/blob/master https://raw.githubusercontent.com/exercism/c/refs/heads/master" --remap "^https://github\.com/exercism/v3/blob/master https://raw.githubusercontent.com/exercism/v3/refs/heads/main"