Random curl failures when installing on GitHub Actions

[ptmkenny]

A couple days ago, I upgraded from the legacy CLI to the new CLI.

I've noticed that the actions are failing fairly frequently (for example, in a series of actions that downloads the CLI 12 times, 2 failures) due to a curl error when downloading platformsh:

Run curl -fsSL https://raw.githubusercontent.com/platformsh/cli/main/installer.sh | bash
  curl -fsSL https://raw.githubusercontent.com/platformsh/cli/main/installer.sh | bash
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    PLATFORMSH_SITE_ID: ***
    GIT_LFS_SKIP_SMUDGE: 1
    COMPOSER_PROCESS_TIMEOUT: 0
    COMPOSER_NO_INTERACTION: 1
    COMPOSER_NO_AUDIT: 1
    BRANCH_NAME: prototype
    PLATFORMSH_DRUSH_ALIAS: @site-aliases.SITE.BRANCH
+--------------------------------------------------+
|                                                  |
|           Platform.sh CLI Installer              |
|                                                  |
+--------------------------------------------------+

Checking environment
  [*] Your kernel (linux) is supported
  [*] Your architecture (amd64) is supported
  [*] Using raw install method
  [*] cURL is installed
curl: (22) The requested URL returned error: 403
Error: Process completed with exit code 1.

Here's the step in the GitHub action, copied from the platformsh docs:

    - name: "Install platform CLI."
      shell: bash
      run: curl -fsSL https://raw.githubusercontent.com/platformsh/cli/main/installer.sh | bash

I can't remember getting such a failure with the legacy CLI, and if I did, I didn't get one every test run like I'm getting now.

Any ideas about how to debug this would be much appreciated.

[akalipetis]

@ptmkenny it's quite possible that you're running into GitHub's rate limit. Is it possible to use the following in your GitHub action?

        env:
          GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}"

This will inject the GitHub token paired with your CI runner in the environment and in turn, the installer will use it for downloading the CLI and circumvent the GitHub API rate limit.

You can see how the token is used here

cli/installer.sh

Lines 381 to 389 in 17e7009

	function github_curl {
	if [ -z "${GITHUB_TOKEN}" ]; then
	curl -fsSL $1
	return $?
	else
	curl -fsSL -H "Authorization: Bearer ${GITHUB_TOKEN}" $1
	return $?
	fi
	}

Then, this is called to get the latest version

cli/installer.sh

Line 181 in 17e7009

version=$(github_curl https://api.github.com/repos/platformsh/cli/releases/latest | grep "tag_name" | cut -d \" -f 4)

and to get the artifacts

cli/installer.sh

Line 399 in 17e7009

if ! github_curl $url > "${tmp_dir}/${tmp_name}.tgz"; then

[ptmkenny]

Thanks that makes sense. I didn't realize the installer needed auth , so I'll try running it with the token then. I won't be able to test for a couple days but I'll confirm when I get the chance.

[lolautruche]

Maybe we should consider hosting the binaries elsewhere to avoid this issue, like for the legacy CLI.

[akalipetis]

Maybe we should consider hosting the binaries elsewhere to avoid this issue, like for the legacy CLI.

Hosting on GitHub simplifies things a lot, but we could consider falling back to an alternative when we're in a CI system without a GITHUB_TOKEN.

[rfay]

I'm a fan of hosting on github, and releases can be accessed without a token, but there is a strict limit if you don't have a token going. However, you can use https://nightly.link/ to get around this. DDEV does it so people can access without those kind of issues, and it works for github-attached zipballs as well.

[lolautruche]

We could also host it on Platform.sh 🙂

[rfay]

Yes, of course. that introduces build/push complexity, but definitely solves this class of problem.

[akalipetis]

@ptmkenny has your issue been resolved after adding the environment variable?

[ptmkenny]

@akalipetis Thanks, I did manage to get this working with an environment variable. But, because I set up the build environment inside an action instead of inside the workflow directly, I have to pass the environment variable as an input to the action; accessing secrets directly within an action is not supported. So switching from the legacy CLI to the new one added ~20 lines of code to my workflow passing the token around.

It would be great if the current version warned the user when the installer was called without GITHUB_TOKEN that you might actually want to provide such a token, especially because the "API rate limit exceeded" message is hidden by a generic 403, which doesn't help the user figure out what they need to do.

[akalipetis]

Thanks @ptmkenny - this would be a nice improvement indeed!