Best way to install and run Concourse in production

[RichardBradley]

We have a commercial project using Concourse, and need a stable install, ideally as low maintenance as possible, with auto updates etc.

We have been using "control tower" for a long time, which installs Concourse on AWS with a web VM, worker VM and a RDS database using BOSH for self healing and includes auto-updated.

https://github.com/EngineerBetter/control-tower

The control-tower project is now no longer maintained and recently our system broke when AWS RDS updated their server certificates but the old certificates are hardcoded inside the control-tower source code (see EngineerBetter/control-tower#340 )

Does anyone have any advice on how we can continue to use Concourse on AWS?

I see one active public fork with a commit that looks related to the RDS cert issue:
InventiveDesigners/control-tower@f386eaa

Does anyone know how we might update our control-tower install to follow that fork or our own fork with similar changes?

I see that there is another BOSH based project at https://github.com/concourse/concourse-bosh-release , but it doesn't have a guide on how to install it. I'm not very familiar with BOSH. Should we be trying to move to that?

Thanks,

Rich

[RichardBradley]

We have been able to continue using control-tower for now, by using a fork based on InventiveDesigners/control-tower@f386eaa

While it's a pain that control-tower is no longer maintained, we can hopefully keep it running ourselves and I prefer that approach to the others suggested at https://github.com/concourse/concourse?tab=readme-ov-file#installation

[corinja]

We have been able to continue using control-tower for now, by using a fork based on InventiveDesigners/control-tower@f386eaa

While it's a pain that control-tower is no longer maintained, we can hopefully keep it running ourselves and I prefer that approach to the others suggested at https://github.com/concourse/concourse?tab=readme-ov-file#installation

Hi @RichardBradley your comment is very interesting to me as we are in the same situation. We have been using control-tower to deploy & maintain Concourse in our small team for a number of years, also hosted on AWS.

We managed to get over the situation with the AWS RDS CA by directly hacking the CONCOURSE_POSTGRES_SSLMODE using bosh and setting its value to require instead of verify-ca which allowed us to bypass the CA checking and continue running Concourse with the new RDS CA. However, this has stopped working for some reason.

We cannot redeploy our Concourse using control-tower currently for reasons we don't understand that result in an error exec format error

I thought to use the fork of control-tower you identified to do a fresh installation. Everything seemed to be working ok until I hit an error:

GENERATING BOSH DIRECTOR CERTIFICATE (18.130.57.111, 10.0.0.6)
2024/11/07 14:59:57 [INFO] acme: Registering account for [email protected]
2024/11/07 14:59:57 [INFO] [new-ci.services.mydomain.com] acme: Obtaining bundled SAN certificate
2024/11/07 14:59:58 [INFO] [new-ci.services.mydomain.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/426870947637
2024/11/07 14:59:58 [INFO] [new-ci.services.mydomain.com] acme: Could not find solver for: tls-alpn-01
2024/11/07 14:59:58 [INFO] [new-ci.services.mydomain.com] acme: Could not find solver for: http-01
2024/11/07 14:59:58 [INFO] [new-ci.services.mydomain.com] acme: use dns-01 solver
2024/11/07 14:59:58 [INFO] [new-ci.services.mydomain.com] acme: Preparing to solve DNS-01
2024/11/07 14:59:58 [INFO] Wait for route53 [timeout: 10m0s, interval: 30s]
2024/11/07 15:01:00 [INFO] [new-ci.services.mydomain.com] acme: Trying to solve DNS-01
2024/11/07 15:01:00 [INFO] [new-ci.services.mydomain.com] acme: Checking DNS record propagation using [8.8.8.8:53]
2024/11/07 15:01:30 [INFO] Wait for propagation [timeout: 10m0s, interval: 30s]
2024/11/07 15:01:34 [INFO] [new-ci.services.mydomain.com] The server validated our request
2024/11/07 15:01:34 [INFO] [new-ci.services.mydomain.com] acme: Cleaning DNS-01 challenge
2024/11/07 15:01:34 [INFO] Wait for route53 [timeout: 10m0s, interval: 30s]
2024/11/07 15:02:36 [INFO] [new-ci.services.mydomain.com] acme: Validations succeeded; requesting certificates
2024/11/07 15:02:39 [INFO] [new-ci.services.mydomain.com] Server responded with a certificate.
Error loading state file after create-env: [open /var/folders/36/jxbwdvbn56327wjxz2q2bzc40000gn/T/control-tower1865910723/state.json: no such file or directory]

I am not sure if we built the local control-tower executable properly as per the instructions here, or whether there is some other issue.

Is this something you have encountered at all?

[RichardBradley]

I've not seen either of those errors. sorry ("exec format error" or "Error loading state file after create-env")

There's nothing on the wider internet for the latter, but it does appear in control-tower here, during bosh init:
https://github.com/Softwire/control-tower/blob/9a921d21324696385eceac105350f46119b95c33/bosh/internal/boshcli/boshcli.go#L215

I can't immediately see why this might happen, and I've never understood the BOSH parts of this system, so I don't have any suggestions other than to try again, either on top of where you got to, or from scratch.

We have put the fork we're using up at https://github.com/softwire/control-tower/ , but it's almost identical to InventiveDesigners's fork at present.

Over the past 5 or so years that I've been using control-tower & concourse, I've had to delete and recreate our entire instance from scratch many times when inscrutable errors have happened, usually inside BOSH.

GL

[corinja]

Thanks a lot for the reply @RichardBradley - much appreciated.

For now, we have abandoned control-tower altogether and gone back to using a version of the Quick Start for Concourse. Of course this brings its own problems. I might try again with the fork of control-tower as above, but we might also try rolling our own deployment using AWS ECS. If we manage to get this working I'll put a post up on our tech blog and add a link here for reference.

[jordan-carlton]

Hi @corinja - I'm a colleague of @RichardBradley, and we did start to encounter the exec format error you outlined above last week, when the renew-https-certificate job tried to renew our soon-to-expire certificate.

If you open the binary that's causing the exec format error, you should see something like the following:

<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>AllAccessDisabled</Code>
  <Message>All access to this object has been disabled</Message>
  <RequestId>ABD</RequestId>
  <HostId>123</HostId>
</Error>

After much digging around, we were able to pin the issue to control-tower trying to download a version of bosh-cli that is no longer accessible. The S3 bucket was likely recently deleted/altered. The download path isn't actually specified in the control-tower repo, but in a separate repo called control-tower-ops that you need to download in order to compile a binary locally: https://github.com/EngineerBetter/control-tower-ops/blob/master/createenv-dependencies-and-cli-versions-aws.json#L10

https://s3.amazonaws.com/bosh-cli-artifacts/bosh-cli-7.2.0-linux-amd64 no longer exists, and trying to access it returns the same error as in the binary giving the exec format error error.

We had already compiled our own binary to circumvent the RDS CA cert issue, so we were able to fix exec format error by recompiling the binary locally, having first updated the bosh-cli source in https://github.com/EngineerBetter/control-tower-ops/blob/master/createenv-dependencies-and-cli-versions-aws.json to the following:

  "bosh-cli": {
    "windows": "https://github.com/cloudfoundry/bosh-cli/releases/download/v7.2.0/bosh-cli-7.2.0-windows-amd64.exe",
    "mac": "https://github.com/cloudfoundry/bosh-cli/releases/download/v7.2.0/bosh-cli-7.2.0-darwin-amd64",
    "linux": "https://github.com/cloudfoundry/bosh-cli/releases/download/v7.2.0/bosh-cli-7.2.0-linux-amd64",
    "version": "7.2.0"
  }

Installing that new binary fixed the errors we were seeing. There were also a few hard coded references to the original Engineer-Better repo that we pointed to our fork, as the renew-https-certificate job was trying to download the original version. You can view these changes in the commit history for our fork: https://github.com/softwire/control-tower/

Hope that helps!