To support org above team

[evanchaoli]

Concourse supports only one level of teams, which is not good enough for a platform to host a lot of teams and pipelines.

In practice, people tend to use small team. Meaning that, a real project/team may have multiple teams on Concourse. I have seen at least two use cases that are hard to archive due to the restriction of one-level-team:

1. A project/team having multiple teams on Concourse may need to share some secrets. Due to this restriction, we have to use var_sources for sharing secrets, which adds complexities of pipeline code and OPs.

2. Some big projects (BU) need dedicate worker pool, but their pipelines are deployed in multiple Concourse teams. We don't want to create a worker pool for a small Concourse team.

I'm thinking to add one level org above team. It could be named org, or group, or else. I'm open for naming suggestion. Let's call it "org" for now.

An org may contain [0..n] teams, but org should not directly contain team member. A team may belong to 0 or 1 org.

Changes for org includes:

1. Add a fly set-org command to manage orgs. Only cluster admin can manager orgs.
2. Add <org>/key to secret search paths.
3. Add --org <org-name> option to concourse worker. To find a worker to run container, consider org as well.
4. Maybe more ... please suggest.

I want to use this discussion to call for some initial comments and feedback, then I may submit a RFC.

[vito]

Thanks for prompting this and including the motivation. 🙂

If you'll humor me for a moment, I want to tell a bit of a story:

In the beginning, Concourse had a single config. It didn't have pipelines or teams - just a single configuration which provided the resources to check and the jobs to run. (Fun fact: this is why the API endpoint is called SaveConfig, not SavePipeline, though it should probably just be renamed now.)

Then we added support for multiple pipelines through fly set-pipeline.

Then we added teams, so that pipelines could be scoped to different people and Concourse could be multi-tenant.

Then we added support for logging in to multiple teams.

Then we added RBAC, which is still developing into something more flexible.

Concourse has always been allergic to complexity, but I think this is one area where we've tended to just add more and more hierarchy and knobs without really questioning it as much as other things (like pipeline config) - which makes me worried every time that we continue the trend.

At various points I've wondered where we'd be if we stuck closer to the original design. With teams and RBAC and all these other things it feels like we're spending a lot of time and effort on "boring problems" that apply to any software project, while Concourse really needs more active growth on things like the v10 roadmap.

If the only reason for adding hierarchy is to add another level of configuration, I'd like to think a bit more about the motivations and see if there are other ways to address them. One of the key things I really disliked about other CI/CD tools, which motivated Concourse's creation, was that in order to figure out where something is configured, I would have to check all possible levels going up the stack until I found my answer. (And if I didn't find it, I'd wonder if there was a level of the stack that I missed, or if I'm going crazy.)

So, I'd like us to feel comfortable asking a more fundamental question: are Teams just not a good design? They've always been a little awkward, e.g. with fly. I wonder if we need to take a step back and think about this more holistically.

Or, maybe there are simpler resolutions: do we need workers to be able to register with multiple teams? Or perhaps register with some generic annotation which can then be associated to teams?

What was the pain point with var_sources? This sounds like one of the main reasons for implementing and using var_sources in the first place, so it's a bit surprising to hear that as a motivator, haha.

[evanchaoli]

@vito Thanks for the story sharing and the long replying. As a software grows, more and more effort would be spent on management, statistics etc peripheral functionality, also frequently get frustrated about that, but that's something we have to face to. (technically, I never learned how to add an emoji.)

This is a rough idea, what is reason why I haven't spent time on RFC yet, instead, I wanted to hear your opinion first.

I have been facing the challenge for long time. You may already know that we are trying to build a big Concourse cluster to server the entire company. The current biggest project running on the cluster, is split into ~100 teams. Problems we need to address includes: performance monitoring, data analysis (e.g. build success rate, etc), charge-back, and so on ...

var_sources was not a trigger of this idea. We are currently leveraging var_sources for secrets sharing among teams from a big project. I was just thinking that if org is supported, secrets sharing would be easier, but var_sources is still very useful for fetching secrets from other secret stores, like an external Vault.

For data analysis, we also have a workaround. Because teams from a project are named with the same prefix, so we can use team name prefix to group teams on the data analysis dashboards.

The main trigger was about worker. I considered a solution where it allows a worker to be marked with multiple teams. But I gave up the solution quickly. Because which would bring extra maintain efforts when adding/deleting teams and workers.

As you suggested, "perhaps register with some generic annotation which can then be associated to teams", that could also solve my problem. But the annotation sounds similar to org, isn't it?

I also wanted to keep the concept of org simple, that's reason why I proposed that an org can only be associated to teams, so that, org could be a pure concept for cluster admins, it may even not require UI changes, pipeline users would still only face to teams.

[vito]

You may already know that we are trying to build a big Concourse cluster to server the entire company.

Maybe I missed this detail or assumed the wrong thing. If it's not too late, I think it's worth reconsidering this approach; in my experience this can lead to a huge amount of pain that may be avoidable.

Concourse is designed so that pipelines can be portable and largely independent of what cluster they're on, so it really shouldn't be necessary to house every single pipeline all on one cluster. Doing so adds a lot of stability and scaling concerns and can make upgrades especially scary, since it's all-or-nothing. It also leads to growing complexity in Concourse as we have to figure out how to have a single Concourse cluster able to house all these different tenants.

If instead there was a model where you have multiple Concourse clusters, perhaps one per 'org'. you will immediately remove a whole lot of maintenance headache and can more gradually roll out changes and upgrades.

As you suggested, "perhaps register with some generic annotation which can then be associated to teams", that could also solve my problem. But the annotation sounds similar to org, isn't it?

To me, calling this 'org' implies a ton more than just having a way to share workers across teams. GitHub has organizations and teams, for example, and if Concourse also has organizations and teams I'll immediately assume they're meant for similar things.

[evanchaoli]

If instead there was a model where you have multiple Concourse clusters, perhaps one per 'org'. you will immediately remove a whole lot of maintenance headache and can more gradually roll out changes and upgrades.

We keep in mind that Concourse is stateless, pipeline can be portable. We try to avoid use "tag" with workers. With "tag", resource/task have to be marked with the tag in order to be dispatched to certain workers; in the other cluster, if no worker marked with the tag, then the pipeline will fail. But marking worker with team has no such a problem. In a cluster, some workers are marked with "team-a", then team-a's pipelines are auto dispatched to those workers; in another cluster, there is no worker is marked with "team-a", team-a's pipeline will still run. Pipeline users won't even be aware of on which workers their pipelines run.

Maintaining multiple clusters would still be pain. So far we are doing good with the single cluster mode. Unless we reach to a limitation of scale-out, we don't have plan to switch to multiple clusters mode.

To me, calling this 'org' implies a ton more than just having a way to share workers across teams. GitHub has organizations and teams, for example, and if Concourse also has organizations and teams I'll immediately assume they're meant for similar things.

I think I mentioned in the initial post that "org" is nothing more than a set of teams, it could be named whatever, I'm open for any naming suggestion. If only the word of "org" makes you feel uncomfortable, I'm totally fine to rename to any word else. As a non-English speaker, I excuse myself for my English, I always accept comments/suggestions wrt English.

My initial statement:
I'm thinking to add one level org above team. It could be named org, or group, or else. I'm open for naming suggestion. Let's call it "org" for now. An org may contain [0..n] teams, but org should not directly contain team member. A team may belong to 0 or 1 org.

[vito]

Maintaining multiple clusters may be more work, but it becomes a lot easier once you have the proper tooling and automation. In my experience this is much less painful than dealing with a company-wide CI cluster outage, chronic performance and reliability issues, or risky upgrades that may affect some teams more than others or cause problems cluster-wide because of usage patterns of one tenant.

I think it's fine to start with one cluster for now - not trying to rock the boat. This is just my advice from having to provide support to folks who have gone down this path before. 🙂

[vito]

Going back to the original motivation - what do you think of adding a 'worker pool' concept? So rather than worker registering with a team, a worker could register with a pool, and then teams could be associated to a pool by a cluster admin. Would that suit your needs?

I think this is probably mechanically similar to 'orgs' except it's not something that's "above" a team - it's just a grouping of workers that can be associated to many teams.

Something like this is probably doable:

$ fly set-worker-pool -p my-pool --authorized-key key-1
$ concourse worker --pool my-pool
$ fly grant-worker-pool -p my-pool --team my-team

An alternative to set-worker-pool would be to have the set of pools and keys could be configured cluster-wide, which would be closer to how per-team authorized keys are configured today. Just throwing the idea out there since I know some folks find it painful to reconfigure the cluster when adding new keys and such.

[evanchaoli]

@vito I'm happy to hear the concept "worker pool" from you. Actually we have been using this concept for long time in our worker manager. Currently we use team+tags to identify a worker pool, but due to the small teams issue, we are hard to move forward. If Concourse can officially supports worker pool, which would perfectly map to our current model.

Well, I'm not sure if --authorized-key is a must. In our cluster, I think that is not needed. Because all workers are maintained by us, we just need a pool concept to isolate workloads and charge-back teams. Currently all workers use same authorized key, I think that is ok even for workers of different pools. What's your thoughts behind?

would be closer to how per-team authorized keys are configured today

I don't get what "per-team authorized keys", we don't use that in our clusters. Can you clarify that? Maybe we miss something in our clusters?

[vito]

Cool!

I don't get what "per-team authorized keys", we don't use that in our clusters. Can you clarify that? Maybe we miss something in our clusters?

The TSA can be configured with authorized keys that map to individual teams. This is helpful in allowing the team themselves to maintain their own worker without giving them the power to register their worker globally or for arbitrary teams.

This is done via these (undocumented!) flags on concourse web:

--tsa-team-authorized-keys=NAME:PATH                                                                  Path to file containing keys to authorize, in SSH authorized_keys format (one public key per line). [$CONCOURSE_TSA_TEAM_AUTHORIZED_KEYS]
--tsa-team-authorized-keys-file=                                                                      Path to file containing a YAML array of teams and their authorized SSH keys, e.g. [{team:foo,ssh_keys:[key1,key2]}]. [$CONCOURSE_TSA_TEAM_AUTHORIZED_KEYS_FILE]

I imagine configuring keys for pools would probably look pretty similar, except we'll also need some sort of pool object in the database so that they can be associated to teams and workers. So it's tempting to me to just allow specifying those keys on the same command that creates the pool. It'd be a bit more work though, so definitely not a requirement.

[evanchaoli]

Okey, in our clusters, we don't allow users to maintain their own workers, thus per team authorized keys are not needed by us. I'll leave an open question for that in the RFC maybe. So, based on the discussion, I'll compose a RFC for worker pool.

[vito]

@evanchaoli Ah, yeah makes sense. For context, in our case this was needed so that teams could register workers that were able to access certain networks, or any other random requirement - and having our team operate all those random workers didn't really make sense to either party.

Looking forward to the RFC!

[tlwr]

This is awesome

---

We've also got abstractions:

• worker-pool (maps to a single team's AWS Autoscaling group and IAM role)
• network-pool (in which a worker-pool goes) (maps to egress IP addresses)

See https://github.com/alphagov/tech-ops/tree/master/reliability-engineering/terraform/modules

With the var_sources changes (comment) and the concept of a concourse worker pool, we would be able to break the relationship between IAM/ASG and Concourse teams, whilst still being able to achieve isolation where necessary.

I also look forward to the RFC.

---

We use --tsa-team-authorized-keys to stop one team from impersonating another team.