Support Pkl configuration langauge

[AP-Hunt]

Hi @taylorsilva

As I'm sure you've experienced, and heard from users often enough, writing and maintaining YAML for big pipelines isn't a great experience, especially when they can grow to be thousands of lines long. Would you be interested in supporting a second way of declaring pipelines, namely Pkl?

Pkl is a configuration language built, supported, and used by Apple. To quote its own website, Pkl aims to be "configuration that is Programmable, Scalable, and Safe". It's also uses the same license as Concourse (Apache 2.0).

The team behind GOV.UK Pay, a payments service inside the UK government, is already using Pkl to define Concourse pipelines, and the teams behind https://gov.uk/ will begin doing the same in the near future. There is also interest in using Pkl for this purpose from other teams in the organization.

How does it work for Pkl users today?

The teams using Pkl for Concourse pipelines today have built out a Pkl module that matches the Concourse pipeline YAML structure, and published it under the MIT license. They extend and use that module to generate full pipelines, and have Pkl serialize them as YAML which is passed to Concourse.

They've done all of this without the support of the Concourse project, and can of course continue to do so quite happily.

What's the opportunity?

There's general dissatisfaction with YAML across the industry, especially when it grows to a large number of lines (citation needed; it's a bit hard to evidence this, but I'm trusting you've experienced it too).

I think by adding native support for Pkl, there are a number of potential benefits for Concourse:

1. Differentiation from, and potential advantage over, other CI/CD systems, most of which use YAML alone
2. Providing a configuration mechanism which is flexible and programmable, without needing a full general purpose programming language and all the baggage that comes with that (such as dependency churn)
3. Supporting two mechanisms gives users a path to easing the pain of large YAML files when they grow; they can start with YAML and move to Pkl

What levels of support could there be?

Concourse could support Pkl at a few different levels. It doesn't need to be all or nothing.

• Package support
  Concourse could produce and maintain its own Pkl package, with useful helpers, good documentation etc. Users of the package can generate YAML from Pkl themselves and pass it to fly

• fly support
  Pkl has Go bindings which fly can use to turn Pkl configuration into YAML itself. I've done just this in a proof of concept branch. It's a reasonably small change to fly in all. If you were to support Pkl at this level, publishing an official package would be a sensible move too.

• Native support
  Pkl's Go bindings also allow it to read Pkl code and create Go structs. Concourse could fully support Pkl an equal sibling of YAML in ATC, with diffs for Pkl in the same way it has for YAML. If you were to support Pkl at this level, publishing an official package would be a sensible move too.

Support offered

It would obviously be quite rude to ask you if you're interested in supporting this and then ask you to build it! I'd be very happy to work on this as an individual (I'm not raising this on behalf of my org, just citing our uses)

[taylorsilva]

Oh yes, I am familiar with this pain. I have spent a little bit of time exploring this problem space and played around with Cue for a while to see if there was something there. I haven't heard of Pkl before your post here, but it seems great as well and possibly a better choice over Cue at least in the features department. The language bindings are great to have!

The package support level is what I had in mind when I was playing with Cue. Though given the language bindings with Pkl, I think you're right that it wouldn't be too hard to add support in the fly CLI and even for the set_pipeline step. It looks straight forward from a quick look at your fork.

The native level of support is probably too much effort. It's unclear to me what the project or end users would gain if we did the work for native support that we don't also get with the prior options. Maybe I'm not thinking creatively enough here?

So yes, this idea really excites me because OMG 1000+ line YAML files suck so much. I'd like more discussion on the configuration language we should support. Pkl seems great, but I really don't know any of the other languages in this space. Dhall seems similar and has language bindings as well: https://dhall-lang.org/

I'm curious to know how your org settled on Pkl? What other options were considered?

---

It would obviously be quite rude to ask you if you're interested in supporting this and then ask you to build it! I'd be very happy to work on this as an individual (I'm not raising this on behalf of my org, just citing our uses)

Appreciate you clearly stating this. I really want something like what you're proposing, but TIME and PRIORITIES are limited 😅 I'm happy to help shepherd us to a solution as project maintainer.

---

Other thought I had, it would be nice if Pkl or any of these config languages had a YAML/JSON to [config-lang] tool. Maybe they do, I haven't dug into this. Re-writing a 1000+ line YAML file into one of these config languages is not a fun activity for a human to do ☹️

[taylorsilva]

Hmmm neat: https://cuelang.org/docs/concept/how-cue-works-with-go/

[taylorsilva]

We'll want to eventually write up an RFC: https://github.com/concourse/rfcs/
We can keep discussing here for now though.

[AP-Hunt]

The native level of support is probably too much effort. It's unclear to me what the project or end users would gain if we did the work for native support that we don't also get with the prior options. Maybe I'm not thinking creatively enough here?

In my head, I saw Concourse doing the Pkl transformation on the server side, and storing the Pkl that was submitted. That way it could provide direct Pkl-to-Pkl diffs, and it could remove the mental translation from Pkl to YAML and back when comparing and fixing diffs.

Also, the Pkl bindings require that the Pkl binary be available in $PATH, and doing it server side would let Concourse bundle the correct Pkl version alongside, and remove the need for users to install it where they do their pipeline updates.

I agree it comes with additional complexity, though. Least of all from one Pkl file being able to import another: it'd need to accept an archive of the files involved, in their relative paths. Pkl can also pull packages from the internet, so there would need to be consideration for installations which don't allow internet egress too.

I'm curious to know how your org settled on Pkl? What other options were considered?

We've just been through a discovery process for a new language to do all sorts of previously YAML-based config in - not just Concourse. We evaluated each of these with some slightly woolly criteria like standard library support, IDE support, and third-party package libraries.

• KCL
• Pkl
• Jsonnet
• Dhall
• Cue
• Ruby
• Go
• JS/TS
• Python

Of those, we decided to try out KCL, Pkl, and Go. KCL was like a less mature, less featured rich version of Pkl, and the amount of effort required to do it well in Go made us worry about how different the implementations could end up being - plus the additional language version and dependency maintenance burden.

We chose Pkl because we tried it and liked it, and we knew another team was already using it successfully.

To quote our internal ADR on Cue:

Cue
We chose not to continue with Cue because it lacks any official IDE support, and its package library ecosystem is slim to non-existent.

I can quote it for the other ones too if you need.

it would be nice if Pkl or any of these config languages had a YAML/JSON to [config-lang] tool

I agree it would be nice. I'd be surprised if any of them did (and I know Pkl doesn't), because the question quickly becomes "how? what should the code look like?"

If Concourse had an official Pkl library with strong opinions, it feels feasible to write something that converts the YAML pipelines to Pkl. At least as a starting point for teams.

[AP-Hunt]

@taylorsilva as project maintainer, what would you like the next steps to be?

[taylorsilva]

I keep oscillating between "this could be cool" and "it feels bolted on" as I think about this more.

In my head, I saw Concourse doing the Pkl transformation on the server side, and storing the Pkl that was submitted. That way it could provide direct Pkl-to-Pkl diffs, and it could remove the mental translation from Pkl to YAML and back when comparing and fixing diffs.

That would be cool, but we store everything in jsonb type columns in postgres. Realizing this now, I don't think we could bring Pkl to native-level support unless we make massive changes to how we store our configuration.

I agree it comes with additional complexity, though. Least of all from one Pkl file being able to import another: it'd need to accept an archive of the files involved, in their relative paths. Pkl can also pull packages from the internet, so there would need to be consideration for installations which don't allow internet egress too.

This would be a big shift within the codebase as well, as everything assumes that the configuration is basically static, except for vars ((...)). Concourse has been designed with YAML in mind, and trying to replace YAML with something else is a massive shift, especially since we would need to always support YAML or somehow provide a migration path of YAML -> Pkl.

Native support is definitely not possible at this time with the resources the project has.

---

At it's core, I really like the idea of trying to give users a better way to manage their pipelines when they grow to thousands of lines long. The more I think about this though, the maintainer side of me is pulling me towards a general "no", at least in regards to tightly integrating Concourse with Pkl, even at the fly level. Since it's still early days for Pkl too, I'd worry about Concourse releases drifting behind Pkl releases and users asking us to release a new version of Concourse just to bump Pkl to get at new features/fixes.

I think the package-level support is the best thing the project could offer at this time. It would be nice to give users a recommended "next step" for when their pipelines grow really large and complex.

So you guys have https://github.com/alphagov/pkl-concourse-pipeline which I assume users reference as a module(?) in their own pkl files. Would the idea be to migrate that repo to Concourse? github.com/concourse/pkl-concourse or similar? Would folks from your side continue to help maintain and contribute to it?

[AP-Hunt]

I completely get you decision, it's probably the one I'd make in your position too.

wrt to that package, I'll have to have a conversation with the folks who maintain it currently, as well as any teams who might be interested in using it. We're pretty busy right now, I'm on annual leave for the first half of December, and the holidays are right around the corner, so there's a chance it won't get to happen until the new year.

I'll update you after it happens though, for sure.

[AP-Hunt]

The team this side is keen to contribute what we have and take ownership of its maintenance. However, we need to hold off on doing it until we start active work on our Concourse plans; we've had to put them down for a bit to attend to more urgent matters.