View, Edit and Set pipelines from the UI

[ceco556]

Hi everyone,
I've been using Concourse for many years and there is one thing that annoys me a bit. Often when i am testing something with a pipeline i need to edit the config via the fly CLI for every small change. Because of that i would like to propose adding a pipeline config management in the Concourse UI.

The idea is to let people quickly edit and set a pipeline's config directly from the Concourse UI, without needing to use fly for every small change. This would be nice to do in the UI so i can easily follow what i have applied in the pipeline.

I love using fly CLI, but in practice it is a lot quicker to do such changes in the UI.

How do i imagine it in the UI:
Edit pipeline:

• Add an edit button(pencil) in the top bar.
• Opens a right-side panel with a YAML and JSON editor for the pipeline.
• Set pipeline button(check icon)
• Before actually applying, the set pipeline button will show errors and warnings and require confirmation.

Create pipeline

• Add a button somewhere in the dashboard (probably a + icon)
• Opens a right-side panel to create a pipeline. (picking the team, setting name and pipeline configuration)
• Set pipeline button(check icon), again showing errors and warnings before confirmation.

Technical notes:

• I don't think there is need to do any API changes - it can be implemented in the UI only.
• I think this feature should have a toggle flag to enable/disable easily. (maybe even per pipeline/team?)

I've already did a small POC over the last couple of days and can continue working on it if the community thinks this is a good feature to implement.

Here as you can see we have a message saying that the pipeline was set successfully. When there are warnings or errors they will appear at the top of the panel.

The create pipeline button is still not ready but you can get the whole idea behind this feature.

For now it works only with yaml but i can create a button to toggle YAML/JSON.

If needed i will be happy to write a RFC for this, but before that i would like to start a discussion and get a feedback first.

Questions popped up during the POC:

• Will this open opportunities for security issues? Maybe attacks?
• Would it be good to show credentials when the pipeline is loaded in the panel?
• Do we want to let the users to edit a pipeline in the UI or we can just have something like a view panel of the pipeline configuration?

[taylorsilva]

This does look very cool and I know it's something that's been asked for many times in various forms. An even smaller scoped ask that's related is people wanting to be able to see the pipeline config in the UI instead of trying to chase down the file or do fly get-pipeline. Folks have mentioned to me wanting to pull up the config of a single job in their pipeline while looking at the build logs.

I'm a little hesitant to accept a feature like this right now. I'm not 100% no, but I'm wary of taking in bigger features right now. I also know that historically the project has wanted to keep the UI as read-only focused as possible, which is why features like this have never been prioritized by the team or even appeared on the roadmap. I'd prefer to continue following this preference. It makes the UI much easier to maintain.

I do see how this can speed up development of pipelines though and allow users to iterate faster. I agree with you that I think this would be an improvement for the developer experience of writing a pipeline. I also worry about users modifying a pipeline that's set by a set_pipeline step. Introducing a feature like this when set_pipeline is a thing creates a footgun for users (i.e. User manually edits pipeline in UI, then wonders what reversed their change a few hours later). We would need a warning like what fly set-pipeline emits.

Other things I worry about with taking in a feature like this is with scope-creep over time. I know this is a POC, but some things I can imagine users already asking for:

• Syntax highlighting
• Display line numbers
• Highlight lines for errors/warnings
• Format the YAML/JSON

Suddenly we're building a little IDE inside our CI/CD system. We either build a bunch of these features ourselves or we take on more dependencies.

Then there are the open questions you posed as well. Suddenly there's a lot to consider!

I do really like the goal of this: speeding up the feedback loop for pipeline creation. I do agree that it could be faster.

---

Trying to think about other way we can speed up the feedback loop for creating pipelines:

• A Concourse LSP for folks to use in their editors of choice. No UI feedback though. Maintaining an LSP seems like a bit of work too.
  • I fell down a rabbit hole looking at LSP docs and then remembered a middle-ground is Schema store https://www.schemastore.org/. Could we create a Concourse pipeline schema that editors can easily use?
• What if fly set-pipeline had a --live-reload flag where it watches a file and constantly sets the pipeline whenever the file changes? You could have a browser open to the pipeline and the file open in your IDE of choice. Terminal can sit in the background or a pane in your IDE, watching the file and updating it
  • Idea taken from mkdocs serve --live-reload
  • I like this idea because ideally you'd be editing the file within the relevant repo, so you're at a point where you could iterate and then commit the final pipeline once you're happy with the changes

[ceco556]

I like the idea of having --live-reload flag to the fly set-pipeline. This way we can update easily pipelines during testing.

I will take a look at the code and will try to implement this.