Next steps of Notation weekly build

[yizha1]

Here is the discussion thread on the next steps of notation weekly build, which is a complement for item 4 of discussion #257. We can also discuss any issues to be created in order to achieve weekly build.

Rational
If you check the notation releases https://github.com/notaryproject/notation/releases, you will find the release frequency is too low. The last milestone release is on Jun 1, which is almost two months ago. There are more around 15 commits and more than 40 files changes. If you build notation binary, some commands are broken. The quality is out of control, and it will be a big risk to make a milestone release
In order to improve the quality, release predictability and short customer feedback loop, it is recommended to introduce weekly dev build release for notation.
It takes time to fix and merge the issues found during a dev build, so it is recommended to build at least one week.
It's better to mark features status: alpha, beta or stable, so that user is aware of the feature readiness.
It is also recommended to introduce automatic E2E testing cases to ensure the quality and avoid duplicated manual work.

Checkpoints

• It should be an auto-release as an ultimate goal
• Build results will be sent out before a new work week start, so that team can receive results in time and start to fix the issues at the beginning of a new work week
• Align the release versioning for weekly build

[dtzar]

Everything here makes sense to me except for the "User can trigger on demand dev build easily when the decision is made in the community".

If we are doing weekly automated builds, why would we have a need for a user to trigger an on-demand build for us? IMO if people want more than this, then they likely are a developer and should have proper instructions to easily do a local build.

[FeynmanZhou]

SemVer-dev-yyyymmdd is much clear to me.

[dtzar]

If the SemVer is whatever the last actual release is, then I like the suggestion.
For example, you would have:
0.9.0-alpha.1-dev-20220727
0.9.0-alpha.1-dev-20220803

We would just need to double check this doesn't cause a problem with SemVer thinking the -dev is a newer one. If so, then we'd need to switch to put dev in the front perhaps.

[SteveLasker]

That’s how I read this as well. How does the user know which version the daily build is based on?

[yizha1]

@dtzar Just did a test. It will be either pre-release or release on the release page. Please have a look at below image. If we use SemVer plus dev-20220729 or anything, it will be a newer pre-prelease. If we use dev-20220729, it will still be pre-release, but it will be showed before SemVer release.

[dtzar]

It seems that if we do v0.9.0-alpha.1-dev-DATE then it might be that dependabot would discover a newer release. It is likely not a problem with notation since no one likely takes a dependency on us, but for the notation-go and notation-core-go this could be a problem.

I'm fine with even the pre-releases showing on the same release page.

[JeyJeyGao]

Junjie Gao will work on this proposal.

[yizha1]

We need to find a way to release dev builds, here are two options:

Option 1: The weekly dev build will be released the same way as normal release.

Pros:

• Easy to do, follow the current WoW of a release
• The releases can be found in one-stop place, which is https://github.com/notaryproject/notation/releases
• User can use curl or wget command to download the artifacts easily.

Cons:

• The release page will be mixed with different kinds of releases, that is, dev build, pre-releases and GA releases.
• The dev build will be released every week, so there will be many dev build releases over time, the pre-release and GA release will not be easily found.

Option 2: Use object storage to store dev build artifacts, please refer to istio dev builds as example https://github.com/istio/istio/wiki/Dev-Builds, A wiki is created for dev builds, and these build artifacts are stored in backend storage googlestorage, not in the GitHub

Pros:

• Dev build release will not "mess" the release page

Cons:

• Need a storage service for storing dev build artifacts and lifecycle management. It takes time introducing a storage service and more effort need to be spent on maintenance.

Any comments are welcome.

[FeynmanZhou]

I support Option 2. Meanwhile, we can configure a domain name for the dev build to download the latest weekly build without memory of the tag name, such as https://notaryproject.dev/notation-build/dev/latest.

[dtzar]

I vote for option 1 since we're only doing weekly builds and I don't see a need to go more frequently than this in the future. People can filter out releases with the search option and/or visit the tags page for the actual releases. We also have the roadmap repo with release information there.
If we're doing daily builds (like Istio), then you're right I think option 2 would be best. The overhead of setting up and maintaining the infrastructure here is not worth the cost of only the "mess" of the release page.

There is also one more advantage IMO - is that people can easily see the progress and download the weekly builds. If you go option 2, then you have to make sure they discover this is available (versus just going to the standard release page).

[yizha1]

I vote for option 1 considering the time and cost are potential big to set up and maintain storage service.

[SteveLasker]

I’m very supportive of frequent builds, easing validation testing. It’s particularly important as we have coordination across 3 notation repos (notation-go, notation-core and notation), in addition to dependencies on oras-go
While a simple make command helps devs with single repo builds, it’s hard to coordinate across.
I’m not sure if automated build would solve this as opposed to human initiated
when it’s known cross dependencies are stable.
Ultimately, getting to functional testing on check-in will provide the stability.

@Wwwsylvia had worked on some automation might provide insight to those conversations with @sajayantony

[dtzar]

I believe we should automate the notation build to automatically include the latest commit that is present on the main branch of notation-go and notation-core-go. oras-go automatic upgrade I could go either way, but would vote for this one to be on the latest release (not main branch commit). We should not IMO involve manual decisions of what version to include for the weekly builds.

This plus the automated testing will help us to find where there are instability/issues. The stability in the features should also be addressed with the other option(s) discussed in #257

[SteveLasker]

I yield too what the team working daily wishes to do.
I, like others, would just like to be able to get a binary on random times with condense it will work

[yizha1]

I believe we should automate the notation build to automatically include the latest commit that is present on the main branch of notation-go and notation-core-go. oras-go automatic upgrade I could go either way, but would vote for this one to be on the latest release (not main branch commit). We should not IMO involve manual decisions of what version to include for the weekly builds.

This plus the automated testing will help us to find where there are instability/issues. The stability in the features should also be addressed with the other option(s) discussed in #257

@dtzar I understand your point. Here is a scenario: There are interface changes in library repo in order to implement a new feature, then the weekly build will always fail if notation cli pull the latest commits since the interfaces are changed. This problem will be solved after the feature is done in the library, and notation cli completes the adaption. It is not a bug, instead it is due to the feature development in main branch. It will block the weekly build for some time if notation fetches the latest commit from main branch. After checking with the dev, the dependencies of notation cli are managed by go.mod file, which will be kept updated once the update in the library is done. The dependencies in the go.mod file will be in most cases the latest available. So, I would propose the weekly build triggered on notation cli repo, and the dependencies version will be decided by go.mod file. For weekly builds, no manual decision on changing the dependencies in the go.mod file. Does this make sense?

[dtzar]

I understand go.mod contains the dependencies and also that the libraries might not be integrated yet. For the initial weekly creation, I'm fine if we don't auto or manually bump go.mod to main commit for notation-go or notation-core-go. I believe this is a way we can push to test if things are still working - so perhaps we can consider this as a separate test associated with the release to discover problems we might have not been aware of.

[yizha1]

#275 was created for implementation.