Docker tag names

[carver]

We haven't revisited the docker tag names in a while. testnet is being used in our production network, for example.

@morph-dev mentioned wanting to improve the docker image-building process during release, including coming up with better names.

A proposal for new tag names:

• push a new tag for every version, to be permanently tied to the that version, as released. It would match the github tag, like v1.2.3
• a stable tag should be updated to the most recently released version (the non-prerelease ones, alpha/beta versions should not update the stable tag of course)
• rename testnet to prod or production (to be used by orchestration software)
• prod is updated to stable on every release
• Rollbacks become quite straightforward: manually switch the prod docker tag to a desired tagged version (probably the previously released version), and then re-run the orchestrated release

[KolbyML]

I agree with most of this. If prod is just an alias for stable, why not just not have prod.

So I agree with everything, but

rename testnet to prod or production (to be used by orchestration software)

^ I think we should remove testnet and just use stable from now on

As prod should be stable. I prefer the stable tag over prod, since stable is more descriptive of what it is, where prod feels like an intention, I don't think intentions should be tagified

[KolbyML]

Our production nodes should always be running stable, is what I am saying. Me and Milos talked about it a bit here for reference as well https://github.com/ethereum/cluster/pull/1176#issuecomment-2826648334

[carver]

If prod is just an alias for stable, why not just not have prod.

Because it's not just an alias, it's the pointer for what to deploy via orchestration, and the stable tag isn't always what you want to deploy. A rollback is the first example that comes to mind. A second example is that there might be some time that we want to test a version in the fleet before it is stable, or a hotpatch test after a rollback.

Our production nodes should always be running stable, is what I am saying.

I agree that they should generally be on stable, but don't agree that we should restrict ourselves from ever running anything besides stable on prod. I like having flexibility.

[KolbyML]

A rollback is the first example that comes to mind.

If we are doing a rollback shouldn't we use v1.2.3., to avoid potentially messing up re-tagging stuff?

I agree that they should generally be on stable, but don't agree that we should restrict ourselves from ever running anything besides stable on prod. I like having flexibility.

I agree

Because it's not just an alias, it's the pointer for what to deploy via orchestration, and the stable tag isn't always what you want to deploy. [..] A second example is that there might be some time that we want to test a version in the fleet before it is stable, or a hotpatch test after a rollback.

Hmm, that sounds reasonable. prod feels like a weird name for this then to me. Wasn't a goal Milos brought up for us to stop manually tagging things?

I am fine pushing custom tags to production as it can be very useful, for important situations. Maybe a tag called prerelease unstable or we could just latest, if we want to deploy a hotpatch. If we are deploying a custom tag does it make sense to push it to the prod tag? I am not sure. Maybe pushing a custom tag to the prod tag could be error prone? As it adds additional steps. Would a hotpatch be commited to master? If it is we could leave the default on ethereum/cluster to stable then temporarily update tag to ship to latest locally.

I am not sure just ideas, I wonder what others think.

[morph-dev]

Regarding previous discussion, if prod is the image that we deploy to nodes, then what is the usage of stable? This is something that confuses me.

I will provide my view / opinion below.

---

I believe that we shouldn't optimize too much about future use cases, because it's hard to say when we will have pre-release deployment, what would be its purpose, etc. I also have opinions on those (see below), but i don't think we should try to optimize for those use cases as tagging strategy is something that is easy to change.

I think that this workflow makes the most sense at the moment:

• on every commit on the master branch:
  • build and publish docker image with latest docker tag
• on every "vX.Y.Z" git tag on the master branch:
  • builds and publish docker image with <git-tag>-<git-commit> and prod docker tags (see naming opinion below)
    • I want to include git commit as well because git tags can be changed or deleted, and it's nice to have something immutable as a reference

If in the future we want some pre-release tags as well, I think one or more of these would make sense (which one I think we should use will would most likely depend on the use case, and as I said above, I don't see a point of discussing that now):

• in the steps above, when git tag is pushed, set staging instead of prod
  • this would be done manually by flamingo, once a week, e.g. on Monday
  • it should be followed by deployment on small subset of nodes (this deployment could be automatic)
  • if metrics and other tools show no issues by Wednesday / Thursday, flamingo marks the same docker image as prod and deploys it
• once a day, at fixed time, we automatically add daily-YYYYMMDD git tag at the head of the master branch, build and publish docker image with daily and daily-YYYYMMDD-<git-commit> tags
  • we would deploy this to small subset of nodes, automatically (these nodes use daily tag)
  • when doing staging/prod release, flamingo will pick one of the commits with daily-YYYYMMDD and add vX.Y.Z tag to it as well
  • In my opinion, this is very robust, and in our case probably too robust (at least for the foreseeable future).

---

On the topic of naming, here is the summary of what I used above:

tag	meaning	trigger	immutable	git tag
latest	The latest master branch commit	automatic, on every master commit	false
daily	The latest automatic build, done once a day	automatic, once a day	false
daily-YYYYMMDD-<git-commit>	The daily that was created on the specific day	automatic, once a day	true	daily-YYYYMMDD
staging	Image that is being deployed to pre-release nodes	manual, once a week	false
prod	Image that is deployed to most of the nodes	manual, once a week	false
vX.Y.Z-<git-commit>	The image that is being considered for staging/prod	manual, once a week	true	vX.Y.Z

The immutable tags should always include some immutable reference (e.g. git-commit). They are used only as a reference point, if we want to re-deploy some of the older images (e.g. in case of a rollback).

The tags that are created automatically (i.e. latest, daily), have the tag that describes the trigger for their creating.

The tags that are created manually, have a tag that describes their purpose. I used staging and prod, as these are the terms I'm used to, but I'm fine with some other naming conventions as I'm not strongly opinionated on these. Here are some possible options, in order of my preferences:

• staging and prod
• testing and stable
• dev and prod
• unstable and stable
• prerelease and release

[morph-dev]

Wasn't a goal Milos brought up for us to stop manually tagging things?

And to clarify a bit on this. I don't think that most common flow should require manual tagging (e.g. weekly release), but I'm fine if we do it in exceptional cases (e.g. doing a rollback, more risky rollout, etc).

Current flow is:

• preparation
  • bump cargo versions for both workspace and ethportal-api and run cargo update
  • create PR
  • wait for all checks to pass
  • merge PR
• release
  • create and push git tag on the commit that is associated with the merged PR
  • wait for the circle ci jobs that was triggered when PR was merged to finish publishing the docker image
  • make sure that dockerhub has the latest docker image
    • there can be an race condition here if another PR was merged at the same time
  • download latest images, apply testnet tag, and publish it
• deployment
  • run ansible script to deploy new image to nodes

All I'm saying is that we should improve the "release" steps below to be more automatic (as I don't think that other steps can be much more automated):
- create and push git tag on the commit that is associated with the merged PR
- this triggers a job that creates new docker image, on that specific commit and publishes it
- wait until this is complete and proceed with deployment

But if we need to manually tag something (e.g. rollback), I'm fine with us doing so.

[morph-dev]

Also, what are thoughts on differentiating between trin and bridge images?

I propose that we prefix them with their names, so with current proposal, we would have all these tags:
trin-latest, trin-prod, trin-vX.Y.Z-<commit>, bridge-latest, bridge-prod, bridge-vX.Y.Z-<commit>.

Currently we use bridge as suffix, but I find prefix to be better as related images are then grouped together.

[KolbyML]

Also, what are thoughts on differentiating between trin and bridge images?

I propose that we prefix them with their names, so with current proposal, we would have all these tags: trin-latest, trin-prod, trin-vX.Y.Z-<commit>, bridge-latest, bridge-prod, bridge-vX.Y.Z-<commit>.

Currently we use bridge as suffix, but I find prefix to be better as related images are then grouped together.

I think the bridge should just be under a different repo on docker hub

[KolbyML]

I will be creating images for trin-execution and e2hs-writer as well, so using suffix's would get messy quick

[carver]

Regarding previous discussion, if prod is the image that we deploy to nodes, then what is the usage of stable?

It's not for our deploy flow at all. As I understand it, some users might run trin via docker, so the docs that describe how to do that should probably say to pull the docker image with the stable tag.

If we are doing a rollback shouldn't we use v1.2.3., to avoid potentially messing up re-tagging stuff?

Yeah, we could rework how the ansible script works so that it could take a tag argument to deploy with. It doesn't do that right now, as far as I know. Updating the ansible script to require a tag argument would probably remove the need for the prod tag, in my mind.

I will be creating images for trin-execution and e2hs-writer as well, so using suffix's would get messy quick

Do you think all of these should get their own docker org? I don't have an opinion yet.

in the steps above, when git tag is pushed, set staging instead of prod

• this would be done manually by flamingo, once a week, e.g. on Monday
• it should be followed by deployment on small subset of nodes (this deployment could be automatic)
• if metrics and other tools show no issues by Wednesday / Thursday, flamingo marks the same docker image as prod and deploys it

Yeah, I like this one. Automatic deploy of staging would be awesome, though the last I talked to Paul about it, it seemed like the system was not really designed for it.

[KolbyML]

I will be creating images for trin-execution and e2hs-writer as well, so using suffix's would get messy quick

Do you think all of these should get their own docker org? I don't have an opinion yet.

Currently trin is under portalnetwork/trin

I was thinking it would be like

• portalnetwork/trin
• portalnetwork/bridge
• portalnetwork/trin-execution
• portalnetwork/e2hs-writer

I don't think a new org is needed, we can just make new repo's. But you think we should use a different org sure, but what would we call it then?

[morph-dev]

Regarding previous discussion, if prod is the image that we deploy to nodes, then what is the usage of stable?

It's not for our deploy flow at all. As I understand it, some users might run trin via docker, so the docs that describe how to do that should probably say to pull the docker image with the stable tag.

In which case would stable and prod be different? When would we like to recommend others to use something that we are not using ourself?

If we are doing a rollback shouldn't we use v1.2.3., to avoid potentially messing up re-tagging stuff?

Yeah, we could rework how the ansible script works so that it could take a tag argument to deploy with. It doesn't do that right now, as far as I know. Updating the ansible script to require a tag argument would probably remove the need for the prod tag, in my mind.

This doesn't sound like a good idea to me. I would rather have a prod tag and assume that's what we always deploy.