Canary deployments persistance

[benpettman]

Hello, I am using the latest version of linkerd and have got 2 front end demo applications, one is classed as normal and the other is red, I have created a TrafficSplit and weighted them 90/10 respectively.

I then visit the front end url and get some weird behaviour, the app is a NextJS react website, running 2 different image builds, and each build uniquely names js and css, what is happening is I refresh the page and get 404 errors because it cant load either the js or css as its making that request to a pod that doesnt contain those files.

I have setup 3 different Ingress controllers to test this, Nginx, Traefik and Ambassador, applying their cookie persistence, but the same issue occurs.

I am looking for the correct way to get a canary TrafficSplit with persistence working with any Ingress at the moment, as far as I can tell from my testing the only way Id be able to achieve this is with an A/B style canary but then thats not a canary.

Hopefully someone can shed some light as to what I am doing wrong.

[ihcsim]

Assuming that your ingress controller is injected with the Linkerd proxy, the first thing I'd would check is whether the ingress controller is adding the l5d-dst-override header to the requests. You can use the linkerd tap -ojson commandto do that. If you haven't already, see https://linkerd.io/2/tasks/using-ingress/ on how to annotate the ingress resource.

If the header is present, the next thing to check is the Linkerd proxy logs, to see if there are any hints there.

[benpettman]

Hi @ihcsim Yes ingress is injected with the sidecar. I will check the header, but I would of thought if it wasnt adding this header then it wouldn't route to a service at all?

[ihcsim]

Yes, you are right about the header. I missed the session stickiness part in the original problem description. Linkerd doesn't provide that today. There is an open issue at #3504. The workaround for now is to uninject the Linkerd proxy from your ingress controller, and let the controller handle this.

[benpettman]

@ihcsim I get you. Totally makes sense. I will give it a go and report back, thank you

[benpettman]

I reverted back to using the nginx controller and that seems to work.

Its a shame session stickiness doesnt work via Linkerd as thats kinda crucial for canary, A/B testing etc.

I guess until that is fixed then I cant use any of the SMI TrafficSplit controls? Or can I?

[ihcsim]

You can use SMI trafficsplit for other stuff that don't require sessions. If this is important to your setup, feel free to +1 #3504. That will help to bump its priority.

[benpettman]

When you say "You can use SMI trafficsplit for other stuff that don't require sessions", what examples are these?

[ihcsim]

The canary deployment with Linkerd + Flagger is a good one. See the doc here.