linkerd fails to find Lease component for service-mirror

[lboutin-mwm]

Hello,

I use the linkerd multicluster plugin to ease communications between some of my clusters and deploy it using its helm chart.

I recently upgraded the stack to stable-2.14.1 (from stable-2.14.0) and one of my clusters is unable to find the lease created when linking two clusters:

This is the result of linkerd multicluster check on my central cluster:

linkerd-multicluster
--------------------
√ Link CRD exists
√ Link resources are valid
        * ml-demos-us
        * ml-staging-us
√ remote cluster access credentials are valid
        * ml-demos-us
        * ml-staging-us
√ clusters share trust anchors
        * ml-demos-us
        * ml-staging-us
√ service mirror controller has required permissions
        * ml-demos-us
        * ml-staging-us
√ service mirror controllers are running
        * ml-demos-us
        * ml-staging-us
√ probe services able to communicate with all gateway mirrors
        * ml-demos-us
        * ml-staging-us
√ all mirror services have endpoints
√ all mirror services are part of a Link
√ multicluster extension proxies are healthy
√ multicluster extension proxies are up-to-date
√ multicluster extension proxies and cli versions match

Status check results are √

and this is the result of the same command on my faulty cluster:

linkerd-multicluster
--------------------
√ Link CRD exists
√ Link resources are valid
        * tools-eu
√ remote cluster access credentials are valid
        * tools-eu
√ clusters share trust anchors
        * tools-eu
√ service mirror controller has required permissions
        * tools-eu
√ service mirror controllers are running
        * tools-eu
× probe services able to communicate with all gateway mirrors
        failed to get the service-mirror component Lease for target cluster tools-eu: leases.coordination.k8s.io "service-mirror-write-tools-eu" not found
    see https://linkerd.io/2.14/checks/#l5d-multicluster-gateways-endpoints for hints
√ all mirror services have endpoints
√ all mirror services are part of a Link
√ multicluster extension proxies are healthy
√ multicluster extension proxies are up-to-date
√ multicluster extension proxies and cli versions match

Status check results are ×

For some reason it seems to work one way and not the other.
The lease does in fact exist:

kctl get leases -n linkerd-multicluster
NAME                            HOLDER                                             AGE
service-mirror-write-tools-eu   linkerd-service-mirror-tools-eu-7854857ffc-j6c8k   44m

and I can get its description just fine:

Name:         service-mirror-write-tools-eu
Namespace:    linkerd-multicluster
Labels:       <none>
Annotations:  <none>
API Version:  coordination.k8s.io/v1
Kind:         Lease
Metadata:
  Creation Timestamp:  2023-10-18T15:28:29Z
  Managed Fields:
    API Version:  coordination.k8s.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        f:acquireTime:
        f:holderIdentity:
        f:leaseDurationSeconds:
        f:leaseTransitions:
        f:renewTime:
    Manager:         controller
    Operation:       Update
    Time:            2023-10-18T16:14:20Z
  Resource Version:  178043801
  UID:               99550f50-0d88-4076-a7fb-1abcc07c7386
Spec:
  Acquire Time:            2023-10-18T15:28:29.396070Z
  Holder Identity:         linkerd-service-mirror-tools-eu-7854857ffc-j6c8k
  Lease Duration Seconds:  30
  Lease Transitions:       0
  Renew Time:              2023-10-18T16:14:20.345628Z
Events:                    <none>

All of the aforementioned clusters are running in GKE with version 1.25.12-gke.500 of k8s

These are the commands I used to link both clusters:
linkerd --context=tools-eu multicluster link --cluster-name tools-eu | kubectl --context=ml-demos-us apply -f -
linkerd --context=ml-demos-us multicluster link --cluster-name ml-demos-us | kubectl --context=tools-eu apply -f -

Only one of three clusters connected to the central one is affected and there was no issue with the earlier 2.14.0 version of linkerd.
It also does seem like the mirroring and connection between clusters is working, this might just only be an error with the check command

[hawkw]

Hi @lboutin-mwm, this looks like a potential bug in the linkerd check command. Since traffic is being mirrored successfully, it is probably just an issue with check and not indicative of a larger problem, but we're going to investigate and see what's going on here. I've opened a new issue #11509 to track this as a bug.

[mateiidavid]

@lboutin-mwm hey! I had a look at the issue. It's a bit odd that you're experiencing this. The only place where we try to list Lease resources during a check is fallible only through the Kubernetes client. linkerd mc check --verbose might help (in my case, it didn't really display a lot of information).

There are several reasons why a Kubernetes Get call may fail:

• The resource may not exist (which does not seem to be the case here)
• Transient network errors (which may not be the case if you reliably hit this)
• Missing privileges
  ...and possibly some other reasons I'm missing.

Can you have a look and see if any of these apply to you? I would try to troubleshoot why the Kubernetes API Server would fail in the check command. As you have noted, everything works fine in-cluster (where roles and resources exist).

[lboutin-mwm]

Thanks for the hints

I tried to use linkerd mc check --verbosebut the command didn't yield any additional info:

linkerd mc check --verbose
linkerd-multicluster
--------------------
√ Link CRD exists
√ Link resources are valid
        * tools-eu
√ remote cluster access credentials are valid
        * tools-eu
√ clusters share trust anchors
        * tools-eu
√ service mirror controller has required permissions
        * tools-eu
√ service mirror controllers are running
        * tools-eu
× probe services able to communicate with all gateway mirrors
        failed to get the service-mirror component Lease for target cluster tools-eu: leases.coordination.k8s.io "service-mirror-write-tools-eu" not found
    see https://linkerd.io/2.14/checks/#l5d-multicluster-gateways-endpoints for hints
√ all mirror services have endpoints
√ all mirror services are part of a Link
√ multicluster extension proxies are healthy
√ multicluster extension proxies are up-to-date
√ multicluster extension proxies and cli versions match

Status check results are ×

I didn't know which service account's permissions to check so I checked all that were associated with linkerd using kubectl auth can-i --listand this was the result:

NS: linkerd-multicluster / SA: default
Resources                                       Non-Resource URLs                     Resource Names   Verbs
selfsubjectaccessreviews.authorization.k8s.io   []                                    []               [create]
selfsubjectrulesreviews.authorization.k8s.io    []                                    []               [create]
                                                [/.well-known/openid-configuration]   []               [get]
                                                [/api/*]                              []               [get]
                                                [/api]                                []               [get]
                                                [/apis/*]                             []               [get]
                                                [/apis]                               []               [get]
                                                [/healthz]                            []               [get]
                                                [/healthz]                            []               [get]
                                                [/livez]                              []               [get]
                                                [/livez]                              []               [get]
                                                [/openapi/*]                          []               [get]
                                                [/openapi]                            []               [get]
                                                [/openid/v1/jwks]                     []               [get]
                                                [/readyz]                             []               [get]
                                                [/readyz]                             []               [get]
                                                [/version/]                           []               [get]
                                                [/version/]                           []               [get]
                                                [/version]                            []               [get]
                                                [/version]                            []               [get]
----------------------------------------

NS: linkerd-multicluster / SA: linkerd-gateway
Resources                                       Non-Resource URLs                     Resource Names   Verbs
selfsubjectaccessreviews.authorization.k8s.io   []                                    []               [create]
selfsubjectrulesreviews.authorization.k8s.io    []                                    []               [create]
                                                [/.well-known/openid-configuration]   []               [get]
                                                [/api/*]                              []               [get]
                                                [/api]                                []               [get]
                                                [/apis/*]                             []               [get]
                                                [/apis]                               []               [get]
                                                [/healthz]                            []               [get]
                                                [/healthz]                            []               [get]
                                                [/livez]                              []               [get]
                                                [/livez]                              []               [get]
                                                [/openapi/*]                          []               [get]
                                                [/openapi]                            []               [get]
                                                [/openid/v1/jwks]                     []               [get]
                                                [/readyz]                             []               [get]
                                                [/readyz]                             []               [get]
                                                [/version/]                           []               [get]
                                                [/version/]                           []               [get]
                                                [/version]                            []               [get]
                                                [/version]                            []               [get]
----------------------------------------

NS: linkerd-multicluster / SA: linkerd-service-mirror-remote-access-default
Resources                                       Non-Resource URLs                     Resource Names     Verbs
events                                          []                                    []                 [create patch]
selfsubjectaccessreviews.authorization.k8s.io   []                                    []                 [create]
selfsubjectrulesreviews.authorization.k8s.io    []                                    []                 [create]
                                                [/.well-known/openid-configuration]   []                 [get]
                                                [/api/*]                              []                 [get]
                                                [/api]                                []                 [get]
                                                [/apis/*]                             []                 [get]
                                                [/apis]                               []                 [get]
                                                [/healthz]                            []                 [get]
                                                [/healthz]                            []                 [get]
                                                [/livez]                              []                 [get]
                                                [/livez]                              []                 [get]
                                                [/openapi/*]                          []                 [get]
                                                [/openapi]                            []                 [get]
                                                [/openid/v1/jwks]                     []                 [get]
                                                [/readyz]                             []                 [get]
                                                [/readyz]                             []                 [get]
                                                [/version/]                           []                 [get]
                                                [/version/]                           []                 [get]
                                                [/version]                            []                 [get]
                                                [/version]                            []                 [get]
configmaps                                      []                                    [linkerd-config]   [get]
endpoints                                       []                                    []                 [list get watch]
pods                                            []                                    []                 [list get watch]
services                                        []                                    []                 [list get watch]
replicasets.apps                                []                                    []                 [list get watch]
jobs.batch                                      []                                    []                 [list get watch]
endpointslices.discovery.k8s.io                 []                                    []                 [list get watch]
servers.policy.linkerd.io                       []                                    []                 [list get watch]
----------------------------------------

NS: linkerd-multicluster / SA: linkerd-service-mirror-tools-eu
Resources                                       Non-Resource URLs                     Resource Names                   Verbs
leases.coordination.k8s.io                      []                                    []                               [create get update patch]
selfsubjectaccessreviews.authorization.k8s.io   []                                    []                               [create]
selfsubjectrulesreviews.authorization.k8s.io    []                                    []                               [create]
                                                [/.well-known/openid-configuration]   []                               [get]
                                                [/api/*]                              []                               [get]
                                                [/api]                                []                               [get]
                                                [/apis/*]                             []                               [get]
                                                [/apis]                               []                               [get]
                                                [/healthz]                            []                               [get]
                                                [/healthz]                            []                               [get]
                                                [/livez]                              []                               [get]
                                                [/livez]                              []                               [get]
                                                [/openapi/*]                          []                               [get]
                                                [/openapi]                            []                               [get]
                                                [/openid/v1/jwks]                     []                               [get]
                                                [/readyz]                             []                               [get]
                                                [/readyz]                             []                               [get]
                                                [/version/]                           []                               [get]
                                                [/version/]                           []                               [get]
                                                [/version]                            []                               [get]
                                                [/version]                            []                               [get]
endpoints                                       []                                    []                               [list get watch create delete update]
services                                        []                                    []                               [list get watch create delete update]
namespaces                                      []                                    []                               [list get watch]
secrets                                         []                                    [cluster-credentials-tools-eu]   [list get watch]
links.multicluster.linkerd.io                   []                                    []                               [list get watch]
----------------------------------------

NS: linkerd / SA: default
Resources                                       Non-Resource URLs                     Resource Names   Verbs
selfsubjectaccessreviews.authorization.k8s.io   []                                    []               [create]
selfsubjectrulesreviews.authorization.k8s.io    []                                    []               [create]
                                                [/.well-known/openid-configuration]   []               [get]
                                                [/api/*]                              []               [get]
                                                [/api]                                []               [get]
                                                [/apis/*]                             []               [get]
                                                [/apis]                               []               [get]
                                                [/healthz]                            []               [get]
                                                [/healthz]                            []               [get]
                                                [/livez]                              []               [get]
                                                [/livez]                              []               [get]
                                                [/openapi/*]                          []               [get]
                                                [/openapi]                            []               [get]
                                                [/openid/v1/jwks]                     []               [get]
                                                [/readyz]                             []               [get]
                                                [/readyz]                             []               [get]
                                                [/version/]                           []               [get]
                                                [/version/]                           []               [get]
                                                [/version]                            []               [get]
                                                [/version]                            []               [get]
----------------------------------------

NS: linkerd / SA: linkerd-destination
Resources                                       Non-Resource URLs                     Resource Names   Verbs
leases.coordination.k8s.io                      []                                    []               [create get patch]
selfsubjectaccessreviews.authorization.k8s.io   []                                    []               [create]
selfsubjectrulesreviews.authorization.k8s.io    []                                    []               [create]
pods                                            []                                    []               [get list watch]
httproutes.gateway.networking.k8s.io            []                                    []               [get list watch]
authorizationpolicies.policy.linkerd.io         []                                    []               [get list watch]
httproutes.policy.linkerd.io                    []                                    []               [get list watch]
meshtlsauthentications.policy.linkerd.io        []                                    []               [get list watch]
networkauthentications.policy.linkerd.io        []                                    []               [get list watch]
serverauthorizations.policy.linkerd.io          []                                    []               [get list watch]
servers.policy.linkerd.io                       []                                    []               [get list watch]
                                                [/.well-known/openid-configuration]   []               [get]
                                                [/api/*]                              []               [get]
                                                [/api]                                []               [get]
                                                [/apis/*]                             []               [get]
                                                [/apis]                               []               [get]
                                                [/healthz]                            []               [get]
                                                [/healthz]                            []               [get]
                                                [/livez]                              []               [get]
                                                [/livez]                              []               [get]
                                                [/openapi/*]                          []               [get]
                                                [/openapi]                            []               [get]
                                                [/openid/v1/jwks]                     []               [get]
                                                [/readyz]                             []               [get]
                                                [/readyz]                             []               [get]
                                                [/version/]                           []               [get]
                                                [/version/]                           []               [get]
                                                [/version]                            []               [get]
                                                [/version]                            []               [get]
deployments.apps                                []                                    []               [get]
endpoints                                       []                                    []               [list get watch]
nodes                                           []                                    []               [list get watch]
services                                        []                                    []               [list get watch]
replicasets.apps                                []                                    []               [list get watch]
jobs.batch                                      []                                    []               [list get watch]
endpointslices.discovery.k8s.io                 []                                    []               [list get watch]
serviceprofiles.linkerd.io                      []                                    []               [list get watch]
httproutes.gateway.networking.k8s.io/status     []                                    []               [patch]
httproutes.policy.linkerd.io/status             []                                    []               [patch]
----------------------------------------

NS: linkerd / SA: linkerd-heartbeat
Resources                                       Non-Resource URLs                     Resource Names   Verbs
selfsubjectaccessreviews.authorization.k8s.io   []                                    []               [create]
selfsubjectrulesreviews.authorization.k8s.io    []                                    []               [create]
                                                [/.well-known/openid-configuration]   []               [get]
                                                [/api/*]                              []               [get]
                                                [/api]                                []               [get]
                                                [/apis/*]                             []               [get]
                                                [/apis]                               []               [get]
                                                [/healthz]                            []               [get]
                                                [/healthz]                            []               [get]
                                                [/livez]                              []               [get]
                                                [/livez]                              []               [get]
                                                [/openapi/*]                          []               [get]
                                                [/openapi]                            []               [get]
                                                [/openid/v1/jwks]                     []               [get]
                                                [/readyz]                             []               [get]
                                                [/readyz]                             []               [get]
                                                [/version/]                           []               [get]
                                                [/version/]                           []               [get]
                                                [/version]                            []               [get]
                                                [/version]                            []               [get]
namespaces                                      []                                    []               [list]
serviceprofiles.linkerd.io                      []                                    []               [list]
----------------------------------------

NS: linkerd / SA: linkerd-identity
Resources                                       Non-Resource URLs                     Resource Names   Verbs
events                                          []                                    []               [create patch]
tokenreviews.authentication.k8s.io              []                                    []               [create]
selfsubjectaccessreviews.authorization.k8s.io   []                                    []               [create]
selfsubjectrulesreviews.authorization.k8s.io    []                                    []               [create]
                                                [/.well-known/openid-configuration]   []               [get]
                                                [/api/*]                              []               [get]
                                                [/api]                                []               [get]
                                                [/apis/*]                             []               [get]
                                                [/apis]                               []               [get]
                                                [/healthz]                            []               [get]
                                                [/healthz]                            []               [get]
                                                [/livez]                              []               [get]
                                                [/livez]                              []               [get]
                                                [/openapi/*]                          []               [get]
                                                [/openapi]                            []               [get]
                                                [/openid/v1/jwks]                     []               [get]
                                                [/readyz]                             []               [get]
                                                [/readyz]                             []               [get]
                                                [/version/]                           []               [get]
                                                [/version/]                           []               [get]
                                                [/version]                            []               [get]
                                                [/version]                            []               [get]
----------------------------------------

NS: linkerd / SA: linkerd-proxy-injector
Resources                                       Non-Resource URLs                     Resource Names   Verbs
events                                          []                                    []               [create patch]
selfsubjectaccessreviews.authorization.k8s.io   []                                    []               [create]
selfsubjectrulesreviews.authorization.k8s.io    []                                    []               [create]
                                                [/.well-known/openid-configuration]   []               [get]
                                                [/api/*]                              []               [get]
                                                [/api]                                []               [get]
                                                [/apis/*]                             []               [get]
                                                [/apis]                               []               [get]
                                                [/healthz]                            []               [get]
                                                [/healthz]                            []               [get]
                                                [/livez]                              []               [get]
                                                [/livez]                              []               [get]
                                                [/openapi/*]                          []               [get]
                                                [/openapi]                            []               [get]
                                                [/openid/v1/jwks]                     []               [get]
                                                [/readyz]                             []               [get]
                                                [/readyz]                             []               [get]
                                                [/version/]                           []               [get]
                                                [/version/]                           []               [get]
                                                [/version]                            []               [get]
                                                [/version]                            []               [get]
namespaces                                      []                                    []               [list get watch]
replicationcontrollers                          []                                    []               [list get watch]
daemonsets.apps                                 []                                    []               [list get watch]
deployments.apps                                []                                    []               [list get watch]
replicasets.apps                                []                                    []               [list get watch]
statefulsets.apps                               []                                    []               [list get watch]
cronjobs.batch                                  []                                    []               [list get watch]
jobs.batch                                      []                                    []               [list get watch]
cronjobs.extensions                             []                                    []               [list get watch]
daemonsets.extensions                           []                                    []               [list get watch]
deployments.extensions                          []                                    []               [list get watch]
jobs.extensions                                 []                                    []               [list get watch]
replicasets.extensions                          []                                    []               [list get watch]
statefulsets.extensions                         []                                    []               [list get watch]
pods                                            []                                    []               [list watch]
----------------------------------------

There is something odd about this result, I did the same for my other 2 clusters, prod and demos had the exact same result but staging had some discrepancy, somehow it had even less permissions for its linkerd-service-mirror-tools-eu service account (which I would suspect is the one needing the rights ?)
(These are the differences:)

NS: linkerd-multicluster / SA: linkerd-service-mirror-tools-eu
Resources                                       Non-Resource URLs                     Resource Names   Verbs
selfsubjectaccessreviews.authorization.k8s.io   []                                    []               [create]
selfsubjectrulesreviews.authorization.k8s.io    []                                    []               [create]
                                                [/.well-known/openid-configuration]   []               [get]
                                                [/api/*]                              []               [get]
                                                [/api]                                []               [get]
                                                [/apis/*]                             []               [get]
                                                [/apis]                               []               [get]
                                                [/healthz]                            []               [get]
                                                [/healthz]                            []               [get]
                                                [/livez]                              []               [get]
                                                [/livez]                              []               [get]
                                                [/openapi/*]                          []               [get]
                                                [/openapi]                            []               [get]
                                                [/openid/v1/jwks]                     []               [get]
                                                [/readyz]                             []               [get]
                                                [/readyz]                             []               [get]
                                                [/version/]                           []               [get]
                                                [/version/]                           []               [get]
                                                [/version]                            []               [get]
                                                [/version]                            []               [get]
endpoints                                       []                                    []               [list get watch create delete update]
services                                        []                                    []               [list get watch create delete update]
namespaces                                      []                                    []               [list get watch]
----------------------------------------

NS: linkerd / SA: linkerd-destination
Resources                                       Non-Resource URLs                     Resource Names   Verbs
leases.coordination.k8s.io                      []                                    []               [create get patch]
selfsubjectaccessreviews.authorization.k8s.io   []                                    []               [create]
selfsubjectrulesreviews.authorization.k8s.io    []                                    []               [create]
pods                                            []                                    []               [get list watch]
secrets                                         []                                    []               [get list watch]
httproutes.gateway.networking.k8s.io            []                                    []               [get list watch]
authorizationpolicies.policy.linkerd.io         []                                    []               [get list watch]
httproutes.policy.linkerd.io                    []                                    []               [get list watch]
meshtlsauthentications.policy.linkerd.io        []                                    []               [get list watch]
networkauthentications.policy.linkerd.io        []                                    []               [get list watch]
serverauthorizations.policy.linkerd.io          []                                    []               [get list watch]
servers.policy.linkerd.io                       []                                    []               [get list watch]
                                                [/.well-known/openid-configuration]   []               [get]
                                                [/api/*]                              []               [get]
                                                [/api]                                []               [get]
                                                [/apis/*]                             []               [get]
                                                [/apis]                               []               [get]
                                                [/healthz]                            []               [get]
                                                [/healthz]                            []               [get]
                                                [/livez]                              []               [get]
                                                [/livez]                              []               [get]
                                                [/openapi/*]                          []               [get]
                                                [/openapi]                            []               [get]
                                                [/openid/v1/jwks]                     []               [get]
                                                [/readyz]                             []               [get]
                                                [/readyz]                             []               [get]
                                                [/version/]                           []               [get]
                                                [/version/]                           []               [get]
                                                [/version]                            []               [get]
                                                [/version]                            []               [get]
deployments.apps                                []                                    []               [get]
endpoints                                       []                                    []               [list get watch]
nodes                                           []                                    []               [list get watch]
services                                        []                                    []               [list get watch]
replicasets.apps                                []                                    []               [list get watch]
jobs.batch                                      []                                    []               [list get watch]
endpointslices.discovery.k8s.io                 []                                    []               [list get watch]
serviceprofiles.linkerd.io                      []                                    []               [list get watch]
httproutes.gateway.networking.k8s.io/status     []                                    []               [patch]
httproutes.policy.linkerd.io/status             []                                    []               [patch]
----------------------------------------

NS: linkerd / SA: linkerd-heartbeat
Resources                                       Non-Resource URLs                     Resource Names     Verbs
selfsubjectaccessreviews.authorization.k8s.io   []                                    []                 [create]
selfsubjectrulesreviews.authorization.k8s.io    []                                    []                 [create]
                                                [/.well-known/openid-configuration]   []                 [get]
                                                [/api/*]                              []                 [get]
                                                [/api]                                []                 [get]
                                                [/apis/*]                             []                 [get]
                                                [/apis]                               []                 [get]
                                                [/healthz]                            []                 [get]
                                                [/healthz]                            []                 [get]
                                                [/livez]                              []                 [get]
                                                [/livez]                              []                 [get]
                                                [/openapi/*]                          []                 [get]
                                                [/openapi]                            []                 [get]
                                                [/openid/v1/jwks]                     []                 [get]
                                                [/readyz]                             []                 [get]
                                                [/readyz]                             []                 [get]
                                                [/version/]                           []                 [get]
                                                [/version/]                           []                 [get]
                                                [/version]                            []                 [get]
                                                [/version]                            []                 [get]
configmaps                                      []                                    [linkerd-config]   [get]
namespaces                                      []                                    []                 [list]
serviceprofiles.linkerd.io                      []                                    []                 [list]
----------------------------------------

From what I can see, demos & prod's linkerd-service-mirror-tools-eu service account can get the lease, while staging cannot, yet staging and prod do work properly, and none can list them...

Any idea why there might be differences in permissions from one cluster to the other ? I unlinked and relinked all and still got this discrepancy.
All clusters run the latest versions of the crds, control-plane & multicluster helm charts (1.8.0, 1.16.2 & 30.11.2 respectively)

[mateiidavid]

The difference in permissions does seem a bit odd. Is your client (linkerd CLI) up-to-date? Linking and re-linking will be done through the CLI. Also, the CLI is the one effectively making the request during linkerd check. Whatever service account your kubeconfig context is using should allow for listing / getting Lease resources.

[lboutin-mwm]

linkerd CLI version:

> linkerd version --client
Client version: stable-2.14.1

I used this loop to check my permissions across clusters:

for env in ml-demos-us ml-staging-us ml-prod-us tools-eu; do
  kubectx $env
  kubectl auth can-i list leases --namespace linkerd
  kubectl auth can-i get leases --namespace linkerd
  kubectl auth can-i list leases --namespace linkerd-multicluster
  kubectl auth can-i get leases --namespace linkerd-multicluster
done

Resulting in:

> bash debug
Switched to context "ml-demos-us".
yes
yes
yes
yes
Switched to context "ml-staging-us".
yes
yes
yes
yes
Switched to context "ml-prod-us".
yes
yes
yes
yes
Switched to context "tools-eu".
yes
yes
yes
yes

[lboutin-mwm]

Upgrading to linkerd version stable-2.14.2, then unlink / re-linking didn't have any impact
Still unable to get the leases but the traffic still goes through

[lboutin-mwm]

No worries, linkerd's functionalities are still working so the bug isn't critical, I'd just like to be understand why the check fails.

Deleting the Lease doesn't do much, it gets recreated instantly but there is no effect on the result of the check.

Trying to create a new lease with the same name does result in an error:

> kctl create -f test.yaml -n linkerd-multicluster
Error from server (AlreadyExists): error when creating "test.yaml": leases.coordination.k8s.io "service-mirror-write-tools-eu" already exists

I guess I could try the custom CLI, but I'm unsure how to get access to it ?

[mateiidavid]

@lboutin-mwm I got an example CLI build for you to try. Hopefully this might give us more information. We can tweak this if we need more information still.

Here's what I've done:

• Turn up klog debugging levels to 7 (this should be the equivalent of a DEBUG). This is the logger used by client-go's http client. We should now be able to see at the very least the response from the server.
• When the check errors, we will now also have some additional CLI log records that will fetch all of the Lease resources in the multicluster namespace.

Let's see if this is enough for us to have a better idea about the error. I've attached the binary to this discussion post. I built it for osx-amd64. If you run a different OS and/or arch lmk and I can re-build it.

linkerd-darwin-amd64.tar.gz

[lboutin-mwm]

Hi, sorry for the late reply
I used the custom build and we seem to have some insight into the problem

It looks like the check command isn't looking for the lease in the correct namespace:

"failed to find service-mirror lease for tools-eu in linkerd: %!s(MISSING)"
"fetching leases"
"Leases in linkerd namespace: [...]"

But the lease is in the linkerd-multicluster namespace

What I don't understand now is that the lease is in the linkerd-multicluster for all clusters, but it only fails for this one
What's the standard here ? Should the lease be in the linkerd namespace or the linkerd-multicluster one ?

Sidenote: I tried to unlink / re-link with the new cli but it seems to point to an image that I don't have access to (cr.l5d.io/linkerd/controller:dev-1e80aec2-matei) so I can only link with the stable-2.14.2 realease and check with the custom build

[mateiidavid]

@lboutin-mwm interesting! the check command finds the extension's namespace by using an annotation (or label, can't remember) selector. Can you paste the annotations and labels of the linkerd namespace? It might be erroneously selected.

The CLI binary I shipped has an embedded tag, any manifests that come from it will use images pinned to that tag. There is no functional difference between that and the stable CLI though.

[lboutin-mwm]

Okay, I'm guessing we found the problem then, these are the labels of my linkerd namespace for this cluster (and this cluster only) :

  labels:
    kubernetes.io/metadata.name: linkerd
    linkerd.io/extension: multicluster
    pod-security.kubernetes.io/enforce: privileged

all other clusters have only:

  labels:
    kubernetes.io/metadata.name: linkerd

Any idea how these labels might have been added ?

I deleted the additional labels and the check is working just fine, thanks a lot

[mateiidavid]

@lboutin-mwm it's hard to say. Did you install through helm? We have a namespace metadata job that will correctly label the install namespace with the extension name. Perhaps there was some configuration mismanagement there? Either way, after talking to the rest of the team we think a core check that will assert there aren't two namespaces with the same label value would be good to have to quickly debug mysterious cases like this one :)

[lboutin-mwm]

Namespace, trust anchros and trust roots were installed beforehand through terraform, then the actual resources were deployed through kubectl via argocd. I'll be sure to double check
Thank you for your help, hope this will help others if such an error occurs again

[mateiidavid]

@lboutin-mwm we've merged a CLI change that will surface duplicate extension labels in the future. That'll hopefully make it easier for us all to debug this if it ever happens again :)