Proxies sidecars not being injected EKS

[Ken501]

Cluster: EKS V.128
OS: Linux
CNI: Calico

Currently deploying linkerd with linkerd cli. Everything goes smoothly except sidecar proxies are not being created for any release, especially releases/pods running with hostNetworking. Annotation linkerd.io/inject: enabled. After applying the following:

kubectl get -n emojivoto deploy -o yaml \                                                                                                             ─╯
  | linkerd inject - \
  | kubectl apply -f -

Nothing happens other than annotation mention above being included.

linkerd check cli command exceptions:

❯ linkerd check
kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version

linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ control plane pods are ready
√ cluster networks contains all pods
√ cluster networks contains all services

linkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ proxy-init container runs as root user if docker container runtime is used

linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust anchor

linkerd-webhooks-and-apisvc-tls
-------------------------------
√ proxy-injector webhook has valid cert
√ proxy-injector cert is valid for at least 60 days
√ sp-validator webhook has valid cert
√ sp-validator cert is valid for at least 60 days
√ policy-validator webhook has valid cert
√ policy-validator cert is valid for at least 60 days

linkerd-version
---------------
√ can determine the latest version
√ cli is up-to-date

control-plane-version
---------------------
√ can retrieve the control plane version
√ control plane is up-to-date
√ control plane and cli versions match

linkerd-control-plane-proxy
---------------------------
√ control plane proxies are healthy
√ control plane proxies are up-to-date
√ control plane proxies and cli versions match

linkerd-extension-checks
------------------------
√ namespace configuration for extensions

Status check results are √

• LinkerD pod logs:

  • There does seem to be warning on pod logs but I have been unable to solve or actually find out what may be the root cause:

  linkerd-destination-* logs:

│ linkerd-proxy [ 0.728099s] WARN ThreadId(01) policy:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to con │
│ nect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)] │
│ linkerd-proxy [ 0.747320s] WARN ThreadId(01) dst:controller{addr=localhost:8086}:endpoint{addr=127.0.0.1:8086}: linkerd_reconnect: Failed to connec │
│ t error=endpoint 127.0.0.1:8086: Connection refused (os error 111) error.sources=[Connection refused (os error 111)] │
│ linkerd-proxy [ 1.253964s] WARN ThreadId(01) policy:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to con │
│ nect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]

`linkerd-identity-6f558c4545-*`

```text
│ linkerd-proxy [    30.006486s]  INFO ThreadId(01) policy:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_proxy_balance_queue::wo │
│ rker: Unavailable; entering failfast timeout=30.0                                                                                                        │
│ linkerd-proxy [    30.006525s]  WARN ThreadId(01) watch{port=9990}: linkerd_app_inbound::policy::api: Unexpected policy controller response; retrying wi │
│ th a backoff grpc.status=Unknown error grpc.message="controller linkerd-policy.linkerd.svc.cluster.local:8090: service in fail-fast"                     │
│ linkerd-proxy [    30.006558s]  WARN ThreadId(01) watch{port=4191}: linkerd_app_inbound::policy::api: Unexpected policy controller response; retrying wi │
│ th a backoff grpc.status=Unknown error grpc.message="controller linkerd-policy.linkerd.svc.cluster.local:8090: service in fail-fast"                     │
│ linkerd-proxy [    30.006569s]  WARN ThreadId(01) watch{port=8080}: linkerd_app_inbound::policy::api: Unexpected policy controller response; retrying wi │
│ th a backoff grpc.status=Unknown error grpc.message="controller linkerd-policy.linkerd.svc.cluster.local:8090: service in fail-fast"                     │
│ linkerd-proxy [    30.685482s]  WARN ThreadId(01) linkerd_app::dst: Unexpected policy controller response; retrying with a backoff grpc.status=Deadline  │
│ expired before operation could complete grpc.message="initial item not received within timeout"                                                          │
│ linkerd-proxy [    32.008348s]  INFO ThreadId(01) dst:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_pool_p2c: Adding end │
│ point addr=10.244.221.82:8086

linkerd-proxy-injector-*

│ linkerd-proxy [     0.007308s]  INFO ThreadId(01) dst:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_pool_p2c: Adding end │
│ point addr=10.244.221.82:8086                                                                                                                            │
│ linkerd-proxy [     0.008472s]  INFO ThreadId(02) identity:identity{server.addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:controller{add │
│ r=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_pool_p2c: Adding endpoint addr=10.244.221.81:8080                                   │
│ linkerd-proxy [     0.023423s]  INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity id=linkerd-proxy-injector.linkerd.serviceaccount.iden │
│ tity.linkerd.cluster.local                                                                                                                               │
│ linkerd-network-validator 2024-07-12T20:02:43.302796Z  INFO linkerd_network_validator: Listening for connections on 0.0.0.0:4140                         │
│ linkerd-network-validator 2024-07-12T20:02:43.302826Z DEBUG linkerd_network_validator: token="l5yGT9FvTb3CKffzXNsKSbMgmYlaxDwIUDA0DfNFbwYkFRUhdrnEfb450U │
│ IJpjr\n"

Exception when installing on a release with hostNetwork enabled (cert-manager):

❯ kubectl get -n cert-manager deploy -o yaml \
  | linkerd inject - \
  | kubectl apply -f -
Error transforming resources:
failed to inject deployment/cert-manager-webhook: hostNetwork is enablederror: no objects passed to apply

No particular exception shows when running same command against emojivoto demo release. Any help to diagnose further would be greatly appreciated.

[alpeb]

This is expected; linkerd doesn't inject pods with host networking, because it needs to apply rerouting rules to capture traffic (with iptables) constrained to the pod network, and not at the host level.