tap.linkerd.io/v1alpha1: the server is currently unable to handle the request #10381

joscabmar · 2023-02-16T11:42:58Z

joscabmar
Feb 16, 2023

What is the issue?

I am having issues with getting tap working on EKS. I installed the following helm charts: linkerd-crds, linkerd-control-plane, linkerd-viz and linkerd-jaeger. When I check in the viz dashboard, and I click on tap, I get the following error:

How can it be reproduced?

I used the official Helm charts (https://artifacthub.io/packages/helm/linkerd2/linkerd-viz)

The values are:

prometheusUrl: "http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090"
jaegerUrl: jaeger.linkerd-jaeger.svc.cluster.local:16686
prometheus:
  enabled: false
grafana:
  externalUrl: "http://localhost:9080"
tap:
  externalSecret: true
  injectCaFrom: linkerd-viz/tap
tapInjector:
  externalSecret: true
  injectCaFrom: linkerd-viz/linkerd-tap-injector

Logs, error output, etc

When I check the linkerd pods, they all seem to be working fine. And tap shows the following logs, which seem ok as well:

$ kubectl get pods -A |grep linkerd
linkerd-jaeger     collector-8447cddbd-hh4dl                                    2/2     Running     0                7h30m
linkerd-jaeger     jaeger-794c896c4d-b9tfw                                      2/2     Running     0                169m
linkerd-jaeger     jaeger-injector-545dd46598-jl222                             2/2     Running     0                7h30m
linkerd-viz        metrics-api-5f467bcd87-qsck9                                 2/2     Running     0                115m
linkerd-viz        prometheus-57b5d94d78-z7vn9                                  2/2     Running     0                7h30m
linkerd-viz        tap-6cd779f9df-sxxrr                                         2/2     Running     0                4h38m
linkerd-viz        tap-injector-6d78b4bb85-b4v9g                                2/2     Running     0                7h30m
linkerd-viz        web-9bcb756bd-kl4h7                                          2/2     Running     0                115m
linkerd            linkerd-destination-8447f5d877-b84cc                         4/4     Running     0                7h31m
linkerd            linkerd-destination-8447f5d877-gxp82                         4/4     Running     0                7h31m
linkerd            linkerd-destination-8447f5d877-rtth6                         4/4     Running     0                3h25m
linkerd            linkerd-identity-77d4b595cd-94qfg                            2/2     Running     0                7h31m
linkerd            linkerd-identity-77d4b595cd-brlqf                            2/2     Running     0                3h25m
linkerd            linkerd-identity-77d4b595cd-zkj9n                            2/2     Running     0                169m
linkerd            linkerd-proxy-injector-764d47fb9d-ht5hj                      2/2     Running     0                4h38m
linkerd            linkerd-proxy-injector-764d47fb9d-mzn8k                      2/2     Running     0                169m
linkerd            linkerd-proxy-injector-764d47fb9d-xvbkh                      2/2     Running     0                7h31m
$ kubectl logs tap-6cd779f9df-sxxrr -n linkerd-viz -c tap
time="2023-01-25T08:54:37Z" level=info msg="running version dev-undefined"
time="2023-01-25T08:54:37Z" level=info msg="Using trust domain: cluster.local"
time="2023-01-25T08:54:37Z" level=info msg="waiting for caches to sync"
time="2023-01-25T08:54:38Z" level=info msg="caches synced"
time="2023-01-25T08:54:38Z" level=info msg="starting admin server on :9998"
time="2023-01-25T08:54:38Z" level=info msg="starting tap API server on :8089" addr=":8089" component=tap

But when I check the api-resources, I get an error from tap:

$ kubectl api-resources
NAME                              SHORTNAMES              APIVERSION                                NAMESPACED   KIND
...
error: unable to retrieve the complete list of server APIs: tap.linkerd.io/v1alpha1: the server is currently unable to handle the request

output of `linkerd check -o short`

Linkerd core checks
===================

linkerd-identity
----------------
‼ issuer cert is valid for at least 60 days
    issuer certificate will expire on 2023-02-18T06:01:42Z
    see https://linkerd.io/2.12/checks/#l5d-identity-issuer-cert-not-expiring-soon for hints

linkerd-webhooks-and-apisvc-tls
-------------------------------
‼ proxy-injector cert is valid for at least 60 days
    certificate will expire on 2023-02-17T06:01:41Z
    see https://linkerd.io/2.12/checks/#l5d-proxy-injector-webhook-cert-not-expiring-soon for hints
‼ sp-validator cert is valid for at least 60 days
    certificate will expire on 2023-02-17T06:01:41Z
    see https://linkerd.io/2.12/checks/#l5d-sp-validator-webhook-cert-not-expiring-soon for hints
‼ policy-validator cert is valid for at least 60 days
    certificate will expire on 2023-02-17T06:01:41Z
    see https://linkerd.io/2.12/checks/#l5d-policy-validator-webhook-cert-not-expiring-soon for hints

Linkerd extensions checks
=========================

linkerd-viz
-----------
‼ tap API server cert is valid for at least 60 days
    certificate will expire on 2023-02-17T06:01:42Z
    see https://linkerd.io/2.12/checks/#l5d-tap-cert-not-expiring-soon for hints
‼ tap API service is running
    FailedDiscoveryCheck: failing or missing response from https://10.1.133.241:8089/apis/tap.linkerd.io/v1alpha1: Get "https://10.1.133.241:8089/apis/tap.linkerd.io/v1alpha1": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
    see https://linkerd.io/2.12/checks/#l5d-tap-api for hints

Status check results are √

Environment

Kubernetes 1.22
EKS
Linkerd stable-2.12.4

Possible solution

I was told it might be due to a firewall issue. But I am allowing the following ports:

cluster_security_group_additional_rules = {
    egress_nodes_ephemeral_ports_tcp = {
      description                = "To node 1025-65535"
      protocol                   = "tcp"
      from_port                  = 1025
      to_port                    = 65535
      type                       = "egress"
      source_node_security_group = true
    }
    ingress_linkerd = {
      description = "Linkerd"
      protocol    = "tcp"
      from_port   = 8443
      to_port     = 8443
      type        = "egress"
      source_node_security_group = true
    }
    ingress_linkerd_tap = {
      description = "Linkerd Tap"
      protocol    = "tcp"
      from_port   = 8089
      to_port     = 8089
      type        = "egress"
      source_node_security_group = true
    }
    ingress_linkerd_tap = {
      description = "Linkerd Tap"
      protocol    = "tcp"
      from_port   = 9443
      to_port     = 9443
      type        = "egress"
      source_node_security_group = true
    }
  }
# Extend node-to-node security group rules
  node_security_group_additional_rules = {
    ingress_self_all = {
      description = "Node to node all ports/protocols"
      protocol    = "-1"
      from_port   = 0
      to_port     = 0
      type        = "ingress"
      self        = true
    }
    egress_all = {
      description = "Node all egress"
      protocol    = "-1"
      from_port   = 0
      to_port     = 0
      type        = "egress"
      cidr_blocks = ["0.0.0.0/0"]
    }
    ingress_linkerd_tap_8443 = {
      description = "linkerd/tap service"
      protocol    = "tcp"
      from_port   = 8443
      to_port     = 8443
      type        = "ingress"
      source_node_security_group = true
    }
    ingress_linkerd_tap_8089 = {
      description = "linkerd/tap service"
      protocol    = "tcp"
      from_port   = 8089
      to_port     = 8089
      type        = "ingress"
      source_node_security_group = true
    }
    ingress_linkerd_tap_8089 = {
      description = "linkerd/tap service"
      protocol    = "tcp"
      from_port   = 9443
      to_port     = 9443
      type        = "ingress"
      source_node_security_group = true
    }
  }

Additional context

No response

Would you like to work on fixing this bug?

yes

Answered by joscabmar

Feb 24, 2023

Sure @jeremychase. I attached here the logs with debug enabled in tap. And its linkerd-proxy log: https://gist.github.com/joscabmar/912fce3100b271047c26ac39f9fa0368

View full answer

kleimkuhler · 2023-02-17T22:06:31Z

kleimkuhler
Feb 17, 2023

Can you provide any more information in regards to the tap API Service causing the error in linkerd check?

‼ tap API service is running
    FailedDiscoveryCheck: failing or missing response from https://10.1.133.241:8089/apis/tap.linkerd.io/v1alpha1: Get "https://10.1.133.241:8089/apis/tap.linkerd.io/v1alpha1": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
    see https://linkerd.io/2.12/checks/#l5d-tap-api for hints

I know you've listed that the firewall rules should allow it, but based off debugging this hint it does seem like that may the issue, or something else is incorrect about the Service/Pod. I'd go through the documentation at that link and double check that things seem okay.

0 replies

joscabmar · 2023-02-20T11:50:15Z

joscabmar
Feb 20, 2023
Author

Well, I can only spot the error in the linkerd-viz dashboard when trying to use tap. Or when I search for the api-resources:

$ kubectl api-resources
NAME                              SHORTNAMES              APIVERSION                                NAMESPACED   KIND
...
error: unable to retrieve the complete list of server APIs: tap.linkerd.io/v1alpha1: the server is currently unable to handle the request

You can see that the logs in tap service seem to be ok when trying to debug it:

$ kubectl logs tap-6cd779f9df-sxxrr -n linkerd-viz -c tap
time="2023-01-25T08:54:37Z" level=info msg="running version dev-undefined"
time="2023-01-25T08:54:37Z" level=info msg="Using trust domain: cluster.local"
time="2023-01-25T08:54:37Z" level=info msg="waiting for caches to sync"
time="2023-01-25T08:54:38Z" level=info msg="caches synced"
time="2023-01-25T08:54:38Z" level=info msg="starting admin server on :9998"
time="2023-01-25T08:54:38Z" level=info msg="starting tap API server on :8089" addr=":8089" component=tap

0 replies

joscabmar · 2023-02-23T16:21:19Z

joscabmar
Feb 23, 2023
Author

Here you can find the full linkerd check command output:

linkerd check
Linkerd core checks
===================

kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version

linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ control plane pods are ready
√ cluster networks contains all pods
√ cluster networks contains all services

linkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ proxy-init container runs as root user if docker container runtime is used

linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
‼ issuer cert is valid for at least 60 days
    issuer certificate will expire on 2023-02-25T06:01:48Z
    see https://linkerd.io/2.12/checks/#l5d-identity-issuer-cert-not-expiring-soon for hints
√ issuer cert is issued by the trust anchor

linkerd-webhooks-and-apisvc-tls
-------------------------------
√ proxy-injector webhook has valid cert
‼ proxy-injector cert is valid for at least 60 days
    certificate will expire on 2023-02-24T06:01:47Z
    see https://linkerd.io/2.12/checks/#l5d-proxy-injector-webhook-cert-not-expiring-soon for hints
√ sp-validator webhook has valid cert
‼ sp-validator cert is valid for at least 60 days
    certificate will expire on 2023-02-24T06:01:47Z
    see https://linkerd.io/2.12/checks/#l5d-sp-validator-webhook-cert-not-expiring-soon for hints
√ policy-validator webhook has valid cert
‼ policy-validator cert is valid for at least 60 days
    certificate will expire on 2023-02-24T06:01:47Z
    see https://linkerd.io/2.12/checks/#l5d-policy-validator-webhook-cert-not-expiring-soon for hints

linkerd-version
---------------
√ can determine the latest version
√ cli is up-to-date

control-plane-version
---------------------
√ can retrieve the control plane version
√ control plane is up-to-date
√ control plane and cli versions match

linkerd-control-plane-proxy
---------------------------
√ control plane proxies are healthy
√ control plane proxies are up-to-date
√ control plane proxies and cli versions match

linkerd-ha-checks
-----------------
√ pod injection disabled on kube-system

Linkerd extensions checks
=========================

linkerd-jaeger
--------------
√ linkerd-jaeger extension Namespace exists
√ jaeger extension pods are injected
√ jaeger injector pods are running
√ jaeger extension proxies are healthy
√ jaeger extension proxies are up-to-date
√ jaeger extension proxies and cli versions match

linkerd-viz
-----------
√ linkerd-viz Namespace exists
√ linkerd-viz ClusterRoles exist
√ linkerd-viz ClusterRoleBindings exist
√ tap API server has valid cert
‼ tap API server cert is valid for at least 60 days
    certificate will expire on 2023-02-24T06:01:47Z
    see https://linkerd.io/2.12/checks/#l5d-tap-cert-not-expiring-soon for hints
‼ tap API service is running
    FailedDiscoveryCheck: failing or missing response from https://10.1.133.93:8089/apis/tap.linkerd.io/v1alpha1: Get "https://10.1.133.93:8089/apis/tap.linkerd.io/v1alpha1": context deadline exceeded
    see https://linkerd.io/2.12/checks/#l5d-tap-api for hints
√ linkerd-viz pods are injected
√ viz extension pods are running
√ viz extension proxies are healthy
√ viz extension proxies are up-to-date
√ viz extension proxies and cli versions match
√ can initialize the client
√ viz extension self-check

0 replies

jeremychase · 2023-02-23T20:51:48Z

jeremychase
Feb 23, 2023

@joscabmar I converted this to a discussion until we figure out if there is an issue here!

The tap pod is running and you are able to get its logs, but check is reporting that the tap API service isn't running. Could you switch the proxy log level to info,linkerd2_proxy=debug, reproduce, and then send the proxy logs?

0 replies

joscabmar · 2023-02-24T12:32:36Z

joscabmar
Feb 24, 2023
Author

Sure @jeremychase. I attached here the logs with debug enabled in tap. And its linkerd-proxy log: https://gist.github.com/joscabmar/912fce3100b271047c26ac39f9fa0368

2 replies

alpeb Mar 2, 2023
Collaborator

Sure @jeremychase. I attached here the logs with debug enabled in tap. And its linkerd-proxy log: https://gist.github.com/joscabmar/912fce3100b271047c26ac39f9fa0368

Given those logs, it appears the tap service itself is running fine, but not receiving traffic from the kube-api. I don't know what firewall setup you're using so this is probably a dumb question: there are a couple of entries with duplicate keys ingress_linkerd_tap and ingress_linkerd_tap_8089; is it possible they might be conflicting?

joscabmar Mar 14, 2023
Author

Thank you so much @alpeb indeed, that was the issue. I was naming both rules with the same name (ingress_linkerd_tap_8089) and the first one was getting ignored. Sorry for my mistake and thank you again.

bbatha · 2023-03-02T16:43:44Z

bbatha
Mar 2, 2023

I am also experiencing this issue on a brand new kubernetes 1.25 EKS cluster. This issue also causes problems when deleting namespaces as the kube-api server times out when it attempts to verify that the there are no namespaced objects in the v1alpha1.tap.linkerd.io group.

0 replies

robbo10 · 2024-09-09T15:24:24Z

robbo10
Sep 9, 2024

Hi there,

We are also facing this issue on EKS, we are not running the VPC CNI and therefore pods don't have direct access to the EKS API.

The error from the tap APIService is shown below:
Message: failing or missing response from https://100.1.2.3:9723/apis/tap.linkerd.io/v1alpha1: bad status from https://100.1.2.3:9723/apis/tap.linkerd.io/v1alpha1: 400

100.1.2.3 is our nginx pod proxy IP which runs on the host network to forward traffic to the tap svc.

We have enabled trace logging on the tap deployment which shows the below:

time="2024-09-09T14:28:09Z" level=debug msg="ServeHTTP(): &{Method:GET URL:/apis/tap.linkerd.io/v1alpha1 Proto:HTTP/2.0 ProtoMajor:2 ProtoMinor:0 Header:map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/2.0] X-Amzn-Trace-Id:[Root=1-66df05f9-6e5840c] X-Forwarded-For:[100.1.1.5, 100.1.1.6] X-Forwarded-Host:[linkerd-tap-webhook.sys.example.com] X-Forwarded-Port:[443] X-Forwarded-Proto:[https] X-Forwarded-Server:[traefik-ingress-controller] X-Real-Ip:[100.1.1.6] X-Remote-Group:[system:masters] X-Remote-User:[system:kube-aggregator]] Body:0xc0017cef00 GetBody:<nil> ContentLength:0 TransferEncoding:[] Close:false Host:linkerd-tap-webhook.sys.example.com Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr:10.42.7.53:60966 RequestURI:/apis/tap.linkerd.io/v1alpha1 TLS:0xc00088edc0 Cancel:<nil> Response:<nil> ctx:0xc0017ceff0 pat:<nil> matches:[] otherValues:map[]}" addr=":8089" component=tap time="2024-09-09T14:28:09Z" level=debug msg="no valid CN found. allowed names: [front-proxy-client], client names: []" addr=":8089" component=tap

From doing research it looks like we might need to update the extension-apiserver-authentication configmap in the kube-system namespace, however we have had no joy in trying to update this :(

Has anyone got any recommendations for how to get around this issue?

Thank you :)

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tap.linkerd.io/v1alpha1: the server is currently unable to handle the request #10381

{{title}}

Replies: 7 comments 2 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

tap.linkerd.io/v1alpha1: the server is currently unable to handle the request #10381

joscabmar Feb 16, 2023

What is the issue?

How can it be reproduced?

Logs, error output, etc

output of linkerd check -o short

Environment

Possible solution

Additional context

Would you like to work on fixing this bug?

Replies: 7 comments · 2 replies

kleimkuhler Feb 17, 2023

joscabmar Feb 20, 2023 Author

joscabmar Feb 23, 2023 Author

jeremychase Feb 23, 2023

joscabmar Feb 24, 2023 Author

alpeb Mar 2, 2023 Collaborator

joscabmar Mar 14, 2023 Author

bbatha Mar 2, 2023

robbo10 Sep 9, 2024

joscabmar
Feb 16, 2023

output of `linkerd check -o short`

Replies: 7 comments 2 replies

kleimkuhler
Feb 17, 2023

joscabmar
Feb 20, 2023
Author

joscabmar
Feb 23, 2023
Author

jeremychase
Feb 23, 2023

joscabmar
Feb 24, 2023
Author

alpeb Mar 2, 2023
Collaborator

joscabmar Mar 14, 2023
Author

bbatha
Mar 2, 2023

robbo10
Sep 9, 2024