otel python auto instrumentation

[tejaaa45]

Component(s)

collector, auto-instrumentation

What happened?

Description

Steps to Reproduce

Expected Result

Actual Result

I'm setting up distributed tracing and end-to-end transaction monitoring for my services, which are hosted in an AKS (Azure Kubernetes Service) cluster and built using the Python framework. Since we use a microservices architecture, I'm implementing OpenTelemetry (OTel) with auto-instrumentation. This involves deploying the OTel Operator in AKS along with an OTel Collector and using Jaeger for tracing and ELK as backend storage

As part of the setup, I added the necessary Python-related annotations to enable tracing for four of my services. However, while I can see trace data for two of the services, the other two are not sending any data. Instead, I see an error message:

Could not find the ORIG_HANDLER or _HANDLER in the environment variables. This instrumentation requires the OpenTelemetry Lambda extension installed.

Since our environment is in Azure and not AWS, I don't understand why this error is appearing. What could be causing this issue, and how can I resolve it?

Kubernetes Version

1.29.5

Operator version

0.76.0

Collector version

latest

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

Log output

No response

Additional context

No response

[tejaaa45]

@swiatekm @TylerHelmuth @opentelemetrybot can any of you guys help me on this?

[tejaaa45]

@pmcollins can you help me on this?

[swiatekm]

Operator version

0.76.0

Do you actually mean the operator version here, or the Helm Chart version?

[tejaaa45]

@swiatekm its the helm chart version this is how i have done my setup--

helm install opentelemetry-operator open-telemetry/opentelemetry-operator --namespace opentelemetry-operator --create-namespace --set "manager.collectorImage.repository=otel/opentelemetry-collector-k8s"
--set admissionWebhooks.certManager.enabled=false
--set admissionWebhooks.autoGenerateCert.enabled=true

otel-collector.yaml

---otel collector deployment ---

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
name: traces
spec:
mode: deployment
config:
receivers:
otlp:
protocols:
grpc: {}
http: {}
processors:
memory_limiter:
check_interval: 1s
limit_percentage: 75
spike_limit_percentage: 15
batch:
send_batch_size: 10000
timeout: 10s
exporters:
otlp:
# endpoint: "jaeger-collector:4317"
endpoint: "jaeger-collector.observability.svc.cluster.local:4317"
# endpoint: "jaeger-collector.jaeger.svc.cluster.local:4317"
tls:
insecure: true
debug: {}
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [debug, otlp]

instrumentation,yaml

---instrumentation for collector deployment---

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: otel-instrumentation
spec:
exporter:
endpoint: http://traces-collector.opentelemetry-operator.svc.cluster.local:4317
propagators:
- baggage
- b3
sampler:
type: parentbased_traceidratio
argument: "1"
python:
env:
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: http://traces-collector.opentelemetry-operator.svc.cluster.local:4318/v1/traces
- name: OTEL_PYTHON_DISABLED_INSTRUMENTATIONS
value: aws-lambda,metrics
- name: OTEL_METRICS_EXPORTER
value: none
- name: OTEL_LOG_LEVEL
value: "debug"
image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest

dotnet:

env:

- name: OTEL_EXPORTER_OTLP_ENDPOINT

value: http://otel-collector:4318

go:

env:

- name: OTEL_EXPORTER_OTLP_ENDPOINT

value: http://otel-collector:4318

again i changed the instrumentation.yaml

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
annotations:
instrumentation.opentelemetry.io/default-auto-instrumentation-apache-httpd-image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-apache-httpd:latest
instrumentation.opentelemetry.io/default-auto-instrumentation-dotnet-image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-dotnet:latest
instrumentation.opentelemetry.io/default-auto-instrumentation-go-image: ghcr.io/open-telemetry/opentelemetry-go-instrumentation/autoinstrumentation-go:latest
instrumentation.opentelemetry.io/default-auto-instrumentation-java-image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest
instrumentation.opentelemetry.io/default-auto-instrumentation-nodejs-image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:latest
instrumentation.opentelemetry.io/default-auto-instrumentation-python-image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest
name: otel-instrumentation
spec:

apacheHttpd:

configPath: /usr/local/apache2/conf

image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-apache-httpd:1.0.3

version: "2.4"

dotnet:
env:
- name: OTEL_METRICS_EXPORTER
value: none
- name: OTEL_LOGS_EXPORTER
value: none
- name: OTEL_TRACES_EXPORTER
value: otlp
image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-dotnet:latest
exporter: {}
java:
env:
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: http://traces-collector.observability.svc.cluster.local:4317
- name: OTEL_METRICS_EXPORTER
value: none
- name: OTEL_LOGS_EXPORTER
value: none
- name: OTEL_TRACES_EXPORTER
value: otlp
image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest
nodejs:
env:
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: http://traces-collector.observability.svc.cluster.local:4317
- name: OTEL_METRICS_EXPORTER
value: none
- name: OTEL_LOGS_EXPORTER
value: none
- name: OTEL_TRACES_EXPORTER
value: otlp
image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:latest
propagators:

- tracecontext

• baggage
• b3multi
• b3
  python:
  env:
  • name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: http://traces-collector.observability.svc.cluster.local:4318
  • name: OTEL_METRICS_EXPORTER
    value: none
  • name: OTEL_LOGS_EXPORTER
    value: none
  • name: OTEL_TRACES_EXPORTER
    value: otlp
    resource: {}
    sampler: {}

and i have added these annotations
instrumentation.opentelemetry.io/inject-python:true
instrumentation.opentelemetry.io/otel-python-platform:musl
instrumentation.opentelemetry.io/container-names

before adding this envi variables

• name: OTEL_PYTHON_DISABLED_INSTRUMENTATIONS
  value: aws-lambda

   i'm getting this 
   Could not find the ORIG_HANDLER or _HANDLER in the environment variables. This instrumentation requires the OpenTelemetry Lambda extension installed.
   
   and after adding that envi variables that error is gone but traces are not exporting and there is no logs related to otel aftyer adding that envi variable
  

[swiatekm]

Apologies for not getting back to you sooner. I see you've also opened an issue in python-contrib, which is the better place to ask, as this seems like more of an instrumentation problem. We can keep this one open in case it turns out the operator has something to do with it after all.