TRACING-4752: Add OpenTelemetry-Collector as optional sub-package #4281

copejon · 2024-12-06T14:46:46Z

Which issue(s) this PR addresses:

Closes #

openshift-ci · 2024-12-06T14:46:50Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

openshift-ci · 2024-12-06T14:47:00Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: copejon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [copejon]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

packaging/rpm/microshift.spec

packaging/observability/opentelemetry-collector.yaml

packaging/rpm/microshift.spec

packaging/observability/microshift-observability.service

ggiguash · 2024-12-09T07:37:27Z

packaging/observability/microshift-observability.service

@@ -0,0 +1,27 @@
+[Unit]
+Description=MicroShift Observability
+BindsTo=microshift.service


Do we want to run the collector even when MicroShift fails?

I'd think yes. If MicroShift fails to start, the metrics and log data should still be collectable by the metrics/logging backend remotely.

packaging/observability/microshift-observability.service

ggiguash · 2024-12-09T07:49:41Z

/retitle NO-ISSUE: OpenTelemetry certificates and service for MicroShift

openshift-ci-robot · 2024-12-09T07:49:47Z

@copejon: This pull request explicitly references no jira issue.

In response to this:

Which issue(s) this PR addresses:

Closes #

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

packaging/observability/opentelemetry-collector.yaml

openshift-ci-robot · 2024-12-12T20:47:24Z

@copejon: This pull request references TRACING-4752 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

Which issue(s) this PR addresses:

Closes #

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

copejon · 2024-12-12T20:47:28Z

/jira refresh

openshift-ci-robot · 2024-12-12T20:47:31Z

@copejon: This pull request references TRACING-4752 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

implemented opentelemetry-collector in packaging pipeline. the otel binary is unable to see the cert files, but the config paths are correct and the files exist. file permissions for the observability user have been checked, but are good. WIP Signed-off-by: Jon Cope <[email protected]>

ensure the otelcol process creates necessary dirs added firewall port handling for otel-col exporter Signed-off-by: Jon Cope <[email protected]>

… on the localhost Signed-off-by: Jon Cope <[email protected]>

Signed-off-by: Jon Cope <[email protected]>

… to allow the collection of pod logs from the host filesystem Signed-off-by: Jon Cope <[email protected]>

Signed-off-by: Jon Cope <[email protected]>

…s more sense that observability should a) not be killed if microshift fails during start or runtime b) not cause the microshift service to start, and only be started after microshift completes Signed-off-by: Jon Cope <[email protected]>

…e extension systemd unit sets microshift.service for RequiredBy=, meaning that if observability fails to start, microshift should be killed. otel config now includes microshift-etcd, microshift-tuned services in log collection. Removed debugging exporter. Added filelog pipeline Signed-off-by: Jon Cope <[email protected]>

Signed-off-by: Jon Cope <[email protected]>

ggiguash · 2025-01-23T08:30:03Z

packaging/rpm/microshift.spec

+Requires: opentelemetry-collector
+
+%description observability
+Deploys the Red Hat build of Opentelemetry-collector as a systemd service on host. MicroShift provides client


We need to be consistent in the naming case. Either fix this, or the Summary section, please.

Suggested change

Deploys the Red Hat build of Opentelemetry-collector as a systemd service on host. MicroShift provides client

Deploys the Red Hat build of OpenTelemetry-Collector as a systemd service on host. MicroShift provides client

ggiguash · 2025-01-23T08:32:37Z

packaging/rpm/microshift.spec

+certificates to permit access to the kube-apiserver metrics endpoints. If a user defined opentelemetry-collector exists
+at /etc/microshift/opentelemetry-collector.yaml, this config is used. Otherwise, a default config is provided. Note that
+the default configuration requires the backend endpoint be set by the user.


Suggested change

certificates to permit access to the kube-apiserver metrics endpoints. If a user defined opentelemetry-collector exists

at /etc/microshift/opentelemetry-collector.yaml, this config is used. Otherwise, a default config is provided. Note that

the default configuration requires the backend endpoint be set by the user.

certificates to permit access to the kube-apiserver metrics endpoints. If a user-defined configuration file exists

at /etc/microshift/opentelemetry-collector.yaml, this configuration is used. Otherwise, a default configuration is provided.

Note that the default configuration requires the backend endpoint be set by the user.

For the backend endpoint, should we be specific on what we expect users to set?
I mean, should we say exporters.otlp section must be edited by users?

Added more specific instructions

ggiguash · 2025-01-23T08:39:33Z

packaging/observability/opentelemetry-collector.yaml

+# EXAMPLE OTLP (Prometheus) ENDPOINT CONFIG
+# The otlp exporter requires an endpoint listening for OTLP connections. To prevent spamming the log with Go
+# stack traces, the exporter is disabled. The endpoint is not known at installation, thus a tire-kicking of the
+# microshift-observability package would result in stack traces spam in logs.


Let's think what we can do so that the logs are not "spammed" when the default configuration is used. It sounds as if we should copy this file with .example suffix so that users would have to explicitly rename the file when they enable the collector service.

In any case, the "style" of this comment should be reworded.

I've tweaked the comment and made it a little more informational

ggiguash · 2025-01-23T08:48:46Z

packaging/observability/microshift-observability.service

@@ -0,0 +1,20 @@
+[Unit]
+Description=MicroShift Observability
+After=microshift.service


Should we use ConditionPathExists here for all the files the service expects to have before it starts?

The opentelemetry-collector performs that check for us each time it starts.

Right, but the point of the condition in systemd is not to attempt starting the service if the path does no exist.

Could this help to avoid unnecessary restarts?

ggiguash · 2025-01-23T08:53:35Z

packaging/observability/microshift-observability.service

+
+# It takes a bit for the certs to be created. This service will reach it's burst limit almost immediately, pretty much
+# guaranteeing that it will reach the restart limit before it can possibly succeed.
+RestartSec=200ms


Do we really need this? We've configured the service to start After microshift, so microshift must report readiness to systemd before the current service startup is attempted. MicroShift only reports readiness after creating all certificates.
What am I missing?

In earlier tests this was necessary to keep the service from crash looping, but that doesn't seem to be an issue in the latest opentelemetry-collector. Will remove

packaging/observability/microshift-observability.service

…ng of observability service Signed-off-by: Jon Cope <[email protected]>

…es "debug" exporter instead, which writes to stderr Signed-off-by: Jon Cope <[email protected]>

Signed-off-by: Jon Cope <[email protected]>

ggiguash · 2025-01-28T16:57:52Z

packaging/observability/opentelemetry-collector.yaml

+    auth_type: tls
+    ca_file: /etc/pki/microshift-opentelemetry-collector-client/client-ca.crt
+    key_file: /etc/pki/microshift-opentelemetry-collector-client/client.key
+    cert_file: /etc/pki/microshift-opentelemetry-collector-client/client.crt


These paths need to be updated too -> /var/lib/microshift/..../

ggiguash · 2025-01-28T17:01:59Z

packaging/rpm/microshift.spec

+certificates to permit access to the kube-apiserver metrics endpoints. If a user defined Opentelemetry-Collector exists
+at /etc/microshift/opentelemetry-collector.yaml, this config is used. Otherwise, a default config is provided. Note that
+the default configuration requires the backend endpoint be set by the user. The otlp export must also be specified as
+ .service.pipelines.$RECIEVER.exporter: "otlp".  The specification for the otlp config is:


Let's not use shortened words because it's a user-facing RPM description.

fixed cert signer to be kubelet, not kube-apiserver Signed-off-by: Jon Cope <[email protected]>

openshift-ci · 2025-01-30T14:38:56Z

@copejon: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/test-rebase	`ad4892d`	link	false	`/test test-rebase`
ci/prow/verify	`ad4892d`	link	true	`/test verify`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

ggiguash · 2025-01-30T15:04:46Z

packaging/rpm/microshift.spec

+Requires: opentelemetry-collector
+
+%description observability
+Deploys the Red Hat build of Opentelemetry-Collector as a systemd service on host. MicroShift provides client


Please, fix the case of Opentelemety -> OpenTelemetry to make it consistent with the summary text.

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 6, 2024

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 6, 2024

ggiguash reviewed Dec 9, 2024

View reviewed changes

openshift-ci bot changed the title ~~No issue generate otel cert~~ NO-ISSUE: OpenTelemetry certificates and service for MicroShift Dec 9, 2024

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Dec 9, 2024

pavolloffay reviewed Dec 12, 2024

View reviewed changes

packaging/observability/opentelemetry-collector.yaml Show resolved Hide resolved

copejon changed the title ~~NO-ISSUE: OpenTelemetry certificates and service for MicroShift~~ TRACING-4752: Add OpenTelemetry-Collector as optional sub-package Dec 12, 2024

copejon force-pushed the no-issue-generate-otel-cert branch from fa4f579 to fede276 Compare December 12, 2024 21:05

copejon marked this pull request as ready for review January 21, 2025 16:45

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 21, 2025

openshift-ci bot requested review from agullon and eslutsky January 21, 2025 16:46

copejon added 8 commits January 21, 2025 13:59

flakey, but less so

fea184d

ensure the otelcol process creates necessary dirs added firewall port handling for otel-col exporter Signed-off-by: Jon Cope <[email protected]>

removed hardcoded IP, firewall-cmds. only relevant to running backend…

85ca9f6

… on the localhost Signed-off-by: Jon Cope <[email protected]>

shorten observability handle throughout

788af28

Signed-off-by: Jon Cope <[email protected]>

remove hack to support user "observability" in favor of root. this is…

410327a

… to allow the collection of pod logs from the host filesystem Signed-off-by: Jon Cope <[email protected]>

fixed buildArch setting for observability package

0f47f9e

Signed-off-by: Jon Cope <[email protected]>

copejon force-pushed the no-issue-generate-otel-cert branch from 2042714 to ccfea22 Compare January 22, 2025 23:20

add telemetry pipeline

1732399

Signed-off-by: Jon Cope <[email protected]>

ggiguash reviewed Jan 23, 2025

View reviewed changes

packaging/observability/microshift-observability.service Outdated Show resolved Hide resolved

copejon added 3 commits January 23, 2025 16:08

tweaking the systemd file to enforce starting, restarting, and stoppi…

3c6ca6c

…ng of observability service Signed-off-by: Jon Cope <[email protected]>

otlp endpoint disabled by default, with comments on how to enable. us…

157fa1f

…es "debug" exporter instead, which writes to stderr Signed-off-by: Jon Cope <[email protected]>

doc tweaking

5d99407

Signed-off-by: Jon Cope <[email protected]>

ggiguash reviewed Jan 28, 2025

View reviewed changes

added restricted user

ad4892d

fixed cert signer to be kubelet, not kube-apiserver Signed-off-by: Jon Cope <[email protected]>

copejon force-pushed the no-issue-generate-otel-cert branch from aee833a to ad4892d Compare January 30, 2025 13:13

ggiguash reviewed Jan 30, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TRACING-4752: Add OpenTelemetry-Collector as optional sub-package #4281

TRACING-4752: Add OpenTelemetry-Collector as optional sub-package #4281

copejon commented Dec 6, 2024

openshift-ci bot commented Dec 6, 2024

openshift-ci bot commented Dec 6, 2024

ggiguash Dec 9, 2024

copejon Jan 22, 2025

ggiguash commented Dec 9, 2024

openshift-ci-robot commented Dec 9, 2024

openshift-ci-robot commented Dec 12, 2024 •

edited by openshift-ci bot

Loading

copejon commented Dec 12, 2024

openshift-ci-robot commented Dec 12, 2024 •

edited by openshift-ci bot

Loading

ggiguash Jan 23, 2025

ggiguash Jan 23, 2025

ggiguash Jan 23, 2025

copejon Jan 23, 2025

ggiguash Jan 23, 2025

ggiguash Jan 23, 2025 •

edited

Loading

copejon Jan 23, 2025

ggiguash Jan 23, 2025

copejon Jan 23, 2025

ggiguash Jan 27, 2025

ggiguash Jan 27, 2025

ggiguash Jan 23, 2025

copejon Jan 23, 2025

ggiguash Jan 28, 2025

ggiguash Jan 28, 2025

openshift-ci bot commented Jan 30, 2025

ggiguash Jan 30, 2025

	Deploys the Red Hat build of Opentelemetry-collector as a systemd service on host. MicroShift provides client
	Deploys the Red Hat build of OpenTelemetry-Collector as a systemd service on host. MicroShift provides client

TRACING-4752: Add OpenTelemetry-Collector as optional sub-package #4281

Are you sure you want to change the base?

TRACING-4752: Add OpenTelemetry-Collector as optional sub-package #4281

Conversation

copejon commented Dec 6, 2024

openshift-ci bot commented Dec 6, 2024

openshift-ci bot commented Dec 6, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ggiguash commented Dec 9, 2024

openshift-ci-robot commented Dec 9, 2024

openshift-ci-robot commented Dec 12, 2024 • edited by openshift-ci bot Loading

copejon commented Dec 12, 2024

openshift-ci-robot commented Dec 12, 2024 • edited by openshift-ci bot Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ggiguash Jan 23, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

openshift-ci bot commented Jan 30, 2025

Choose a reason for hiding this comment

openshift-ci-robot commented Dec 12, 2024 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Dec 12, 2024 •

edited by openshift-ci bot

Loading

ggiguash Jan 23, 2025 •

edited

Loading