NETOBSERV-1248: shared k8s cache #681
Conversation
|
Skipping CI for Draft Pull Request. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
jotak
force-pushed
the
poc-k8s-cache
branch
2 times, most recently
from
0f501fb to
0f0b00d
Compare
|
@jotak: This pull request references NETOBSERV-1248 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the spike to target the "4.17.0" version, but no target version was set. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
@jotak: This pull request references NETOBSERV-1248 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the spike to target the "4.17.0" version, but no target version was set. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
The goal is to allow deploying a dedicated FLP instance as just a k8s cache component, alongside with the "traditional" FLP deployments. The k8s cache is a Kafka producer, and traditional instances are consumers. - New FLP binary, k8s-cache, that just starts the informers and writes update events to Kafka - New optional config for kubernetes enrichment, allowing to read from kafka instead of using informers
|
@jotak: This pull request references NETOBSERV-1248 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the spike to target the "4.17.0" version, but no target version was set. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
Related: netobserv#681 This is a preliminary work preparing the ground for NETOBSERV-1248. No functional change here. The goal is to minimize rebase hassle in netobserv#681
Related: netobserv#681 This is a preliminary work preparing the ground for NETOBSERV-1248. No functional change here. The goal is to minimize rebase hassle in netobserv#681 Details: - Kafka read/write logic extracted from ingest/encode pipelines and put into a dedicated kafka package - New k8s "datasource" struct for k8s enrichment - currently only has the usual informers datasource, but later will include a kafka-based datasource as well - Config related to k8s datasource moved into its own package. It includes the SecondaryNetwork config. - Some minor variables/functions renaming
| define build_target | ||
| echo 'building image for arch $(1)'; \ | ||
| DOCKER_BUILDKIT=1 $(OCI_BIN) buildx build --load --build-arg TARGETPLATFORM=linux/$(1) --build-arg TARGETARCH=$(1) --build-arg BUILDPLATFORM=linux/amd64 ${OCI_BUILD_OPTS} -t ${IMAGE}-$(1) -f contrib/docker/Dockerfile .; | ||
| DOCKER_BUILDKIT=1 $(OCI_BIN) buildx build --load --build-arg TARGETPLATFORM=linux/$(1) --build-arg TARGETARCH=$(1) --build-arg BUILDPLATFORM=linux/amd64 ${OCI_BUILD_OPTS} -t ${IMAGE_CACHE}-$(1) -f contrib/docker/cache.Dockerfile .; | ||
| endef | ||
|
|
||
| # push a single arch target image | ||
| define push_target | ||
| echo 'pushing image ${IMAGE}-$(1)'; \ | ||
| DOCKER_BUILDKIT=1 $(OCI_BIN) push ${IMAGE}-$(1); | ||
| DOCKER_BUILDKIT=1 $(OCI_BIN) push ${IMAGE_CACHE}-$(1); | ||
| endef | ||
|
|
||
| # manifest create a single arch target provided as argument | ||
| define manifest_add_target | ||
| echo 'manifest add target $(1)'; \ | ||
| DOCKER_BUILDKIT=1 $(OCI_BIN) manifest add ${IMAGE} ${IMAGE}-$(1); | ||
| DOCKER_BUILDKIT=1 $(OCI_BIN) manifest add ${IMAGE_CACHE} ${IMAGE_CACHE}-$(1); | ||
| endef |
There was a problem hiding this comment.
Would it be better to have dedicated functions for cache case ?
It would avoid building both when you need only one 🤔
There was a problem hiding this comment.
actually I rewrote that differently in #848
I don't build a new image anymore, instead I include both binaries in the same image. It avoids having to refactor all the build process. It also makes it a little simpler in the operator.
|
PR needs rebase. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
Related: #681 This is a preliminary work preparing the ground for NETOBSERV-1248. No functional change here. The goal is to minimize rebase hassle in #681 Details: - Kafka read/write logic extracted from ingest/encode pipelines and put into a dedicated kafka package - New k8s "datasource" struct for k8s enrichment - currently only has the usual informers datasource, but later will include a kafka-based datasource as well - Config related to k8s datasource moved into its own package. It includes the SecondaryNetwork config. - Some minor variables/functions renaming
|
Superseded by #848 |
The goal is to allow deploying a dedicated FLP instance as just a k8s cache component, alongside with the "traditional" FLP deployments.
The k8s cache is a Kafka producer, and traditional instances are
consumers.
Kafka
kafka instead of using informers
cf #641
Operator PR: netobserv/network-observability-operator#684