Hostname override for poller

[SgtMate]

Is your feature request related to a problem? Please describe.
I'm trying to deploy harvest using K8s with a replicaSet (multiple instances of each poller are deployed to ensure HA).
By default, K8s sets the hostname to be the pods name, which has to be unique.
In my dashboards this causes problems as every time a new pod generates it shows up as a new poller/system.

Describe the solution you'd like
To ensure replication transparency, I would like to be able to override the hostname harvest communicates using an environment variable. The default should be the current behaviour, but when the variable is set, it overrides it with what ever is set.

Describe alternatives you've considered
For me the only alternative would be build my own image that overrides the entrypoint to run a script that forces the hostname change at container startup, similar to the implementation i described. Have not tested that so far but I'll try that.
Additional context
The current K8s implementation guide on the wiki is not realy suitable for most enterprise K8s environments, so i build my own using Helm and a custom containerfile.

[cgrinds]

hi @SgtMate per https://kubernetes.io/docs/concepts/workloads/pods/pod-hostname/ can you use the metadata.name or hostnameOverride to change the pod hostname?

[SgtMate]

@cgrinds i thought so as well at first, but after consulting our K8s experts I found out its not possible.
The first would change the pod name as well which is not possible if you deploy replicaSets, that deploy multiple pods at once.
The other might be a solution in the future, but its only a beta release (and we are still 2 versions behind 1.35) so we cant use that, it might take a longer time till the feature get GA.

[cgrinds]

@SgtMate gotcha, understood on the beta comment. I'm not sure adding a Harvest managed environment variable makes sense when it will be replaced by a better K8s specific feature in a few months. It seems like that K8s feature would be the best way to handle this K8s specific problem.

Let's see how your alternative of overriding the entrypoint to set the hostname at container startup works.

[SgtMate]

I'll keep you updated on the process. I agree that a K8s implementation would be the best solution.

[SgtMate]

@cgrinds one thing that might help is to know how harvest get the hostname from the system. How is this handled in the container imgage, by reading /etc/hostname ?

[cgrinds]

Harvest uses Go's standard lib function os.Hostname() which says // Hostname returns the host name reported by the kernel. My read of Go's standard lib, is that function tries uname and if that doesn't work, reads /proc/sys/kernel/hostname.

[rahulguptajss]

@SgtMate Are you using a VictoriaMetrics/Prometheus-type database for data ingestion? Is the hostname causing problems because the instance label changes when pods restart? If so, you can use relabelConfigs in database scrape config to change instance to something unique for the poller.

[SgtMate]

@rahulguptajss Mostly correct. The label is the issue as this causes wired things with the dashboards to happen when ur scaling up the pods. We are using GrafanaCloud and picking up date with Alloy (not managed by me, im just the storage guy 😄) I talked to the colleges regarding your idea, they said this could also be a solution. Thanks for bringing that up!