Hostname log field overwritten or even cleared when k8s attribute is present

[oldium]

I came to this while processing OTEL log messages. When host.name OTEL attribute is present in the resource attributes, it is converted to Host.Hostname field:

apm-data/input/otlp/metadata.go

Lines 173 to 177 in 46f8e38

	case semconv.AttributeHostName:
	if out.Host == nil {
	out.Host = &modelpb.Host{}
	}
	out.Host.Hostname = truncate(v.Str())

That is fine. But when there is any k8s attribute present, it is either rewritten, or even cleared in post-processing phase:

apm-data/model/modelprocessor/hostname.go

Lines 41 to 53 in 46f8e38

	case event.GetKubernetes().GetNodeName() != "":
	if event.Host == nil {
	event.Host = &modelpb.Host{}
	}
	// host.kubernetes.node.name is set: set host.hostname to its value.
	event.Host.Hostname = event.Kubernetes.NodeName
	case event.GetKubernetes().GetPodName() != "" || event.GetKubernetes().GetPodUid() != "" || event.GetKubernetes().GetNamespace() != "":
	if event.Host != nil {
	// kubernetes.* is set, but kubernetes.node.name is not: don't set host.hostname at all.
	event.Host.Hostname = ""
	}
	default:
	// Otherwise use the originally specified host.hostname value.

The code is there for years (previously in apm-server repo), so I believe this was done to fill some information APM agent did not have.

I discovered this by setting k8s.pod.name from host.name attribute in OTEL Collector to fill some missing information manually and suddenly the smart resource column in Kibana Logs stopped showing any host name. This looks like a bug to me.

I also discovered that the value of OTEL attribute host.name has no influence on Kibana Logs resource column display and it always shows k8s.node.name when present. For us more valuable is to actually show k8s.pod.name, so we know the originating instance, not the Kubernetes Node with many PODs. Showing the Node name serving many PODs does not make much sense for us actually.

I think it would make sense to keep the Host.Hostname value as-is in case it is non-empty. So no overwriting, no clearing.