[pod logs] Use the K8s `deployment/cronjob/job` name in the `service_name` logic

[cyrille-leclerc]

We want to align the service_name logic of the K8s Monitoring Helm Chart Pod Logs feature on OpenTelemetry resource attributes specs on Kubernetes.

For this, we need to retrieve the deployment/cronjob/job of the pods through the "controller-by" attribute.
If the OpenTelemetry Collector K8s Processor is capable of retrieving these metadata, it's not clear if Alloy's discovery.relabel can also do it.

• https://github.com/grafana/k8s-monitoring-helm/blob/v2.0.28/charts/k8s-monitoring/charts/feature-pod-logs/templates/_common_pod_discovery.alloy.tpl
• The source code of the logic to retrieve pod metadata seems to be https://github.com/prometheus/prometheus/blob/main/discovery/kubernetes/pod.go

OpenTelemetry logic for service.name

Choose the first value found:

• pod.annotation[resource.opentelemetry.io/service.name]
• pod.label[app.kubernetes.io/instance] (well-known label app.kubernetes.io/instance)
• pod.label[app.kubernetes.io/name] (well-known label app.kubernetes.io/name)
• k8s.deployment.name
• k8s.replicaset.name
• k8s.statefulset.name
• k8s.daemonset.name
• k8s.cronjob.name
• k8s.job.name
• k8s.pod.name
• k8s.container.name

See also

• Revert logs service_name preference of container.name vs k8s.pod.name #1533

[cyrille-leclerc]

There may be a hack looking at __meta_kubernetes_pod_controller_kind (e.g. ReplicaSet) and __meta_kubernetes_pod_controller_name (e.g. orders-prometheus-postgres-exporter-6f65896bd5) and extracting the substringBeforeLast(__meta_kubernetes_pod_controller_name, '-').

{
	__address__                                              = "10.0.12.114:9187",
	__meta_kubernetes_namespace                              = "ditl-demo-prod",
	__meta_kubernetes_pod_annotation_checksum_config         = "01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b",
	__meta_kubernetes_pod_annotation_checksum_secrets        = "4d8241978bef61f991832cebf24c44d8aad147d9a9629663a01a24c05ec414d7",
	__meta_kubernetes_pod_annotationpresent_checksum_config  = "true",
	__meta_kubernetes_pod_annotationpresent_checksum_secrets = "true",
	__meta_kubernetes_pod_container_id                       = "containerd://bbb03a33212e7ce2b5a649c5a460df4e6ad6a460c959f7d1da811eee226b08b7",
	__meta_kubernetes_pod_container_image                    = "quay.io/prometheuscommunity/postgres-exporter:v0.17.1",
	__meta_kubernetes_pod_container_init                     = "false",
	__meta_kubernetes_pod_container_name                     = "prometheus-postgres-exporter",
	__meta_kubernetes_pod_container_port_name                = "http",
	__meta_kubernetes_pod_container_port_number              = "9187",
	__meta_kubernetes_pod_container_port_protocol            = "TCP",
	__meta_kubernetes_pod_controller_kind                    = "ReplicaSet",
	__meta_kubernetes_pod_controller_name                    = "orders-prometheus-postgres-exporter-6f65896bd5",
	__meta_kubernetes_pod_host_ip                            = "10.0.12.211",
	__meta_kubernetes_pod_ip                                 = "10.0.12.114",
	__meta_kubernetes_pod_label_app                          = "prometheus-postgres-exporter",
	__meta_kubernetes_pod_label_pod_template_hash            = "6f65896bd5",
	__meta_kubernetes_pod_label_release                      = "orders",
	__meta_kubernetes_pod_labelpresent_app                   = "true",
	__meta_kubernetes_pod_labelpresent_pod_template_hash     = "true",
	__meta_kubernetes_pod_labelpresent_release               = "true",
	__meta_kubernetes_pod_name                               = "orders-prometheus-postgres-exporter-6f65896bd5-rwfct",
	__meta_kubernetes_pod_node_name                          = "ip-10-0-12-211.us-west-2.compute.internal",
	__meta_kubernetes_pod_phase                              = "Running",
	__meta_kubernetes_pod_ready                              = "true",
	__meta_kubernetes_pod_uid                                = "d52e8acb-971c-48f6-badf-726e9000bea4",
}, {
	__address__                                              = "10.0.12.176:9187",
	__meta_kubernetes_namespace                              = "ditl-demo-prod",
	__meta_kubernetes_pod_annotation_checksum_config         = "01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b",
	__meta_kubernetes_pod_annotation_checksum_secrets        = "8daa3ce1a12d2d9ba4af63b10f7cebe378bcd68908bdb369e0726c2d1ef5d5f1",
	__meta_kubernetes_pod_annotationpresent_checksum_config  = "true",
	__meta_kubernetes_pod_annotationpresent_checksum_secrets = "true",
	__meta_kubernetes_pod_container_id                       = "containerd://f0bf7a8288f5c7bbf3e5b77334b1153e32d59ef5fd1586bb8c78ff4febb7955b",
	__meta_kubernetes_pod_container_image                    = "quay.io/prometheuscommunity/postgres-exporter:v0.17.1",
	__meta_kubernetes_pod_container_init                     = "false",
	__meta_kubernetes_pod_container_name                     = "prometheus-postgres-exporter",
	__meta_kubernetes_pod_container_port_name                = "http",
	__meta_kubernetes_pod_container_port_number              = "9187",
	__meta_kubernetes_pod_container_port_protocol            = "TCP",
	__meta_kubernetes_pod_controller_kind                    = "ReplicaSet",
	__meta_kubernetes_pod_controller_name                    = "productcatalog-prometheus-postgres-exporter-6768c4cd7c",
	__meta_kubernetes_pod_host_ip                            = "10.0.12.211",
	__meta_kubernetes_pod_ip                                 = "10.0.12.176",
	__meta_kubernetes_pod_label_app                          = "prometheus-postgres-exporter",
	__meta_kubernetes_pod_label_pod_template_hash            = "6768c4cd7c",
	__meta_kubernetes_pod_label_release                      = "productcatalog",
	__meta_kubernetes_pod_labelpresent_app                   = "true",
	__meta_kubernetes_pod_labelpresent_pod_template_hash     = "true",
	__meta_kubernetes_pod_labelpresent_release               = "true",
	__meta_kubernetes_pod_name                               = "productcatalog-prometheus-postgres-exporter-6768c4cd7c-wjmxl",
	__meta_kubernetes_pod_node_name                          = "ip-10-0-12-211.us-west-2.compute.internal",
	__meta_kubernetes_pod_phase                              = "Running",
	__meta_kubernetes_pod_ready                              = "true",
	__meta_kubernetes_pod_uid                                = "7ffa9251-eb9f-4e04-848a-962c038f3999",
}, {
	__address__                                                    = "10.0.12.224:53",
	__meta_kubernetes_namespace                                    = "kube-system",
	__meta_kubernetes_pod_container_id                             = "containerd://34126b768b988df8289ff09e4849bd22da373d04eeaa7b7f8b28d4163e7136ae",
	__meta_kubernetes_pod_container_image                          = "602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/coredns:v1.11.4-eksbuild.10",
	__meta_kubernetes_pod_container_init                           = "false",
	__meta_kubernetes_pod_container_name                           = "coredns",
	__meta_kubernetes_pod_container_port_name                      = "dns",
	__meta_kubernetes_pod_container_port_number                    = "53",
	__meta_kubernetes_pod_container_port_protocol                  = "UDP",
	__meta_kubernetes_pod_controller_kind                          = "ReplicaSet",
	__meta_kubernetes_pod_controller_name                          = "coredns-5c8b6956d4",
	__meta_kubernetes_pod_host_ip                                  = "10.0.12.211",
	__meta_kubernetes_pod_ip                                       = "10.0.12.224",
	__meta_kubernetes_pod_label_eks_amazonaws_com_component        = "coredns",
	__meta_kubernetes_pod_label_k8s_app                            = "kube-dns",
	__meta_kubernetes_pod_label_pod_template_hash                  = "5c8b6956d4",
	__meta_kubernetes_pod_labelpresent_eks_amazonaws_com_component = "true",
	__meta_kubernetes_pod_labelpresent_k8s_app                     = "true",
	__meta_kubernetes_pod_labelpresent_pod_template_hash           = "true",
	__meta_kubernetes_pod_name                                     = "coredns-5c8b6956d4-5vvhv",
	__meta_kubernetes_pod_node_name                                = "ip-10-0-12-211.us-west-2.compute.internal",
	__meta_kubernetes_pod_phase                                    = "Running",
	__meta_kubernetes_pod_ready                                    = "true",
	__meta_kubernetes_pod_uid                                      = "597b98bb-bb78-4def-8d4b-8cc776e3abe7",
}, {
	__address__                                                    = "10.0.12.224:53",
	__meta_kubernetes_namespace                                    = "kube-system",
	__meta_kubernetes_pod_container_id                             = "containerd://34126b768b988df8289ff09e4849bd22da373d04eeaa7b7f8b28d4163e7136ae",
	__meta_kubernetes_pod_container_image                          = "602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/coredns:v1.11.4-eksbuild.10",
	__meta_kubernetes_pod_container_init                           = "false",
	__meta_kubernetes_pod_container_name                           = "coredns",
	__meta_kubernetes_pod_container_port_name                      = "dns-tcp",
	__meta_kubernetes_pod_container_port_number                    = "53",
	__meta_kubernetes_pod_container_port_protocol                  = "TCP",
	__meta_kubernetes_pod_controller_kind                          = "ReplicaSet",
	__meta_kubernetes_pod_controller_name                          = "coredns-5c8b6956d4",
	__meta_kubernetes_pod_host_ip                                  = "10.0.12.211",
	__meta_kubernetes_pod_ip                                       = "10.0.12.224",
	__meta_kubernetes_pod_label_eks_amazonaws_com_component        = "coredns",
	__meta_kubernetes_pod_label_k8s_app                            = "kube-dns",
	__meta_kubernetes_pod_label_pod_template_hash                  = "5c8b6956d4",
	__meta_kubernetes_pod_labelpresent_eks_amazonaws_com_component = "true",
	__meta_kubernetes_pod_labelpresent_k8s_app                     = "true",
	__meta_kubernetes_pod_labelpresent_pod_template_hash           = "true",
	__meta_kubernetes_pod_name                                     = "coredns-5c8b6956d4-5vvhv",
	__meta_kubernetes_pod_node_name                                = "ip-10-0-12-211.us-west-2.compute.internal",
	__meta_kubernetes_pod_phase                                    = "Running",
	__meta_kubernetes_pod_ready                                    = "true",
	__meta_kubernetes_pod_uid                                      = "597b98bb-bb78-4def-8d4b-8cc776e3abe7",
}

[petewall]

We could do this if controller is DaemonSet or StatefulSet. If the controller is "ReplicaSet" or "Job", we can't really make the assumption that it's from a RS or Deployment, or from a Job or CronJob. It's too ambiguous...

Perhaps a better, long term idea would be to use something like this:
https://grafana.com/docs/alloy/latest/reference/components/loki/loki.enrich

[MrLuje]

@cyrille-leclerc @petewall

We have a specific use-case where we'd like to customize this logic (labels order or even custom ones)
Would it be possible to override the source_labels from the .Values ?

  rule {
    action = "replace"
    source_labels = [
      "__meta_kubernetes_pod_annotation_resource_opentelemetry_io_service_name",
      "__meta_kubernetes_pod_label_app_kubernetes_io_name",
      "__meta_kubernetes_pod_container_name",
    ]
    separator = ";"
    regex = "^(?:;*)?([^;]+).*$"
    replacement = "$1"
    target_label = "service_name"
  }

This may not be the best issue to discuss this, let me know if you want me to create a separate issue

[cyrille-leclerc]

@rlankfo may be able to help on how to change the naming logic.

We have a specific use-case where we'd like to customize this logic (labels order or even custom ones)

To better understand your requirements and ensure alignment with industry standards, could you please elaborate on your specific use case? We're particularly interested in how the Kubernetes annotation resource.opentelemetry.io/service.name and the label app.kubernetes.io/name could address your needs.

We're seeking feedback to ensure our approach aligns with established best practices.