BREAKING CHANGE 3.19.0: a33e897b0e7b8a1bd71bfe68eb6e10e749a0c6d7

[volker-raschek]

The patch a33e897 introduced probes. The default scheme for all probes is now HTTP. This leads in environments, where the TLS features, introduced with ff99541 to issues, because the application is listing on HTTPS, but the probe use HTTP to establish a connection.

Furthermore, the scheme can not be configured via the values.yaml. The only way to avoid this issue is currently to rollback environments to v3.18.1.

[npdgm]

Thanks a lot. Obviously we don't test for TLS, I'll see that we improve CI testing and have this scenario implemented.
Working on a fix...

[volker-raschek]

Hi,
maybe with https://github.com/helm-unittest/helm-unittest

I use this plugin for my own helm charts: https://git.cryptic.systems/volker.raschek/prometheus-fail2ban-exporter-charts

Volker

[npdgm]

Hi, maybe with https://github.com/helm-unittest/helm-unittest
I use this plugin for my own helm charts: https://git.cryptic.systems/volker.raschek/prometheus-fail2ban-exporter-charts

Neat tool, thanks! That's an option. Ultimately I would like to also do end-to-end tests with a live Talos or KinD cluster and a Prometheus stack.

[monkeynator]

🎉 This issue has been resolved in version 3.19.1 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

[npdgm]

A new release is out with a patch to allow configuration of httpGet parameters in healthcheck probes, such as scheme.
We could probably streamline the configuration of TLS that now requires about 6 values to be set but that will do for now.

Please try version 3.19.1 and override scheme in every section:

prometheusServiceMonitor:
  scheme: https
  tlsConfig:
    [....]

secretsExporter:
  livenessProbe:
    httpGet:
      scheme: HTTPS
  readinessProbe:
    httpGet:
      scheme: HTTPS

hostPathsExporter:
  livenessProbe:
    httpGet:
      scheme: HTTPS
  readinessProbe:
    httpGet:
      scheme: HTTPS
  daemonSets:
    [....]

[volker-raschek]

Hi @npdgm,
your patch does not solve the issue.

time=2025-06-22T15:46:04.291Z level=INFO msg="http: TLS handshake error from 192.168.179.225:37738: tls: client didn't provide a certificate"
time=2025-06-22T15:46:04.429Z level=INFO msg="http: TLS handshake error from 192.168.179.225:37746: tls: client didn't provide a certificate"
time=2025-06-22T15:46:07.292Z level=INFO msg="http: TLS handshake error from 192.168.179.225:37756: tls: client didn't provide a certificate"

The probes use the /metrics endpoint to check for a sign of life or readiness. The probe (http get) is executed by the kubelet of the node on which the pod is located.

The kubelet is unable to call the /metrics endpoint and receives a TLS error because the endpoint requires the client to be able to authenticate. It simply cannot. For this reason, it is not enough for the probe to support HTTPS as a schema.

According to the documentation, the httpGet probe does not support authentication either. At least I have not found any possibility apart from optional HTTP headers.

I don't know if it wouldn't be better to just introduce a dedicated endpoint for the likeness probe (/live) and readiness probe (/ready) where no TLS authentication is required.