Fix Nethermind Mainnet Metrics Not Appearing in Prometheus

[jacobourar]

Nethermind Metrics Not Appearing in Prometheus

Describe the Bug

Nethermind metrics fail to appear in Prometheus due to incorrect target configuration or DNS alias resolution. The issue arises when Prometheus uses a nethermind.dappnode alias that does not correctly resolve to the Nethermind container.

---

To Reproduce

1. Install DAppNode Core v0.2.101.
2. Run Nethermind as the Execution Layer client.
3. Open Prometheus using DMS within Dappnode's interface and check the Targets page at:

   http://<prometheus-ip>:9090/targets
   

4. Nethermind appears as DOWN with the target nethermind.dappnode:6060.
5. Confirm the issue by entering the Prometheus container:

   docker exec -it <prometheus_container_name> sh

6. Test DNS resolution for Nethermind aliases:

   ping nethermind.dappnode

   Result: DNS resolution fails or outputs permission denied.

---

Expected Behavior

Nethermind metrics should appear as UP in Prometheus, and dashboards should display relevant data.

---

Screenshots

Prometheus Target Status:

• DOWN status for nethermind.dappnode:6060.

---

DAppNode Version

• DAppNode Core version: v0.2.101
• Package version: Nethermind package v1.0.57
• OS: DappNode
• Browser: Brave/Firefox/Chrome (if Grafana visualization applies)

---

Additional Context

Root Cause

• Prometheus relies on file-based service discovery (file_sd) to define targets.
• The default alias (nethermind.dappnode) used in the target configuration does not match the actual DNS name of the Nethermind container.
• Valid aliases can vary and need verification.

---

Steps to Resolve

1. Verify the Correct Alias

Enter the Prometheus container and test possible aliases:

docker exec -it <prometheus_container_name> sh
nslookup nethermind
nslookup nethermind.public.dappnode
nslookup execution.mainnet.dncore.dappnode

---

2. Update the Target File

Locate the Prometheus file-based configuration file:

ls -l /prometheus_file_sd/

Edit the target file:

vi /prometheus_file_sd/<nethermind_target_file>.json

Replace the incorrect target (nethermind.dappnode:6060) with a valid alias:

[
  {
    "labels": {
      "package": "nethermind.dnp.dappnode.eth",
      "service": "nethermind"
    },
    "targets": [
      "nethermind:6060"
    ]
  }
]

---

3. Reload Prometheus

Send the SIGHUP signal to reload the configuration:

docker kill -s HUP <prometheus_container_name>

---

4. Verify the Fix

Check the Prometheus Targets page:

http://<prometheus-ip>:9090/targets

• Ensure Nethermind now appears as UP.
• Refresh Grafana dashboards to confirm metrics are visible.

---

Summary

• Problem: Prometheus fails to resolve the default nethermind.dappnode alias.
• Solution: Verify valid aliases, update the file_sd configuration file, and reload Prometheus.

---

Best Practices

• Use consistent and simple aliases like nethermind:6060.
• Verify DNS resolution within the Prometheus container.
• Ensure Prometheus uses dynamic service discovery (file_sd) for flexibility.