Skip to content

Reload behavior of haproxy in AWS EC2 Service Discovery #392

@islue

Description

@islue

HAProxy version 3.2.5-ba0fd6b
HAProxy Data Plane API v3.2.3

I'm trying to use haproxy with dataplaneapi to integrate with aws ec2 auto scaling group.
I noticed that every time auto scaling group changes, dataplaneapi will apply the changes to backend's servers, write them into haproxy.cfg and reload haproxy.

https://github.com/haproxytech/dataplaneapi/blob/master/discovery/aws_service_discovery_instance.go#L239
-> https://github.com/haproxytech/dataplaneapi/blob/master/discovery/service_discovery_instance.go#L90
-> https://github.com/haproxytech/client-native/blob/master/configuration/service.go#L131
After adding debug loggging to trace code, I think the core of logic to decide whether to reload is the code above.

functions like markRemovedNodes, reorderNodes introduce nodes with modified=true, which would cause reload afterwards.
It looks like, scale out won't trigger a reload but scale in will.
We heavily use spot instances in auto scaling group and both scale out and in happen very often.

My questions:

  1. Do the behaviors about reload above are by design?
  2. I supposed that reload wouldn't happen when the instance count was less than server_slots_base. Is that right or am I missing anything basically?

cat /etc/dataplaneapi/dataplaneapi.yml

config_version: 2
name: ip-10-1-0-152.ap-northeast-1.compute.internal
dataplaneapi:
  host: 0.0.0.0
  port: 5555
  advertised:
    api_address: ""
    api_port: 0
  scheme:
  - http
  userlist:
    userlist: dataplaneapi
    userlist_file: ""
  transaction:
    transaction_dir: /var/lib/dataplaneapi/transactions
    backups_number: 10
    backups_dir: /var/lib/dataplaneapi/backups
  resources:
    maps_dir: /etc/haproxy/maps
    ssl_certs_dir: /etc/haproxy/ssl
    general_storage_dir: /etc/haproxy/general
    spoe_dir: /etc/haproxy/spoe
haproxy:
  config_file: /etc/haproxy/haproxy.cfg
  haproxy_bin: /usr/sbin/haproxy
  master_runtime: /var/run/haproxy.sock
  master_worker_mode: true
  reload:
    reload_delay: 5
    service_name: haproxy
    reload_strategy: systemd
log_targets:
- log_to: file
  log_file: /var/log/dataplaneapi.log
  log_level: debug
  log_types:
  - access
  - app

cat /etc/haproxy/dataplane/service_discovery/aws.json

{
  "aws_regions": [
    {
      "allowlist": [
        {
          "key": "tag:HAProxy:Service:Name",
          "value": "MyApp"
        },
        {
          "key": "tag:HAProxy:Service:Port",
          "value": "80"
        }
      ],
      "description": "MyApp",
      "enabled": true,
      "id": "8328ca23-c6e4-4e9e-815f-1489baf2baf3",
      "ipv4_address": "private",
      "name": "myapp-service-discovery",
      "region": "ap-northeast-1",
      "retry_timeout": 10,
      "server_slots_base": 500,
      "server_slots_growth_increment": 100,
      "server_slots_growth_type": "linear"
    }
  ]
}

head -n 30 /var/log/dataplaneapi.log

time="2025-10-06T08:36:32Z" level=info msg="HAProxy Data Plane API v3.2.3 daff5f5e"
time="2025-10-06T08:36:32Z" level=info msg="Build from: https://github.com/haproxytech/dataplaneapi"
time="2025-10-06T08:36:32Z" level=info msg="Reload strategy: systemd"
time="2025-10-06T08:36:32Z" level=info msg="Build date: 2025-08-07T14:07:34Z"
time="2025-10-06T08:36:32Z" level=debug msg="discovery job starting" ID=8328ca23-c6e4-4e9e-815f-1489baf2baf3 ServiceDiscovery=AWS
time="2025-10-06T08:36:32Z" level=debug msg="-- command socket updating client_native client"
time="2025-10-06T08:36:33Z" level=info msg="Serving data plane at http://[::]:5555"
time="2025-10-06T08:36:33Z" level=debug msg="-- command socket Server. conf ready..."
time="2025-10-06T08:36:33Z" level=debug msg="-- command socket not set (--debug_socket_path). Not running the command socket server"
time="2025-10-06T08:36:33Z" level=debug msg="Mode: single"
time="2025-10-06T08:36:33Z" level=debug msg="-- command socket handleClientNativeClientUpdate stopped"
time="2025-10-06T08:36:42Z" level=debug msg="discovery job update triggered" ID=8328ca23-c6e4-4e9e-815f-1489baf2baf3 ServiceDiscovery=AWS
time="2025-10-06T08:36:45Z" level=debug msg="discovery job reconciliation completed" ID=8328ca23-c6e4-4e9e-815f-1489baf2baf3 ServiceDiscovery=AWS
time="2025-10-06T08:36:45Z" level=debug msg="Scheduling a new reload..." reload_id=2025-10-06-0
time="2025-10-06T08:36:47Z" level=debug msg="Reload started" reload_id=2025-10-06-0
time="2025-10-06T08:36:48Z" level=debug msg="Reload finished in 253.780028ms" reload_id=2025-10-06-0
time="2025-10-06T08:36:48Z" level=debug msg="Reload successful" reload_id=2025-10-06-0
time="2025-10-06T08:36:48Z" level=debug msg="Handling reload completed, waiting for new requests" reload_id=2025-10-06-0
time="2025-10-06T08:36:55Z" level=debug msg="discovery job update triggered" ID=8328ca23-c6e4-4e9e-815f-1489baf2baf3 ServiceDiscovery=AWS
time="2025-10-06T08:36:57Z" level=debug msg="discovery job reconciliation completed" ID=8328ca23-c6e4-4e9e-815f-1489baf2baf3 ServiceDiscovery=AWS
time="2025-10-06T08:36:57Z" level=debug msg="Scheduling a new reload..." reload_id=2025-10-06-1
time="2025-10-06T08:36:57Z" level=debug msg="Reload started" reload_id=2025-10-06-1
time="2025-10-06T08:36:57Z" level=debug msg="Reload finished in 102.89662ms" reload_id=2025-10-06-1
time="2025-10-06T08:36:57Z" level=debug msg="Reload successful" reload_id=2025-10-06-1
time="2025-10-06T08:36:57Z" level=debug msg="Handling reload completed, waiting for new requests" reload_id=2025-10-06-1
time="2025-10-06T08:37:07Z" level=debug msg="discovery job update triggered" ID=8328ca23-c6e4-4e9e-815f-1489baf2baf3 ServiceDiscovery=AWS
time="2025-10-06T08:37:08Z" level=debug msg="discovery job reconciliation completed" ID=8328ca23-c6e4-4e9e-815f-1489baf2baf3 ServiceDiscovery=AWS
time="2025-10-06T08:37:18Z" level=debug msg="discovery job update triggered" ID=8328ca23-c6e4-4e9e-815f-1489baf2baf3 ServiceDiscovery=AWS
time="2025-10-06T08:37:19Z" level=debug msg="discovery job reconciliation completed" ID=8328ca23-c6e4-4e9e-815f-1489baf2baf3 ServiceDiscovery=AWS
time="2025-10-06T08:37:29Z" level=debug msg="discovery job update triggered" ID=8328ca23-c6e4-4e9e-815f-1489baf2baf3 ServiceDiscovery=AWS

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions