CIC Endpoint deletion causes unintended LB member removal across namespaces with similar names

[dhkim07]

Describe the bug
In netscaler-k8s-ingress-controller:2.3.15, when two different namespaces contain services with the same name, deleting an endpoint (e.g., Deployment) in one namespace unintentionally removes LoadBalancer members from another namespace with a similar name.

Root cause (assumed): In configure_cpx_from_endpoints_event, the namespace matching logic seems to use substring matching (namespace in key) instead of strict equality. This causes cross-namespace unbinding when namespace names partially overlap (e.g., tests-cpp-antifraud-admin and tests-cpp-antifraud-admin-v2).

To Reproduce

1. Create two namespaces with similar names:

kubectl create ns tests-cpp-antifraud-admin
kubectl create ns tests-cpp-antifraud-admin-v2

2. In both namespaces, create a Deployment and Service with the same service name:
   tests-cpp-antifraud-admin
3. Confirm both services are working and accessible.
4. Delete the deployment in the first namespace:

kubectl delete deploy tests-cpp-antifraud-admin -n tests-cpp-antifraud-admin

5. Observe that the second namespace service (tests-cpp-antifraud-admin-v2) loses LB members and connections fail with:

curl: (56) Recv failure: Connection reset by peer

• Version of the NetScaler Ingress Controller: 2.3.15
• Version of MPX/VPX/CPX: CPX (Kubernetes environment)
• Environment variables: Default deployment (excluding secrets, no modifications)

Expected behavior

• Endpoint deletion in one namespace should only affect LB members in that namespace.
• Other namespaces with the same service name should remain unaffected.

Logs

Example log snippet when deleting an endpoint in the tests-cpp-antifraud-admin namespace (v1):

2025-09-08 04:29:56,649  - DEBUG - [referencetree.py:lookup_tree_node:37] (MainThread) Searching for tests-cpp-antifraud-admin.Deployment.tests-cpp-antifraud-admin
2025-09-08 04:29:56,650  - DEBUG - [referencetree.py:lookup_tree_node:43] (MainThread) Node not found for tests-cpp-antifraud-admin.Deployment.tests-cpp-antifraud-admin
2025-09-08 04:29:56,650  - DEBUG - [referencetree.py:lookup_regexp_references:151] (MainThread) Lookup for regexp reference markers tests-cpp-antifraud-admin.Deployment.tests-cpp-antifraud-admin
2025-09-08 04:29:56,650  - DEBUG - [referencetree.py:lookup_namespace_regexp_references:95] (MainThread) Lookup in namespace regexp reference markers tests-cpp-antifraud-admin.Deployment.tests-cpp-antifraud-admin
2025-09-08 04:29:56,669  - DEBUG - [kubernetes.py:adjust_service_group_for_single_app:4762] (MainThread) Adjusting application BQGDPRqD-tests-cpp-antifraud-admin_80_tests-cpp-antifraud-admin-v2_svc because of lb service BQGDPRqD-tests-cpp-antifraud-admin_8080_tests-cpp-antifraud-admin-v2
2025-09-08 04:29:56,669  - DEBUG - [nitrointerface.py:adjust_service_group:5351] (MainThread) Processing BQGDPRqD-tests-cpp-antifraud-admin_8080_tests-cpp-antifraud-admin-v2 LB APP for BQGDPRqD-tests-cpp-antifraud-admin_80_tests-cpp-antifraud-admin-v2_svc CS App
2025-09-08 04:29:56,670  - DEBUG - [globalfunc.py:get_entity_name:109] (MainThread) Entity name generated with BQGDPRqD-tests-cpp-antifraud-admin_80 + _SGP_ + BQGDPRqD-tests-cpp-antifraud-admin_80_tests-cpp-antifraud-admin-v2 is bqgdprqd-tests-cpp-antifraud-admin_80_sgp_dluqabbcyrx3z4tlybcsqvaouolcgcfb with length79
2025-09-08 04:29:56,670  - DEBUG - [nitrointerface.py:_create_lbvserver_name:1381] (MainThread) Generated Entity name bqgdprqd-tests-cpp-antifraud-admin_80_sgp_dluqabbcyrx3z4tlybcsqvaouolcgcfb with str BQGDPRqD-tests-cpp-antifraud-admin_80_tests-cpp-antifraud-admin-v2_svc hash BQGDPRqD-tests-cpp-antifraud-admin_80_tests-cpp-antifraud-admin-v2_svc postfix SGP
2025-09-08 04:29:56,670  - DEBUG - [nitrointerface.py:adjust_service_group:5353] (MainThread) Adjust svcgrp members for bqgdprqd-tests-cpp-antifraud-admin_80_sgp_dluqabbcyrx3z4tlybcsqvaouolcgcfb
2025-09-08 04:29:56,670  - DEBUG - [nitrointerface.py:_configure_services:2020] (MainThread) configuring service using traditional API (servicegroup:bqgdprqd-tests-cpp-antifraud-admin_80_sgp_dluqabbcyrx3z4tlybcsqvaouolcgcfb)
2025-09-08 04:29:56,670  - DEBUG - [nitrointerface.py:_configure_services_nondesired:2157] (MainThread) Started  services configuration to servicegroup: bqgdprqd-tests-cpp-antifraud-admin_80_sgp_dluqabbcyrx3z4tlybcsqvaouolcgcfb
2025-09-08 04:29:56,699  - DEBUG - [nitrointerface.py:_configure_services_nondesired:2177] (MainThread) Unbinding xxx.xxx.xxx.xxx:32638 from servicegroup bqgdprqd-tests-cpp-antifraud-admin_80_sgp_dluqabbcyrx3z4tlybcsqvaouolcgcfb
2025-09-08 04:29:56,824  - INFO - [nitrointerface.py:_configure_services_nondesired:2180] (MainThread) Unbinding xxx.xxx.xxx.xxx:32638 from servicegroup bqgdprqd-tests-cpp-antifraud-admin_80_sgp_dluqabbcyrx3z4tlybcsqvaouolcgcfb is successful
2025-09-08 04:29:56,824  - DEBUG - [nitrointerface.py:_configure_services_nondesired:2207] (MainThread) Finished services configuration to servicegroup: bqgdprqd-tests-cpp-antifraud-admin_80_sgp_dluqabbcyrx3z4tlybcsqvaouolcgcfb

Additional context

• Workaround: Avoid using similar namespace and service naming patterns.
• If the issue occurs, manually delete the impacted service and restart the Citrix Ingress Controller pod to resync configuration.
• Suggested fix: Change namespace comparison logic from substring match (in) to exact equality (==).
• Root cause above is based on internal analysis and should be confirmed at code level.

[subashd]

hi @dhkim07 ,
Thank you for raising the issue, could you please share the full log with subash.dangol@cloud.com?

[dhkim07]

hi @dhkim07 , Thank you for raising the issue, could you please share the full log with subash.dangol@cloud.com?

Hi @subashd

I’ve sent the requested log file to subash.dangol@cloud.com via email.
Please let me know if you need any additional data or further reproduction details.