Skip to content

Lack of timeout control in member cluster client config #6829

@Tanxin2333

Description

@Tanxin2333

What happened:
The member cluster client does not have a configured request timeout.
When the creation or request of a member cluster client takes too long, karmada controllers(e.g. execution-controller) has no mechanism to cancel or timeout the request:

clusterObj, err := dynamicClusterClient.DynamicClientSet.Resource(gvr).Namespace(desireObj.GetNamespace()).Create(ctx, desireObj, metav1.CreateOptions{})

As a result, if a member cluster does not respond, the controller’s processing flow is blocked, causing subsequent request events to remain unhandled.
What you expected to happen:
Add a timeout control mechanism for the member cluster client.
When a controller request to a member cluster does not respond within the timeout period, the controller should stop waiting and continue to process other request events.
How to reproduce it (as minimally and precisely as possible):

  1. Prepare two member clusters: member1 and member2.
  2. Inject a 300s response delay into member2.
  3. When deploying workloads:
    Requests to member1 respond immediately.
    Requests to member2 hang for 5 minutes due to the lack of timeout control.

Anything else we need to know?:
The member cluster clientin Karmada is implemented as a DynamicClient. DynamicClient does not set a default timeout internally.
We can refer to the ​Kubernetes DiscoveryClient​, which uses a default timeout of 32s.
It is recommended to define a default timeout constant in the membercluster_client file and then pass this timeout value to clusterConfig inside the BuildClusterConfig function.

const (
    defaultTimeout = 32 * time.Second
)
clusterConfig := &rest.Config{
    BearerToken: string(token),
    Host:        apiEndpoint,
    Timeout:     defaultTimeout,
}

Environment:

  • Karmada version: alpha
  • kubectl-karmada or karmadactl version (the result of kubectl-karmada version or karmadactl version):
  • Others:

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.

    Type

    No type

    Projects

    Status

    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions