Skip to content

Conversation

@dekaihu
Copy link
Contributor

@dekaihu dekaihu commented Sep 28, 2025

What type of PR is this?

/kind feature

What this PR does / why we need it:

Which issue(s) this PR fixes:

Part of #6589

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

`karmada-controller-manager`: Introduced built-in interpreter for Volcano `Job`.

@karmada-bot
Copy link
Collaborator

Welcome @dekaihu! It looks like this is your first PR to karmada-io/karmada 🎉

@gemini-code-assist
Copy link

Summary of Changes

Hello @dekaihu, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request integrates support for Volcano Jobs (batch.volcano.sh/v1alpha1/Job) by adding a dedicated resource interpreter. This interpreter equips the system with the capability to manage and synchronize Volcano Jobs across multiple clusters, providing essential functions for replica management, health monitoring, resource allocation, and comprehensive status aggregation and reflection. This enhancement significantly broadens the types of batch workloads that can be effectively orchestrated and observed.

Highlights

  • New Volcano Job Interpreter: This pull request introduces a new ResourceInterpreterCustomization specifically for batch.volcano.sh/v1alpha1/Job resources, enabling a multi-cluster management system to understand and manage Volcano Jobs.
  • Comprehensive Interpretation Logic: Lua scripts are implemented for various interpretation functions, including replicaRevision to adjust minAvailable, healthInterpretation to determine job health based on phase, and componentResource to extract resource requirements from job tasks.
  • Advanced Status Management: Detailed logic is provided for statusAggregation to combine status information from multiple clusters and statusReflection to accurately reflect the complex status of Volcano Jobs, including task status counts and conditions.
  • Test Coverage: Accompanying test configurations and sample status data are added to validate the functionality of the new Volcano Job interpreter across different operations like status aggregation, replica interpretation, and health checks.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

@karmada-bot karmada-bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Sep 28, 2025
Copy link

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a resource interpreter for Volcano jobs. The implementation is a good start, but I've found several issues, primarily within the Lua scripts for status aggregation and component resource discovery. There are cases where values are overwritten instead of being aggregated, which would lead to incorrect behavior. I've also identified a critical typo that would cause a runtime error. My review includes specific suggestions to address these points and improve the robustness of the interpreter.

@dekaihu dekaihu force-pushed the interpreter-vcjob branch 3 times, most recently from 11d31f9 to 0876da6 Compare September 29, 2025 02:08
@codecov-commenter
Copy link

codecov-commenter commented Sep 29, 2025

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 45.63%. Comparing base (4594ca0) to head (dee6965).
⚠️ Report is 82 commits behind head on master.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6790      +/-   ##
==========================================
- Coverage   45.73%   45.63%   -0.10%     
==========================================
  Files         689      692       +3     
  Lines       57104    57678     +574     
==========================================
+ Hits        26114    26319     +205     
- Misses      29358    29717     +359     
- Partials     1632     1642      +10     
Flag Coverage Δ
unittests 45.63% <ø> (-0.10%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Member

@RainbowMango RainbowMango left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/assign

@RainbowMango
Copy link
Member

/gemini review

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds support for Volcano Job resource interpretation in Karmada by implementing a ResourceInterpreterCustomization for batch.volcano.sh/v1alpha1/Job resources.

  • Implements comprehensive resource interpretation including replica revision, health checks, component resource analysis, and status aggregation/reflection
  • Adds test infrastructure with test configuration and status test data
  • Provides Lua scripts for handling Volcano Job lifecycle operations in multi-cluster environments

Reviewed Changes

Copilot reviewed 3 out of 5 changed files in this pull request and generated 2 comments.

File Description
customizations.yaml Main ResourceInterpreterCustomization with Lua scripts for Volcano Job interpretation
customizations_tests.yaml Test configuration file (currently commented out) for validating the interpreter
status-file.yaml Test data representing aggregated status information for testing status operations

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

Copy link

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a resource interpreter for VolcanoJob (batch.volcano.sh/v1alpha1.Job). The implementation covers replica revision, health interpretation, component resource discovery, status aggregation, and status reflection using Lua scripts.

My review has identified a few issues:

  • A potential runtime error in the InterpretHealth function.
  • A bug in the durationVal function within statusAggregation that would lead to incorrect duration comparisons.
  • Some redundant code in the statusReflection function.
  • The tests for the interpreter are currently commented out, and required test data files are missing, which prevents verification of the implementation.

I've provided specific comments and suggestions to address these points. Please enable and complete the tests to ensure the interpreter is robust.

@dekaihu dekaihu force-pushed the interpreter-vcjob branch 5 times, most recently from fadcbf7 to 4963cf3 Compare September 29, 2025 11:05
@dekaihu
Copy link
Contributor Author

dekaihu commented Sep 30, 2025

Test Steps:

  1. Enable the MultiplePodTemplatesScheduling feature in Karmada 1.15.
  2. Use Karmada to deploy the VolcanoJob interpreter and VolcanoJob CRD.
  3. Distribute the VolcanoJob on the Karmada cluster.
    Results:
    VolcanoJob template reference:
apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: dk-job
spec:
  maxRetry: 3
  minAvailable: 3
  plugins:
    env: []
    ssh: []
    svc:
    - --disable-network-policy=true
  queue: default
  schedulerName: volcano
  tasks:
  - minAvailable: 1
    name: job-nginx1
    replicas: 1
    template:
      metadata:
        name: nginx1
      spec:
        containers:
        - args:
          - sleep 10
          command:
          - bash
          - -c
          image: nginx:latest
          imagePullPolicy: IfNotPresent
          name: nginx
          resources:
            requests:
              cpu: 100m
        nodeSelector:
          kubernetes.io/os: linux
        restartPolicy: OnFailure
  - minAvailable: 2
    name: job-nginx2
    replicas: 3
    template:
      metadata:
        name: nginx2
      spec:
        containers:
        - args:
          - sleep 30
          command:
          - bash
          - -c
          image: nginx:latest
          imagePullPolicy: IfNotPresent
          name: nginx
          resources:
            requests:
              cpu: 100m
        nodeSelector:
          kubernetes.io/os: linux
        restartPolicy: OnFailure
  1. Single-member cluster scheduling
    View resource requests in the resourcebinding.
  clusters:
  - name: member2
  components:
  - name: job-nginx1
    replicaRequirements:
      nodeClaim:
        nodeSelector:
          kubernetes.io/os: linux
      resourceRequest:
        cpu: 100m
    replicas: 1
  - name: job-nginx2
    replicaRequirements:
      nodeClaim:
        nodeSelector:
          kubernetes.io/os: linux
      resourceRequest:
        cpu: 100m
    replicas: 2

The results on the Karmada control plane are as follows:

status:
  conditions:
  - lastTransitionTime: "2025-09-29T10:57:03Z"
    status: Pending
  - lastTransitionTime: "2025-09-29T10:57:07Z"
    status: Running
  - lastTransitionTime: "2025-09-29T10:57:38Z"
    status: Completed
  minAvailable: 3
  runningDuration: 35.688396584s
  state:
    lastTransitionTime: "2025-09-29T10:57:38Z"
    phase: Completed
  succeeded: 4
  taskStatusCount:
    job-nginx1:
      phase:
        Succeeded: 1
    job-nginx2:
      phase:
        Succeeded: 3
  version: 1

Results on the member cluster control plane:

status:
  conditions:
  - lastTransitionTime: "2025-09-29T10:57:03Z"
    status: Pending
  - lastTransitionTime: "2025-09-29T10:57:07Z"
    status: Running
  - lastTransitionTime: "2025-09-29T10:57:38Z"
    status: Completed
  minAvailable: 3
  runningDuration: 35.688396584s
  state:
    lastTransitionTime: "2025-09-29T10:57:38Z"
    phase: Completed
  succeeded: 4
  taskStatusCount:
    job-nginx1:
      phase:
        Succeeded: 1
    job-nginx2:
      phase:
        Succeeded: 3
  version: 1
  1. Aggregated results for multi-cluster scheduling
    View resource requests in the resourcebinding.
spec:
  clusters:
  - name: member1
  - name: member2
  components:
  - name: job-nginx1
    replicaRequirements:
      nodeClaim:
        nodeSelector:
          kubernetes.io/os: linux
      resourceRequest:
        cpu: 100m
    replicas: 1
  - name: job-nginx2
    replicaRequirements:
      nodeClaim:
        nodeSelector:
          kubernetes.io/os: linux
      resourceRequest:
        cpu: 100m
    replicas: 2

The results on the Karmada control plane are as follows:

status:
  conditions:
  - lastTransitionTime: "2025-09-30T02:07:56Z"
    status: Pending
  - lastTransitionTime: "2025-09-30T02:08:08Z"
    status: Running
  - lastTransitionTime: "2025-09-30T02:08:40Z"
    status: Completed
  - lastTransitionTime: "2025-09-30T02:07:56Z"
    status: Pending
  - lastTransitionTime: "2025-09-30T02:08:02Z"
    status: Running
  - lastTransitionTime: "2025-09-30T02:08:34Z"
    status: Completed
  minAvailable: 6
  runningDuration: 44.585525557s
  state:
    lastTransitionTime: "2025-09-30T02:08:34Z"
    phase: Completed
  succeeded: 8
  taskStatusCount:
    job-nginx1:
      phase:
        Succeeded: 2
    job-nginx2:
      phase:
        Succeeded: 6
  version: 1

Results on the member cluster control plane:
member1:

status:
  conditions:
  - lastTransitionTime: "2025-09-30T02:07:56Z"
    status: Pending
  - lastTransitionTime: "2025-09-30T02:08:08Z"
    status: Running
  - lastTransitionTime: "2025-09-30T02:08:40Z"
    status: Completed
  minAvailable: 3
  runningDuration: 44.585525557s
  state:
    lastTransitionTime: "2025-09-30T02:08:40Z"
    phase: Completed
  succeeded: 4
  taskStatusCount:
    job-nginx1:
      phase:
        Succeeded: 1
    job-nginx2:
      phase:
        Succeeded: 3
  version: 1

member2:

status:
  conditions:
  - lastTransitionTime: "2025-09-30T02:07:56Z"
    status: Pending
  - lastTransitionTime: "2025-09-30T02:08:02Z"
    status: Running
  - lastTransitionTime: "2025-09-30T02:08:34Z"
    status: Completed
  minAvailable: 3
  runningDuration: 38.49844198s
  state:
    lastTransitionTime: "2025-09-30T02:08:34Z"
    phase: Completed
  succeeded: 4
  taskStatusCount:
    job-nginx1:
      phase:
        Succeeded: 1
    job-nginx2:
      phase:
        Succeeded: 3
  version: 1

@dekaihu
Copy link
Contributor Author

dekaihu commented Oct 21, 2025

Results on the control side:

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: vc-job
  namespace: default
spec:
  maxRetry: 3
  minAvailable: 3
  plugins:
    env: []
    ssh: []
    svc:
    - --disable-network-policy=true
  queue: default
  schedulerName: volcano
  tasks:
  - minAvailable: 1
    name: job-nginx1
    replicas: 1
    template:
      metadata:
        name: nginx1
      spec:
        containers:
        - args:
          - sleep 10
          command:
          - bash
          - -c
          image: registry.cnbita.com:5000/hero-dev/nginx:latest
          imagePullPolicy: IfNotPresent
          name: nginx
          resources:
            requests:
              cpu: 100m
        nodeSelector:
          kubernetes.io/os: linux
        restartPolicy: OnFailure
  - minAvailable: 2
    name: job-nginx2
    replicas: 3
    template:
      metadata:
        name: nginx2
      spec:
        containers:
        - args:
          - sleep 100
          command:
          - bash
          - -c
          image: registry.cnbita.com:5000/hero-dev/nginx:latest
          imagePullPolicy: IfNotPresent
          name: nginx
          resources:
            requests:
              cpu: 100m
        nodeSelector:
          kubernetes.io/os: linux
        restartPolicy: OnFailure
status:
  conditions:
  - lastTransitionTime: "2025-10-21T07:56:08Z"
    status: Pending
  - lastTransitionTime: "2025-10-21T07:56:12Z"
    status: Running
  - lastTransitionTime: "2025-10-21T07:56:27Z"
    status: Aborting
  - lastTransitionTime: "2025-10-21T07:56:27Z"
    status: Aborted
  controlledResources:
    plugin-env: env
    plugin-ssh: ssh
    plugin-svc: svc
  minAvailable: 6
  running: 3
  runningDuration: 51.070546109s
  state:
    lastTransitionTime: "2025-10-21T07:56:59Z"
    message: 'Job failed in clusters: member1'
    phase: Failed
    reason: VolcanoJobFailed
  succeeded: 2
  taskStatusCount:
    job-nginx1:
      phase:
        Succeeded: 2
    job-nginx2:
      phase:
        Running: 3
  version: 14

member1:

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: vc-job
  namespace: default
spec:
  maxRetry: 3
  minAvailable: 3
  plugins:
    env: []
    ssh: []
    svc:
    - --disable-network-policy=true
  queue: default
  schedulerName: volcano
  tasks:
  - maxRetry: 3
    minAvailable: 1
    name: job-nginx1
    replicas: 1
    template:
      metadata:
        name: nginx1
      spec:
        containers:
        - args:
          - sleep 10
          command:
          - bash
          - -c
          image: registry.cnbita.com:5000/hero-dev/nginx:latest
          imagePullPolicy: IfNotPresent
          name: nginx
          resources:
            requests:
              cpu: 100m
        nodeSelector:
          kubernetes.io/os: linux
        restartPolicy: OnFailure
  - maxRetry: 3
    minAvailable: 2
    name: job-nginx2
    replicas: 3
    template:
      metadata:
        name: nginx2
      spec:
        containers:
        - args:
          - sleep 100
          command:
          - bash
          - -c
          image: registry.cnbita.com:5000/hero-dev/nginx:latest
          imagePullPolicy: IfNotPresent
          name: nginx
          resources:
            requests:
              cpu: 100m
        nodeSelector:
          kubernetes.io/os: linux
        restartPolicy: OnFailure
status:
  conditions:
  - lastTransitionTime: "2025-10-21T07:56:08Z"
    status: Pending
  - lastTransitionTime: "2025-10-21T07:56:12Z"
    status: Running
  - lastTransitionTime: "2025-10-21T07:56:27Z"
    status: Aborting
  - lastTransitionTime: "2025-10-21T07:56:27Z"
    status: Aborted
  minAvailable: 3
  runningDuration: 51.070546109s
  state:
    lastTransitionTime: "2025-10-21T07:56:27Z"
    phase: Aborted
  succeeded: 1
  taskStatusCount:
    job-nginx1:
      phase:
        Succeeded: 1
  version: 14

member2:

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: vc-job
  namespace: default
spec:
  maxRetry: 3
  minAvailable: 3
  plugins:
    env: []
    ssh: []
    svc:
    - --disable-network-policy=true
  queue: default
  schedulerName: volcano
  tasks:
  - maxRetry: 3
    minAvailable: 1
    name: job-nginx1
    replicas: 1
    template:
      metadata:
        name: nginx1
      spec:
        containers:
        - args:
          - sleep 10
          command:
          - bash
          - -c
          image: registry.cnbita.com:5000/hero-dev/nginx:latest
          imagePullPolicy: IfNotPresent
          name: nginx
          resources:
            requests:
              cpu: 100m
        nodeSelector:
          kubernetes.io/os: linux
        restartPolicy: OnFailure
  - maxRetry: 3
    minAvailable: 2
    name: job-nginx2
    replicas: 3
    template:
      metadata:
        name: nginx2
      spec:
        containers:
        - args:
          - sleep 100
          command:
          - bash
          - -c
          image: registry.cnbita.com:5000/hero-dev/nginx:latest
          imagePullPolicy: IfNotPresent
          name: nginx
          resources:
            requests:
              cpu: 100m
        nodeSelector:
          kubernetes.io/os: linux
        restartPolicy: OnFailure
status:
  conditions:
  - lastTransitionTime: "2025-10-21T07:56:08Z"
    status: Pending
  - lastTransitionTime: "2025-10-21T07:56:11Z"
    status: Running
  - lastTransitionTime: "2025-10-21T07:57:53Z"
    status: Completed
  minAvailable: 3
  runningDuration: 1m45.830414234s
  state:
    lastTransitionTime: "2025-10-21T07:57:53Z"
    phase: Completed
  succeeded: 4
  taskStatusCount:
    job-nginx1:
      phase:
        Succeeded: 1
    job-nginx2:
      phase:
        Succeeded: 3
  version: 1

Copy link
Member

@XiShanYongYe-Chang XiShanYongYe-Chang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we currently not considering supporting the split scenario and follow-distribution scenario for Volcano jobs?

@dekaihu
Copy link
Contributor Author

dekaihu commented Oct 22, 2025

Are we currently not considering supporting the split scenario and follow-distribution scenario for Volcano jobs?

Yes, currently VolcanoJob is still focused on batch scheduling within a single cluster.

Copy link
Member

@RainbowMango RainbowMango left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a nit, otherwise LGTM.

Signed-off-by: hudekai <2812840067@qq.com>
Copy link
Member

@RainbowMango RainbowMango left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve
Thanks~

@karmada-bot karmada-bot added the lgtm Indicates that a PR is ready to be merged. label Oct 23, 2025
@karmada-bot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: RainbowMango

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@karmada-bot karmada-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 23, 2025
@karmada-bot karmada-bot merged commit 8a84318 into karmada-io:master Oct 23, 2025
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants