KAFKA-19501: Update OpenJDK base image from buster to bullseye #20165

dalaoqi · 2025-07-14T14:01:44Z

The changes update the OpenJDK base image from 17-buster to 17-bullseye:

Updates tests/docker/Dockerfile to use openjdk:17-bullseye instead of
openjdk:17-buster - Updates tests/docker/ducker-ak script to use the
new default image - Updates documentation in tests/README.md with the
new image name examples

Reviewers: Federico Valeri fedevaleri@gmail.com, TengYao Chi
kitingiao@gmail.com, Ken Huang s7133700@gmail.com, Chia-Ping Tsai
chia7712@gmail.com

Signed-off-by: Tsung-Han Ho (Miles Ho) <mystes3016@gmail.com>

frankvicky

@dalaoqi
Could you run a random e2e test locally and paste the result?
Thanks

dalaoqi · 2025-07-16T04:22:05Z

@frankvicky
Here's the result of "tests/kafkatest/tests/client/consumer_test.py::OffsetValidationTest"

frankvicky

@dalaoqi: LGTM

fvaleri

LGTM. Left one comment for you to consider. Thanks.

I would also like to mention that, currently, the image build is broken due to stale apt-get sources list in the openjdk:17-buster base image:

 > [stage-1  3/60] RUN apt update && apt install -y sudo git netcat iptables rsync unzip wget curl jq coreutils openssh-server net-tools vim python3-pip python3-dev libffi-dev libssl-dev cmake pkg-config libfuse-dev iperf traceroute iproute2 iputils-ping && apt-get -y clean:        
0.267                                                                                                                                                                                                                                                                                      
0.267 WARNING: apt does not have a stable CLI interface. Use with caution in scripts.                                                                                                                                                                                                      
0.267                                                                                                                                                                                                                                                                                      
0.362 Ign:1 http://deb.debian.org/debian buster InRelease
0.362 Ign:2 http://security.debian.org/debian-security buster/updates InRelease
0.367 Ign:3 http://deb.debian.org/debian buster-updates InRelease
0.369 Err:4 http://security.debian.org/debian-security buster/updates Release
0.369   404  Not Found [IP: 151.101.66.132 80]
0.375 Err:5 http://deb.debian.org/debian buster Release
0.375   404  Not Found [IP: 151.101.2.132 80]
0.380 Err:6 http://deb.debian.org/debian buster-updates Release
0.380   404  Not Found [IP: 151.101.2.132 80]
0.384 Reading package lists...
0.391 E: The repository 'http://security.debian.org/debian-security buster/updates Release' does not have a Release file.
0.391 E: The repository 'http://deb.debian.org/debian buster Release' does not have a Release file.
0.391 E: The repository 'http://deb.debian.org/debian buster-updates Release' does not have a Release file.

fvaleri · 2025-07-16T09:36:04Z

tests/docker/Dockerfile

-# Therefore, use openjdk:17-buster instead.
-ARG jdk_version=openjdk:17-buster
+# Therefore, use openjdk:17-bullseye instead.
+ARG jdk_version=openjdk:17-bullseye


Should we remove the default value here since we are taking that from the build script where we also have a default value?

If we also update the README.md using a placeholder (e.g. <JDK_IMAGE>), in the next base image update we would only need to update the default image name in one place. Wdyt?

sounds good

chia7712 · 2025-07-16T12:53:29Z

I would also like to mention that, currently, the image build is broken due to stale apt-get sources list in the openjdk:17-buster base image:

Yes, that's why I created this ticket 😄

fvaleri · 2025-07-16T13:03:12Z

I would also like to mention that, currently, the image build is broken due to stale apt-get sources list in the openjdk:17-buster base image:

Yes, that's why I created this ticket 😄

Oh yeah, sorry. I searched directly in open PRs, and wasn't clear we were updating to address the issue.

Signed-off-by: Tsung-Han Ho (Miles Ho) <mystes3016@gmail.com>

chia7712 · 2025-07-16T13:16:23Z

tools/src/main/java/org/apache/kafka/tools/DelegationTokenCommand.java

@@ -301,7 +301,7 @@ public void checkArgs() {
            CommandLineUtils.checkInvalidArgs(parser, options, createOpt, Set.of(hmacOpt, renewTimePeriodOpt, expiryTimePeriodOpt));
            CommandLineUtils.checkInvalidArgs(parser, options, renewOpt, Set.of(renewPrincipalsOpt, maxLifeTimeOpt, expiryTimePeriodOpt, ownerPrincipalsOpt));
            CommandLineUtils.checkInvalidArgs(parser, options, expiryOpt, Set.of(renewOpt, maxLifeTimeOpt, renewTimePeriodOpt, ownerPrincipalsOpt));
-            CommandLineUtils.checkInvalidArgs(parser, options, describeOpt, Set.of(renewTimePeriodOpt, maxLifeTimeOpt, hmacOpt, renewTimePeriodOpt, expiryTimePeriodOpt));
+            CommandLineUtils.checkInvalidArgs(parser, options, describeOpt, Set.of(renewTimePeriodOpt, maxLifeTimeOpt, hmacOpt, expiryTimePeriodOpt));


please revert this unrelated change

It seems that this will cause the e2e error.
The duplicate arguments will cause Set.of throws IllegalArgumentException.
We could open another patch to fix it.

I mean it would be great to create another PR to fix it with extra unit test.

open #20177

… validation" This reverts commit 5d62424.

Signed-off-by: Tsung-Han Ho (Miles Ho) <mystes3016@gmail.com>

chia7712 · 2025-07-16T13:47:04Z

tests/README.md

@@ -49,11 +49,11 @@ TC_PATHS="tests/kafkatest/tests/streams/streams_upgrade_test.py::StreamsUpgradeT
 ```
 * Run tests with a specific image name
 ```
-image_name="ducker-ak-openjdk:17-buster" bash tests/docker/run_tests.sh
+image_name="ducker-ak-<JDK_IMAGE>" bash tests/docker/run_tests.sh


I think @fvaleri's suggestion is to add an example for the usage of jdk_version

chia7712 · 2025-07-16T13:49:42Z

tests/README.md

@@ -49,11 +49,11 @@ TC_PATHS="tests/kafkatest/tests/streams/streams_upgrade_test.py::StreamsUpgradeT
 ```
 * Run tests with a specific image name
 ```
-image_name="ducker-ak-openjdk:17-buster" bash tests/docker/run_tests.sh


for another, perhaps we could use openjdk:17 for this example to avoid using the explicit os version.

chia7712 · 2025-07-16T13:50:09Z

tests/docker/Dockerfile

@@ -14,8 +14,8 @@
 # limitations under the License.

 # The base image of openjdk:17 is typically oraclelinux:8-slim, which doesn't include apt-get. 


this comment should be moved to ducker-ak

Signed-off-by: Tsung-Han Ho (Miles Ho) <mystes3016@gmail.com>

chia7712

LGTM

fvaleri

@dalaoqi thanks for the changes, just left a small improvement suggestion.

fvaleri · 2025-07-16T14:50:01Z

tests/README.md

 ```
 * Run tests with a different JVM
 ```
 bash tests/docker/ducker-ak up -j '<JDK_IMAGE>'; tests/docker/run_tests.sh
 ```
+  You can customize the OpenJDK base image using the `-j` or `--jdk` parameter. The default is `openjdk:17-bullseye`.


What about rewriting like this:

You can customize the OpenJDK base image using the -j or --jdk parameter, otherwise a default value will be used.

Signed-off-by: Tsung-Han Ho (Miles Ho) <mystes3016@gmail.com>

m1a2st · 2025-07-16T15:10:46Z

tests/docker/ducker-ak

@@ -84,6 +86,9 @@ up [-n|--num-nodes NUM_NODES] [-f|--force] [docker-image]
    or a combination of port/port-range separated by comma (like 2181,9092 or 2181,5005-5008).
    By default no port is exposed. See README.md for more detail on this option.

+    If -j|--jdk is specified, you can customize the OpenJDK base image used for building
+    the ducker container. Defaults to ${default_jdk}. Example: -j openjdk:11-bullseye


Since current Kafka versions no longer support JDK 11, it would be better to use JDK 17 in this example.

fvaleri

Thanks.

Signed-off-by: Tsung-Han Ho (Miles Ho) <mystes3016@gmail.com>

chia7712 · 2025-07-16T19:26:33Z

the failed test is traced by https://issues.apache.org/jira/browse/KAFKA-19513

mimaison · 2025-07-22T14:00:06Z

We probably want to backport this as we now can't run system tests anymore on the existing release branches: 4.1, 4.0, etc

With buster we get the following error message when running apt update:

E: The repository 'http://security.debian.org/debian-security buster/updates Release' does not have a Release file.
E: The repository 'http://deb.debian.org/debian buster Release' does not have a Release file.
E: The repository 'http://deb.debian.org/debian buster-updates Release' does not have a Release file.
ERROR: process "/bin/sh -c apt update && apt install -y sudo git netcat iptables rsync unzip wget curl jq coreutils openssh-server net-tools vim python3-pip python3-dev libffi-dev libssl-dev cmake pkg-config libfuse-dev iperf traceroute iproute2 iputils-ping && apt-get -y clean" did not complete successfully: exit code: 100

The changes update the OpenJDK base image from 17-buster to 17-bullseye: - Updates tests/docker/Dockerfile to use openjdk:17-bullseye instead of openjdk:17-buster - Updates tests/docker/ducker-ak script to use the new default image - Updates documentation in tests/README.md with the new image name examples Reviewers: Federico Valeri <fedevaleri@gmail.com>, TengYao Chi <kitingiao@gmail.com>, Ken Huang <s7133700@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>

…#20165) The changes update the OpenJDK base image from 17-buster to 17-bullseye: - Updates tests/docker/Dockerfile to use openjdk:17-bullseye instead of openjdk:17-buster - Updates tests/docker/ducker-ak script to use the new default image - Updates documentation in tests/README.md with the new image name examples Reviewers: Federico Valeri <fedevaleri@gmail.com>, TengYao Chi <kitingiao@gmail.com>, Ken Huang <s7133700@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>

chia7712 · 2025-07-22T14:31:59Z

@mimaison yes, you are right. Should I backport it to 4.1 after the 4.1.0 release?

mimaison · 2025-07-22T14:33:16Z

I've just backported it to 4.1, otherwise we can't run the system tests.

We'll also need to backport it to 4.0 for the 4.0.1 release, and possibly to 3.9 in case we ever to another bugfix release.

…#20165) The changes update the OpenJDK base image from 17-buster to 17-bullseye: - Updates tests/docker/Dockerfile to use openjdk:17-bullseye instead of openjdk:17-buster - Updates tests/docker/ducker-ak script to use the new default image - Updates documentation in tests/README.md with the new image name examples Reviewers: Federico Valeri <fedevaleri@gmail.com>, TengYao Chi <kitingiao@gmail.com>, Ken Huang <s7133700@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>

chia7712 · 2025-07-22T14:42:06Z

We'll also need to backport it to 4.0 for the 4.0.1 release, and possibly to 3.9 in case we ever to another bugfix release.

ok, I will backport it to 4.0 and 3.9 after fixing the conflicts on my local.

chia7712 · 2025-07-22T15:03:29Z

backport to 74d93ad

3.9 has no such issue.

…#20165) The changes update the OpenJDK base image from 17-buster to 17-bullseye: - Updates tests/docker/Dockerfile to use openjdk:17-bullseye instead of openjdk:17-buster - Updates tests/docker/ducker-ak script to use the new default image - Updates documentation in tests/README.md with the new image name examples Reviewers: Federico Valeri <fedevaleri@gmail.com>, TengYao Chi <kitingiao@gmail.com>, Ken Huang <s7133700@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>

Update OpenJDK base image from buster to bullseye

da2c6a8

Signed-off-by: Tsung-Han Ho (Miles Ho) <mystes3016@gmail.com>

github-actions bot added triage PRs from the community tests Test fixes (including flaky tests) small Small PRs labels Jul 14, 2025

frankvicky added the ci-approved label Jul 14, 2025

frankvicky reviewed Jul 16, 2025

View reviewed changes

github-actions bot removed the triage PRs from the community label Jul 16, 2025

frankvicky approved these changes Jul 16, 2025

View reviewed changes

frankvicky requested a review from chia7712 July 16, 2025 09:36

fvaleri reviewed Jul 16, 2025

View reviewed changes

Remove duplicate renewTimePeriodOpt in DelegationTokenCommand validation

5d62424

Signed-off-by: Tsung-Han Ho (Miles Ho) <mystes3016@gmail.com>

github-actions bot added the tools label Jul 16, 2025

chia7712 reviewed Jul 16, 2025

View reviewed changes

dalaoqi added 2 commits July 16, 2025 21:18

Revert "Remove duplicate renewTimePeriodOpt in DelegationTokenCommand…

297209b

… validation" This reverts commit 5d62424.

Remove duplicate JDK image defaults

f6a5219

Signed-off-by: Tsung-Han Ho (Miles Ho) <mystes3016@gmail.com>

dalaoqi requested a review from chia7712 July 16, 2025 13:44

chia7712 reviewed Jul 16, 2025

View reviewed changes

Update documentation

4880171

Signed-off-by: Tsung-Han Ho (Miles Ho) <mystes3016@gmail.com>

dalaoqi requested a review from chia7712 July 16, 2025 14:13

chia7712 approved these changes Jul 16, 2025

View reviewed changes

fvaleri reviewed Jul 16, 2025

View reviewed changes

Update documentation

3f45d09

Signed-off-by: Tsung-Han Ho (Miles Ho) <mystes3016@gmail.com>

m1a2st reviewed Jul 16, 2025

View reviewed changes

fvaleri approved these changes Jul 16, 2025

View reviewed changes

Update documentation

ebcb35b

Signed-off-by: Tsung-Han Ho (Miles Ho) <mystes3016@gmail.com>

chia7712 merged commit 70824be into apache:trunk Jul 16, 2025
19 of 21 checks passed

		@@ -14,8 +14,8 @@
		# limitations under the License.

		# The base image of openjdk:17 is typically oraclelinux:8-slim, which doesn't include apt-get.

KAFKA-19501: Update OpenJDK base image from buster to bullseye #20165

KAFKA-19501: Update OpenJDK base image from buster to bullseye #20165

Uh oh!

Conversation

dalaoqi commented Jul 14, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

frankvicky left a comment

Choose a reason for hiding this comment

Uh oh!

dalaoqi commented Jul 16, 2025

Uh oh!

frankvicky left a comment

Choose a reason for hiding this comment

Uh oh!

fvaleri left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chia7712 commented Jul 16, 2025

Uh oh!

fvaleri commented Jul 16, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chia7712 left a comment

Choose a reason for hiding this comment

Uh oh!

fvaleri left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fvaleri left a comment

Choose a reason for hiding this comment

Uh oh!

chia7712 commented Jul 16, 2025

Uh oh!

Uh oh!

mimaison commented Jul 22, 2025

Uh oh!

chia7712 commented Jul 22, 2025

Uh oh!

mimaison commented Jul 22, 2025

Uh oh!

chia7712 commented Jul 22, 2025

Uh oh!

chia7712 commented Jul 22, 2025

Uh oh!

Uh oh!

dalaoqi commented Jul 14, 2025 •

edited by github-actions bot

Loading