internal/testrunner/script: add script testing package #3012

efd6 · 2025-10-21T02:14:46Z

This adds script testing for data streams. This is the MVP, future versions can be generalised to operate on input packages.

The current implementation supports:

pipeline testing
system testing
package upgrade testing (only to latest, not to arbitrary versions)
shared stack
independent stack
docker services

Not supported:

test coverage
report output configuration
k8s services
tf services (ish, these can be shimmed via docker)

Depends on a v3.5.1 of package-spec (currently shimmed in via a go.mod replace directive, to be removed)

Please take a look.

For #2944

efd6 · 2025-10-21T05:26:28Z

/test

.go-version

efd6 · 2025-10-21T21:10:58Z

Makefile

 test-check-packages-other:
 	PACKAGE_TEST_TYPE=other ./scripts/test-check-packages.sh

+test-check-packages-independent-script:


This does not appear to run in the CI, but it's entirely unclear how to achieve that.

In order to be executed in the CI, it needs to be updated this shell script that generates dynamically all the Buildkite steps to be executed:

https://github.com/elastic/elastic-package/blob/d5f73ab15af1dcf3547ed789f8ad4631257f4197/.buildkite/pipeline.trigger.integration.tests.sh

Depending on how it is required to launch these steps, all packages in one step or each package in its own CI step, it would be needed to do different modifications:

all packages in one step:

similar to with-kind or other CI step (Buildkite link)

it should be added a new element (Makefile target name) to this list:

elastic-package/.buildkite/pipeline.trigger.integration.tests.sh

Lines 42 to 47 in d5f73ab

CHECK_PACKAGES_TESTS=(

test-check-packages-other

test-check-packages-with-kind

test-check-packages-with-custom-agent

test-check-packages-benchmarks

)

each package in its own CI step:

it would be needed to duplicate this code for the new Makefile target name

for instance, for packages under parallel folder:

elastic-package/.buildkite/pipeline.trigger.integration.tests.sh

Lines 117 to 138 in d5f73ab

pushd test/packages/parallel > /dev/null

while IFS= read -r -d '' package ; do

package_name=$(basename "${package}")

echo " - label: \":go: Integration test: ${package_name}\""

echo " key: \"integration-parallel-${package_name}-agent\""

echo " command: ./.buildkite/scripts/integration_tests.sh -t test-check-packages-parallel -p ${package_name}"

echo " env:"

echo " UPLOAD_SAFE_LOGS: 1"

echo " agents:"

echo " provider: \"gcp\""

echo " image: \"${UBUNTU_X86_64_AGENT_IMAGE}\""

echo " plugins:"

echo " # See https://github.com/elastic/oblt-infra/blob/main/conf/resources/repos/integrations/01-gcp-buildkite-oidc.tf"

echo " # This plugin authenticates to Google Cloud using the OIDC token."

echo " - elastic/oblt-google-auth#v1.2.0:"

echo " lifetime: 10800 # seconds"

echo " project-id: \"elastic-observability-ci\""

echo " project-number: \"911195782929\""

echo " artifact_paths:"

echo " - build/test-results/*.xml"

echo " - build/test-coverage/coverage-*.xml" # these files should not be used to compute the final coverage of elastic-package

done < <(find . -maxdepth 1 -mindepth 1 -type d -print0)

efd6 · 2025-10-21T21:12:29Z

cmd/testrunner.go

+	return cmd
+}
+
+func testRunnerScriptCommandAction(cmd *cobra.Command, args []string) error {


The logic in this function is rudimentary and only intended to allow an MVP to be demonstrated. In order to be able to render reports and redirect to file output this needs enhancement.

go.mod

jsoriano

Thanks! This is a pretty interesting feature.

I haven't checked the implementation in detail, added by now some comments and questions about the overall behavior.

docs/howto/script_testing.md

jsoriano · 2025-11-04T18:41:37Z

docs/howto/script_testing.md

+[!exec:jq] skip 'Skipping test requiring absent jq command'
+
+# Register running stack.
+use_stack -profile ${CONFIG_PROFILES}/default


We should not [allow to] hard-code profiles in scripts. The user may want to use a different one. If needed here, we could pass the current profile as another environment variable.

If we want to allow testing with multiple profiles or something like this, maybe we can provide some function to create and use temporary profiles.

I don't really like this. The point of the testing here is to make it more hermetic, but I think that game is already lost. So, done.

docs/howto/script_testing.md

jsoriano · 2025-11-04T19:36:55Z

internal/testrunner/script/script.go

+	if err != nil {
+		return err
+	}
+	workRoot := filepath.Join(home, filepath.FromSlash(".elastic-package/tmp/script_tests"))


Config directory should be obtained using the location manager. It can be customized with ELASTIC_PACKAGE_DATA_HOME.

The tmp directory can be obtained with loc.TempDir().

Suggested change

workRoot := filepath.Join(home, filepath.FromSlash(".elastic-package/tmp/script_tests"))

workRoot := filepath.Join(home, loc.TempDir(), "script_tests")

jsoriano · 2025-11-04T19:40:10Z

internal/testrunner/script/script.go

+		// root is non-zero, so just let testscript put it where it wants in the
+		// case that we have not requested work to be retained. This will be in
+		// os.MkdirTemp(os.Getenv("GOTMPDIR"), "go-test-script") which on most
+		// systems will be /tmp/go-test-script. However, due to… decisions, we


What decisions are you referring to? 🙂

The use of file system jailing prevents using other paths.

Maybe this should be part of the coment 🙂

test/packages/other/with_script/data_stream/first/_dev/test/scripts/agent_up_down.txt

jsoriano · 2025-11-04T19:46:57Z

docs/howto/script_testing.md

+Tests are written as [txtar format](https://pkg.go.dev/golang.org/x/tools/txtar#hdr-Txtar_format)
+files in a data stream's \_dev/test/scripts directory. The logic for the test is
+written in the txtar file's initial comment section and any additional resource
+files are included in the txtar file's files sections.
+
+The standard commands and behaviors for testscript scripts are documented in
+the [testscript package documentation](https://pkg.go.dev/github.com/rogpeppe/go-internal/testscript).


Just for the record. Could you comment, maybe in the description of the PR, why you chose testscript instead of some other embedded language?

It's the only well maintained and extensively used script testing package available with the features that we need. It's not a language, so I'm not sure what competing systems you are referring to.

It's the only well maintained and extensively used script testing package available with the features that we need.

I see it as a great tool for black-box testing of commands. Here we are exposing many functions as commands, that could stay as functions if we were using some embedded scripting language.

It's not a language

Well, I can agree that it is not a turing-complete language, but it is something that along with txtar is used to express scripts, what could very well called language, but I guess this is a semantic discussion 🙂

so I'm not sure what competing systems you are referring to.

There are other embeddable systems that can be used to express scripts, from goja to CEL, starlark, or even a yaml list.

I agree with the choice, only that it wouldn't have been the first option I would have thought of for this, and I would like to have the decision stored in some place.

I've added it to the issue that this is addressing.

jsoriano · 2025-11-04T19:51:59Z

docs/howto/script_testing.md

+# Only run the test if --external-stack=true.
+[!external_stack] skip 'Skipping external stack test.'
+# Only run the test if the jq executable is in $PATH. This is needed for a test below.
+[!exec:jq] skip 'Skipping test requiring absent jq command'


It looks like several sample tests here depend on jq to compare fields in JSON files. If this is going to be frequent, maybe we can provide a function to do these comparisons? And maybe the same with basic http requests to avoid depending on curl.

This would help to avoid depending on external tools that may not be installed, specially on Windows.

If it becomes necessary, those can be added. Both curl and jq are available in CI and I'd prefer to avoid having to reimplement and maintain tools that already exist.

Agree this can be added later.

I am not so worried about the current CI, but about future deployments where we might forget to install jq (or curl in Windows) if not required for other things, and the test would be silently skipped. Same thing for local execution.

jq and curl are powerful tools, in no case I would think on re-implement them, but the limited logic that we need, also reusing existing tools, in the form of Go code.

mrodm

Added a comment about CI scripts.

The error raised in the latest Buildkite build should be fixed now, since it has been merged the new spec version #3049

mrodm · 2025-11-05T13:24:30Z

Makefile

 	PACKAGE_TEST_TYPE=other ./scripts/test-check-packages.sh

+test-check-packages-independent-script:
+	elastic-package test script -C test/packages/other/with_script --external-stack=false --defer-cleanup 1s


I would create a new folder for this test package like test/packages/independent-script/ and just keep (move) that new test package into this new folder.

The reasoning of this is that currently, this with_script test package is being tested twice in Buildkite builds. In these two steps:

Integration test: other

Integration test: independent-script

So, I would create a new folder and update this target:

Suggested change

elastic-package test script -C test/packages/other/with_script --external-stack=false --defer-cleanup 1s

elastic-package test script -C test/packages/independent-script/with_script --external-stack=false --defer-cleanup 1s

mrodm · 2025-11-05T13:27:56Z

docs/howto/script_testing.md

+- `--run`: run only tests matching the regular expression
+- `--scripts`: path to directory containing test scripts (advanced use only)
+- `--update`: update archive file if a cmp fails
+- `--verbose-scripts`: verbose script test output (show all script logging)


We should probably start using named loggers, and some global selector flag. cc @mrodm

At least, the current logger set in elastic-package supports to set -v and -vv (or -v -v). With the former, DEBUG messages are shown. With the latter, TRACE messages are shown too.

elastic-package/internal/logger/logger.go

Lines 52 to 65 in 457d321

func Trace(a ...interface{}) {

if !IsTraceMode() {

return

}

logMessage("TRACE", a...)

}

// Tracef method logs message with "trace" level and formats it.

func Tracef(format string, a ...interface{}) {

if !IsTraceMode() {

return

}

logMessagef("TRACE", format, a...)

}

elasticmachine · 2025-11-05T22:23:49Z

💚 Build Succeeded

Buildkite Build
Commit: d5eced9

History

💔 Build #6525 failed 3514b1d
💔 Build #6524 failed 3283207
💔 Build #6512 failed c2e7dad
💔 Build #6508 failed c8ce65a
💔 Build #6502 failed 0f9015a
💔 Build #6501 failed 0e84c58

cc @efd6

efd6 self-assigned this Oct 21, 2025

efd6 added the enhancement New feature or request label Oct 21, 2025

efd6 force-pushed the script_tests branch 5 times, most recently from 411a39e to bc55805 Compare October 21, 2025 04:31

efd6 force-pushed the script_tests branch 2 times, most recently from d9e0c1a to 8433c67 Compare October 21, 2025 20:16

efd6 marked this pull request as ready for review October 21, 2025 21:09

efd6 requested a review from a team as a code owner October 21, 2025 21:09

efd6 commented Oct 21, 2025

View reviewed changes

efd6 requested a review from andrewkroh October 21, 2025 21:14

jsoriano self-requested a review October 23, 2025 08:34

efd6 force-pushed the script_tests branch 5 times, most recently from 7f4815f to fdc0951 Compare October 28, 2025 06:46

efd6 force-pushed the script_tests branch from 0e84c58 to 0f9015a Compare November 4, 2025 01:01

jsoriano reviewed Nov 4, 2025

View reviewed changes

efd6 force-pushed the script_tests branch from c8ce65a to c2e7dad Compare November 5, 2025 03:57

mrodm reviewed Nov 5, 2025

View reviewed changes

efd6 added 6 commits November 6, 2025 06:50

internal/testrunner/script: add script testing package

ab30d9e

repair

9d132cf

escape

9825fc9

more repair

09e3c8b

run independent stack script test in ci

f72e96c

bump github.com/elastic/package-spec to v3.5.1

0da1133

efd6 added 3 commits November 6, 2025 06:51

address pr comments: documentation

7588561

address pr comment: use loc.TempDir

819f814

address pr comment: expose PROFILE env var

a524978

efd6 force-pushed the script_tests branch 3 times, most recently from 3514b1d to 31b44ff Compare November 5, 2025 21:29

names

d5eced9

efd6 force-pushed the script_tests branch from 31b44ff to d5eced9 Compare November 5, 2025 21:53

	CHECK_PACKAGES_TESTS=(
	test-check-packages-other
	test-check-packages-with-kind
	test-check-packages-with-custom-agent
	test-check-packages-benchmarks
	)

	pushd test/packages/parallel > /dev/null
	while IFS= read -r -d '' package ; do
	package_name=$(basename "${package}")
	echo " - label: \":go: Integration test: ${package_name}\""
	echo " key: \"integration-parallel-${package_name}-agent\""
	echo " command: ./.buildkite/scripts/integration_tests.sh -t test-check-packages-parallel -p ${package_name}"
	echo " env:"
	echo " UPLOAD_SAFE_LOGS: 1"
	echo " agents:"
	echo " provider: \"gcp\""
	echo " image: \"${UBUNTU_X86_64_AGENT_IMAGE}\""
	echo " plugins:"
	echo " # See https://github.com/elastic/oblt-infra/blob/main/conf/resources/repos/integrations/01-gcp-buildkite-oidc.tf"
	echo " # This plugin authenticates to Google Cloud using the OIDC token."
	echo " - elastic/oblt-google-auth#v1.2.0:"
	echo " lifetime: 10800 # seconds"
	echo " project-id: \"elastic-observability-ci\""
	echo " project-number: \"911195782929\""
	echo " artifact_paths:"
	echo " - build/test-results/*.xml"
	echo " - build/test-coverage/coverage-*.xml" # these files should not be used to compute the final coverage of elastic-package
	done < <(find . -maxdepth 1 -mindepth 1 -type d -print0)

	workRoot := filepath.Join(home, filepath.FromSlash(".elastic-package/tmp/script_tests"))
	workRoot := filepath.Join(home, loc.TempDir(), "script_tests")

	elastic-package test script -C test/packages/other/with_script --external-stack=false --defer-cleanup 1s
	elastic-package test script -C test/packages/independent-script/with_script --external-stack=false --defer-cleanup 1s

	func Trace(a ...interface{}) {
	if !IsTraceMode() {
	return
	}
	logMessage("TRACE", a...)
	}

	// Tracef method logs message with "trace" level and formats it.
	func Tracef(format string, a ...interface{}) {
	if !IsTraceMode() {
	return
	}
	logMessagef("TRACE", format, a...)
	}

internal/testrunner/script: add script testing package #3012

Are you sure you want to change the base?

internal/testrunner/script: add script testing package #3012

Conversation

efd6 commented Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

efd6 commented Oct 21, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jsoriano left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mrodm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Nov 5, 2025

💚 Build Succeeded

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

efd6 commented Oct 21, 2025 •

edited

Loading