You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/guides/modules/test/pages/smarter-testing.adoc
+41-37Lines changed: 41 additions & 37 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,36 +1,37 @@
1
-
= Adaptive testing
1
+
= Smarter Testing
2
2
:page-badge: Preview
3
3
:page-platform: Cloud
4
-
:page-description: This document describes the adaptive testing feature in CircleCI, which enables only running tests that are impacted by code changes and evenly distributes tests across parallel execution nodes.
4
+
:page-description: This page describes CircleCI's Smarter Testing. Only run tests that are impacted by code changes and evenly distribute tests across parallel execution nodes.
5
5
:experimental:
6
6
:page-noindex: true
7
+
:page-aliases: adaptive-testing.adoc
7
8
8
-
CAUTION: *Adaptive testing* is available in closed preview. When the feature is made generally available there will be a cost associated with access and usage.
9
+
CAUTION: *Smarter Testing* is available in closed preview. When the feature is made generally available there will be a cost associated with access and usage.
9
10
10
11
NOTE: This page is currently in development and will be updated as the feature is developed.
11
12
12
-
Use adaptive testing to optimize test runs as follows:
13
+
Use Smarter Testing to optimize test runs as follows:
13
14
14
15
* Run only tests that are impacted by code changes.
15
16
* Evenly distribute tests across parallel execution nodes.
16
17
17
-
Adaptive testing reduces test execution time while maintaining test confidence.
18
+
Smarter Testing reduces test execution time while maintaining test confidence.
18
19
19
-
== Is my project a good fit for adaptive testing?
20
+
== Is my project a good fit for Smarter Testing?
20
21
21
-
The following list shows some examples of where adaptive testing can be most beneficial:
22
+
The following list shows some examples of where Smarter Testing can be most beneficial:
22
23
23
24
* Tests that exercise code within the same repository.
24
-
* Projects with comprehensive test coverage. The more thorough your tests, the more precisely adaptive testing can identify which tests are impacted by changes.
25
+
* Projects with comprehensive test coverage. The more thorough your tests, the more precisely Smarter Testing can identify which tests are impacted by changes.
25
26
* Test frameworks with built-in coverage support (Jest, pytest, Go test, Vitest) where generating coverage reports is straightforward.
26
27
+
27
-
TIP: In codebases with sparse test coverage, adaptive testing cannot accurately determine which tests cover changed code. This causes the system to run more tests, reducing the benefits of intelligent test selection.
28
+
TIP: In codebases with sparse test coverage, Smarter Testing cannot accurately determine which tests cover changed code. This causes the system to run more tests, reducing the benefits of intelligent test selection.
28
29
29
30
== Limitations
30
31
31
-
* Generating code coverage data is essential for determining how tests are related to code. If tests are run in a way that makes generating and accessing code coverage data tricky then adaptive testing may not be a good fit.
32
-
* Adaptive testing works best when testing a single deployable artifact. For example, a monorepo with integration tests that span multiple packages/services, especially when services run in separate container, makes coverage generation and consolidation difficult.
33
-
* Adaptive testing needs to be configured with commands to discover all available tests and run a subset of those tests. If you cannot run commands to discover tests and run a subset of tests on the CLI then adaptive testing may not be a good fit.
32
+
* Generating code coverage data is essential for determining how tests are related to code. If tests are run in a way that makes generating and accessing code coverage data tricky then Smarter Testing may not be a good fit.
33
+
* Smarter Testing works best when testing a single deployable artifact. For example, a monorepo with integration tests that span multiple packages/services, especially when services run in separate container, makes coverage generation and consolidation difficult.
34
+
* Smarter Testing needs to be configured with commands to discover all available tests and run a subset of those tests. If you cannot run commands to discover tests and run a subset of tests on the CLI then Smarter Testing may not be a good fit.
34
35
35
36
== Key benefits
36
37
@@ -40,17 +41,17 @@ TIP: In codebases with sparse test coverage, adaptive testing cannot accurately
40
41
* Scale efficiently as test suites grow.
41
42
42
43
== How it works
43
-
Adaptive testing operates through two main components that work together to optimize your test execution:
44
+
Smarter Testing operates through two main components that work together to optimize your test execution:
44
45
45
-
* Dynamic test splitting
46
-
* Test impact analysis
46
+
* Dynamic test splitting.
47
+
* Test impact analysis.
47
48
48
49
Each component is described in more detail below.
49
50
50
51
=== Dynamic test splitting
51
52
Dynamic test splitting distributes your tests across parallel execution nodes. The system maintains a shared queue that each node pulls from to create a balanced workload.
52
53
53
-
When you configure parallelism in your job, adaptive testing automatically:
54
+
When you configure parallelism in your job, Smarter Testing automatically:
54
55
55
56
* Retrieves timing data from previous test runs.
56
57
* Calculates optimal test distribution across your specified number of parallel nodes.
@@ -293,15 +294,15 @@ options:
293
294
294
295
If tests are still slower, share the pipeline link in the closed beta slack channel.
295
296
296
-
== 2. Enable adaptive testing
297
+
== 2. Enable Smarter Testing
297
298
298
-
We recommend following the steps in <<getting-started>> first before enabling the adaptive testing feature to ensure the `discover` and `run` commands are set up correctly.
299
+
We recommend following the steps in <<getting-started>> first before enabling the Smarter Testing feature to ensure the `discover` and `run` commands are set up correctly.
299
300
300
-
The goal of this section is to enable adaptive testing for your test suite.
301
+
The goal of this section is to enable Smarter Testing for your test suite.
301
302
302
303
=== 2.1 Update the test suites file
303
304
304
-
When using adaptive testing for test impact analysis the following commands are used:
305
+
When using Smarter Testing for test impact analysis the following commands are used:
305
306
306
307
* The `discover` command discovers all tests in a test suite.
307
308
* The `run` command runs only impacted tests and a new command.
@@ -518,7 +519,7 @@ The following options are available to be defined in the options map in config:
518
519
519
520
| `adaptive-testing`
520
521
|false
521
-
|Enables the adaptive testing features, such as test impact analysis.
522
+
|Enables the Smarter Testing features, such as test impact analysis.
522
523
523
524
| `full-test-run-paths`
524
525
a|
@@ -573,9 +574,12 @@ a| * `all` selects and runs all discovered tests, used to run the full test suit
573
574
|===
574
575
--
575
576
576
-
== 3. Start using adaptive testing
577
+
== 3. Start using Smarter Testing
577
578
578
-
Now the test suite is set up, test selection is working and the test analysis is up to date with the latest changes from the feature branch that ran the first test analysis.
579
+
Now the test suite is set up:
580
+
581
+
* Test selection is working.
582
+
* The test analysis is up to date with the latest changes from the feature branch that ran the first test analysis.
579
583
580
584
*Action Items*
581
585
@@ -683,7 +687,7 @@ options:
683
687
684
688
== Limitations
685
689
686
-
The adaptive testing feature has some limitations to consider:
690
+
The Smarter Testing feature has some limitations to consider:
687
691
688
692
*Initial setup period*:: Test impact analysis requires an initial analysis run on all tests before intelligent selection can begin. This first analysis run will be slower than normal test execution.
689
693
@@ -703,9 +707,9 @@ The adaptive testing feature has some limitations to consider:
703
707
704
708
*Debugging steps:*
705
709
706
-
. Check that all test files are discovered with the discover command
707
-
. Verify parallelism is set correctly in your config.yml
708
-
. Look for timing data in previous test runs
710
+
. Check that all test files are discovered with the discover command.
711
+
. Verify parallelism is set correctly in your config.yml.
712
+
. Look for timing data in previous test runs.
709
713
. Ensure test results are being stored with `store_test_results`.
710
714
711
715
=== Test impact analysis not selecting expected tests
@@ -716,10 +720,10 @@ The adaptive testing feature has some limitations to consider:
716
720
717
721
*Debugging steps:*
718
722
719
-
. Verify analysis has run successfully on your configured branch(es)
720
-
. Check that coverage data is being generated correctly
721
-
. Review the full-test-run-paths configuration - changes to these paths trigger full test runs
722
-
. Confirm the analysis command is producing valid LCOV output
723
+
. Verify analysis has run successfully on your configured branch(es).
724
+
. Check that coverage data is being generated correctly.
725
+
. Review the full-test-run-paths configuration - changes to these paths trigger full test runs.
726
+
. Confirm the analysis command is producing valid LCOV output.
723
727
724
728
*When all tests run:* If no impact data exists or all tests are determined to be affected, the system runs all tests as a safety measure.
725
729
@@ -769,9 +773,9 @@ The frequency depends on your test execution speed and development pace:
769
773
770
774
*Consider re-running analysis:*
771
775
772
-
* After major refactoring or code restructuring
773
-
* When test selection seems inaccurate or outdated
774
-
* After adding significant new code or tests
776
+
* After major refactoring or code restructuring.
777
+
* When test selection seems inaccurate or outdated.
778
+
* After adding significant new code or tests.
775
779
776
780
*Remember:* You can customize which branches run analysis through your CircleCI configuration - it does not have to be limited to the main branch.
777
781
@@ -807,7 +811,7 @@ When test selection determines that no existing tests are affected by your chang
807
811
808
812
*Best practice:* Include relevant paths in `full-test-run-paths` to explicitly trigger full test runs for infrastructure changes.
809
813
810
-
=== How do I know if adaptive testing is working?
814
+
=== How do I know if Smarter Testing is working?
811
815
812
816
Look for these indicators in your CircleCI build output:
813
817
@@ -846,7 +850,7 @@ See the <<run-higher-parallelism-on-the-analysis-branch,Run higher parallelism o
846
850
[#baseline-coverage]
847
851
=== Why are there so many files impacting a test?
848
852
849
-
If you see many files impacting each test during analysis (for example, "...found 150 files impacting test..."), this may be caused by shared setup code like global imports or framework initialization being included in coverage.
853
+
If you see many files impacting each test during analysis, for example, `...found 150 files impacting test...`, this may be caused by shared setup code like global imports or framework initialization being included in coverage.
850
854
851
855
This extraneous coverage can be excluded by providing an `analysis-baseline` command to compute the code covered during startup that isn't directly exercised by test code. We call this "baseline coverage data".
852
856
@@ -883,7 +887,7 @@ The `analysis-baseline` command will be run just before running analysis. The co
883
887
884
888
=== What test frameworks are supported?
885
889
886
-
Adaptive testing is runner-agnostic. We provide default configurations for the following test frameworks:
890
+
Smarter Testing is runner-agnostic. We provide default configurations for the following test frameworks:
0 commit comments