Skip to content

Run multiple handles in parallel #233

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
michieldegezelle wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Run multiple handles in parallel #233

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
511d8c3
Run tests with status in parallel for multiple handles
michieldegezelle Nov 25, 2025
a5eff1e
Don't transform the handle string into an array in case of a non-stat…
michieldegezelle Nov 26, 2025
e9ee279
Bump version to correct number
michieldegezelle Dec 3, 2025
0345934
Review comments
michieldegezelle Dec 5, 2025
744946d
Add additional warning in case no handles are provided
michieldegezelle Dec 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 27 additions & 10 deletions bin/cli.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -451,15 +451,15 @@ program
.command("run-test")
.description("Run Liquid Tests for a reconciliation template from a YAML file")
.requiredOption("-f, --firm <firm-id>", "Specify the firm to be used", firmIdDefault)
.option("-h, --handle <handle>", "Specify the reconciliation to be used (mandatory)")
.option("-at, --account-template <name>", "Specify the account template to be used (mandatory)")
.option("-h, --handle <handle...>", "Specify one or more reconciliations to be used (mandatory)")
.option("-at, --account-template <name...>", "Specify one or more account templates to be used (mandatory)")
.option("-t, --test <test-name>", "Specify the name of the test to be run (optional)", "")
.option("--html-input", "Get a static html of the input-view of the template generated with the Liquid Test data (optional)", false)
.option("--html-preview", "Get a static html of the export-view of the template generated with the Liquid Test data (optional)", false)
.option("--preview-only", "Skip the checking of the results of the Liquid Test in case you only want to generate a preview template (optional)", false)
.option("--status", "Only return the status of the test runs as PASSED/FAILED (optional)", false)

.action((options) => {
.action(async (options) => {
if (!options.handle && !options.accountTemplate) {
consola.error("You need to specify either a reconciliation handle or an account template");
process.exit(1);
Expand All @@ -468,15 +468,32 @@ program
const templateType = options.handle ? "reconciliationText" : "accountTemplate";
const templateName = options.handle ? options.handle : options.accountTemplate;

if (!templateName || templateName.length === 0) {
consola.error("You need to provide at least one handle or account template name");
process.exit(1);
}

// Block multiple handles/templates without --status
if (templateName.length > 1 && !options.status) {
consola.error("Multiple handles/templates are only allowed when used with the --status flag");
process.exit(1);
}

if (options.status) {
liquidTestRunner.runTestsStatusOnly(options.firm, templateType, templateName, options.test);
} else {
if (options.previewOnly && !options.htmlInput && !options.htmlPreview) {
consola.info(`When using "--preview-only" you need to specify at least one of the following options: "--html-input", "--html-preview"`);
process.exit(1);
}
liquidTestRunner.runTestsWithOutput(options.firm, templateType, templateName, options.test, options.previewOnly, options.htmlInput, options.htmlPreview);
// Status mode: allow multiple, pass array of template names
await liquidTestRunner.runTestsStatusOnly(options.firm, templateType, templateName, options.test);
return;
}

// Non-status mode: always run a single template, pass string handle/name
const singleTemplateName = templateName[0];

if (options.previewOnly && !options.htmlInput && !options.htmlPreview) {
consola.info(`When using "--preview-only" you need to specify at least one of the following options: "--html-input", "--html-preview"`);
process.exit(1);
}

await liquidTestRunner.runTestsWithOutput(options.firm, templateType, singleTemplateName, options.test, options.previewOnly, options.htmlInput, options.htmlPreview);
});

// Create Liquid Test
Expand Down
60 changes: 42 additions & 18 deletions lib/liquidTestRunner.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -421,33 +421,57 @@ async function runTestsWithOutput(firmId, templateType, handle, testName = "", p

// RETURN (AND LOG) ONLY PASSED OR FAILED
// CAN BE USED BY GITHUB ACTIONS
async function runTestsStatusOnly(firmId, templateType, handle, testName = "") {
async function runTestsStatusOnly(firmId, templateType, handles, testName = "") {
Copy link

@coderabbitai coderabbitai bot Dec 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion | 🟠 Major

Add input validation for handles parameter.

The function now accepts handles as an array, but there's no validation to ensure it's actually an array or that it's not empty. If an invalid value is passed, it could cause runtime errors.

Apply this diff to add validation:

 async function runTestsStatusOnly(firmId, templateType, handles, testName = "") {
+  if (!Array.isArray(handles) || handles.length === 0) {
+    consola.error(`At least one handle must be provided`);
+    process.exit(1);
+  }
+
   if (templateType !== "reconciliationText" && templateType !== "accountTemplate") {
     consola.error(`Template type is missing or invalid`);
     process.exit(1);
   }
🤖 Prompt for AI Agents 
In lib/liquidTestRunner.js around line 424, the runTestsStatusOnly function
accepts a handles parameter but lacks validation; add a check at the top that
ensures handles is an Array and not empty (e.g., if (!Array.isArray(handles) ||
handles.length === 0) throw new TypeError("handles must be a non-empty array")
or return a rejected Promise), so invalid inputs are rejected early with a clear
error; update any callers or tests if they rely on implicit behavior.

if (templateType !== "reconciliationText" && templateType !== "accountTemplate") {
consola.error(`Template type is missing or invalid`);
process.exit(1);
}

let status = "FAILED";
const testResult = await runTests(firmId, templateType, handle, testName, false, "none");
const runSingleHandle = async (singleHandle) => {
let status = "FAILED";
const failedTestNames = [];
const testResult = await runTests(firmId, templateType, singleHandle, testName, false, "none");

if (!testResult) {
status = "PASSED";
consola.success(status);
return status;
}
if (!testResult) {
status = "PASSED";
} else {
const testRun = testResult?.testRun;

const testRun = testResult?.testRun;
if (testRun && testRun?.status === "completed") {
const errorsPresent = checkAllTestsErrorsPresent(testRun.tests);
if (errorsPresent === false) {
status = "PASSED";
} else {
// Extract failed test names
const testNames = Object.keys(testRun.tests).sort();
testNames.forEach((testName) => {
const testErrorsPresent = checkTestErrorsPresent(testName, testRun.tests);
if (testErrorsPresent) {
failedTestNames.push(testName);
}
});
}
}
}

if (testRun && testRun?.status === "completed") {
const errorsPresent = checkAllTestsErrorsPresent(testRun.tests);
if (errorsPresent === false) {
status = "PASSED";
consola.success(status);
return status;
if (status === "PASSED") {
consola.log(`${singleHandle}: ${status}`);
} else {
consola.log(`${singleHandle}: ${status}`);
// Display failed test names
failedTestNames.forEach((testName) => {
consola.log(` ${testName}: FAILED`);
});
}
}
consola.error(status);
return status;

return { handle: singleHandle, status, failedTestNames };
};
Comment on lines +430 to +468
Copy link

@coderabbitai coderabbitai bot Dec 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Risk: Promise.all will abort all handles if one throws an exception.

If any single handle encounters an unhandled exception (not a test failure, but an actual error like network timeout, invalid YAML, etc.), Promise.all will reject immediately and the remaining handles won't complete or report their results. This defeats the purpose of parallel execution for CI/CD pipelines where you want to see results for all handles even if some fail.

Consider wrapping each handle execution to catch exceptions:

   const runSingleHandle = async (singleHandle) => {
+    try {
       let status = "FAILED";
       const failedTestNames = [];
       const testResult = await runTests(firmId, templateType, singleHandle, testName, false, "none");
 
       if (!testResult) {
         status = "PASSED";
       } else {
         const testRun = testResult?.testRun;
 
         if (testRun && testRun?.status === "completed") {
           const errorsPresent = checkAllTestsErrorsPresent(testRun.tests);
           if (errorsPresent === false) {
             status = "PASSED";
           } else {
             // Extract failed test names
             const testNames = Object.keys(testRun.tests).sort();
             testNames.forEach((testName) => {
               const testErrorsPresent = checkTestErrorsPresent(testName, testRun.tests);
               if (testErrorsPresent) {
                 failedTestNames.push(testName);
               }
             });
           }
         }
       }
 
       if (status === "PASSED") {
         consola.log(`${singleHandle}: ${status}`);
       } else {
         consola.log(`${singleHandle}: ${status}`);
         // Display failed test names
         failedTestNames.forEach((testName) => {
           consola.log(`  ${testName}: FAILED`);
         });
       }
 
       return { handle: singleHandle, status, failedTestNames };
+    } catch (error) {
+      consola.error(`${singleHandle}: ERROR - ${error.message}`);
+      return { handle: singleHandle, status: "FAILED", failedTestNames: [], error: error.message };
+    }
   };
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
const runSingleHandle = async (singleHandle) => {
let status = "FAILED";
const failedTestNames = [];
const testResult = await runTests(firmId, templateType, singleHandle, testName, false, "none");
if (!testResult) {
status = "PASSED";
consola.success(status);
return status;
}
if (!testResult) {
status = "PASSED";
} else {
const testRun = testResult?.testRun;
const testRun = testResult?.testRun;
if (testRun && testRun?.status === "completed") {
const errorsPresent = checkAllTestsErrorsPresent(testRun.tests);
if (errorsPresent === false) {
status = "PASSED";
} else {
// Extract failed test names
const testNames = Object.keys(testRun.tests).sort();
testNames.forEach((testName) => {
const testErrorsPresent = checkTestErrorsPresent(testName, testRun.tests);
if (testErrorsPresent) {
failedTestNames.push(testName);
}
});
}
}
}
if (testRun && testRun?.status === "completed") {
const errorsPresent = checkAllTestsErrorsPresent(testRun.tests);
if (errorsPresent === false) {
status = "PASSED";
consola.success(status);
return status;
if (status === "PASSED") {
consola.log(`${singleHandle}: ${status}`);
} else {
consola.log(`${singleHandle}: ${status}`);
// Display failed test names
failedTestNames.forEach((testName) => {
consola.log(` ${testName}: FAILED`);
});
}
}
consola.error(status);
return status;
return { handle: singleHandle, status, failedTestNames };
};
const runSingleHandle = async (singleHandle) => {
try {
let status = "FAILED";
const failedTestNames = [];
const testResult = await runTests(firmId, templateType, singleHandle, testName, false, "none");
if (!testResult) {
status = "PASSED";
} else {
const testRun = testResult?.testRun;
if (testRun && testRun?.status === "completed") {
const errorsPresent = checkAllTestsErrorsPresent(testRun.tests);
if (errorsPresent === false) {
status = "PASSED";
} else {
// Extract failed test names
const testNames = Object.keys(testRun.tests).sort();
testNames.forEach((testName) => {
const testErrorsPresent = checkTestErrorsPresent(testName, testRun.tests);
if (testErrorsPresent) {
failedTestNames.push(testName);
}
});
}
}
}
if (status === "PASSED") {
consola.log(`${singleHandle}: ${status}`);
} else {
consola.log(`${singleHandle}: ${status}`);
// Display failed test names
failedTestNames.forEach((testName) => {
consola.log(` ${testName}: FAILED`);
});
}
return { handle: singleHandle, status, failedTestNames };
} catch (error) {
consola.error(`${singleHandle}: ERROR - ${error.message}`);
return { handle: singleHandle, status: "FAILED", failedTestNames: [], error: error.message };
}
};
🤖 Prompt for AI Agents 
In lib/liquidTestRunner.js around lines 430 to 468, the current runSingleHandle
flow can throw unhandled exceptions which will cause Promise.all to reject and
abort all parallel handle runs; wrap the body of runSingleHandle in a try/catch
(or ensure callers use Promise.allSettled) so any runtime error is caught, set a
clear status (e.g., "ERROR" or keep "FAILED"), capture the error message into
the returned object (or push into failedTestNames) and log it, and always return
{ handle, status, failedTestNames, error?: message } so the aggregator gets a
result for every handle even on exceptions.


const results = await Promise.all(handles.map(runSingleHandle));

const overallStatus = results.every((result) => result.status === "PASSED") ? "PASSED" : "FAILED";

return overallStatus;
}

module.exports = {
Expand Down
2 changes: 1 addition & 1 deletion package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "silverfin-cli",
"version": "1.47.0",
"version": "1.48.0",
"description": "Command line tool for Silverfin template development",
"main": "index.js",
"license": "MIT",
Expand Down
Loading