[TRTC-1922] [feat] Recipe db and its ingestion implementation for OPS #8990

venkywonka · 2025-11-07T00:35:37Z

Description

Seed the recipe db with a tinyllama yml
Add --recipe flag in trtllm-bench and trtllm-serve that accepts the recipe
perf test changes to pick up recipe db and run in the form of a separate test list (tests/integration/test_lists/qa/llm_perf_recipe_db.yml) ; pipeclean with tinyllama-recipe
Add Pydantic model validation for recipe db and associated tests

@coderabbitai summary

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...

Provide a user friendly way for developers to interact with a Jenkins server.

Run /bot [-h|--help] to print this help message.

See details below for each supported subcommand.

run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental)]

Launch build/test pipelines. All previously running jobs will be killed.

--reuse-test (optional)pipeline-id (OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.

--disable-reuse-test (OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.

--disable-fail-fast (OPTIONAL) : Disable fail fast on build/tests/infra failures.

--skip-test (OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.

--stage-list "A10-PyTorch-1, xxx" (OPTIONAL) : Only run the specified test stages. Examples: "A10-PyTorch-1, xxx". Note: Does NOT update GitHub check status.

--gpu-type "A30, H100_PCIe" (OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.

--test-backend "pytorch, cpp" (OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does NOT update GitHub pipeline status.

--only-multi-gpu-test (OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.

--disable-multi-gpu-test (OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.

--add-multi-gpu-test (OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.

--post-merge (OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.

--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" (OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx".

--detailed-log (OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.

--debug (OPTIONAL) : Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in the stage-list parameter to access the appropriate container environment. Note: Does NOT update GitHub check status.

For guidance on mapping tests to stage names, see docs/source/reference/ci-overview.md
and the scripts/test_to_stage_mapping.py helper.

kill

kill

Kill all running builds associated with pull request.

skip

skip --comment COMMENT

Skip testing for latest commit on pull request. --comment "Reason for skipping build/test" is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

reuse-pipeline

reuse-pipeline

Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

…ested) Enables generating optimized TensorRT-LLM configurations from scenario constraints using profile-based logic. Supports dsr1-fp4, dsr1-fp8, and gptoss-fp4 profiles with validated example recipes. Note: This implementation has not been tested yet. Signed-off-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com>

Signed-off-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com>

- Simplify configure.py by removing redundant recipe loading logic - Fix recipe database initialization in db/__init__.py - Update matcher and profiles for improved recipe handling - Integrate recipe system with performance test infrastructure Signed-off-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com>

The README documented a non-existent --recipe flag and used outdated examples showing config.yaml output. Updated to reflect actual CLI behavior: trtllm-configure generates recipe files (scenario + env + config) from scenario parameters only, not from existing recipes. Signed-off-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com>

…trtllm-configure This commit enforces a standardized recipe schema and simplifies trtllm-configure to perform exact matching only, removing dynamic recipe generation. Schema Changes: - Rename 'config' → 'llm_api_config' in all recipe YAML files - Update recipe detection in trtllm-serve and trtllm-bench to use 'llm_api_config' - Update README examples to use new key name trtllm-configure Simplification: - Remove dynamic recipe generation using profiles - Implement exact matching only against tensorrt_llm/recipes/db/ - Add find_all_matching_recipes() to detect multiple matches - Return clear errors for no match or ambiguous (multiple) matches - Remove --profile CLI option (no longer needed) - Update help text and examples to reflect exact matching behavior Validation Changes: - Remove validate_llm_api_config() calls from configure/serve/bench - Comment out validation function pending PR NVIDIA#8331 - PR NVIDIA#8331 standardizes LlmArgs with Pydantic, after which validation will happen automatically when LlmArgs(**kwargs) is instantiated - Add TODO comments referencing PR NVIDIA#8331 Documentation Updates: - Remove "Profiles" section from README (no longer auto-generated) - Remove "Adding Custom Profiles" section - Update "Quick Start" to reflect exact matching behavior - Clarify that trtllm-configure retrieves (not generates) recipes Signed-off-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com>

This commit enables trtllm-bench to parse and apply recipe YAML files that contain scenario parameters, environment variables, and LLM API configuration in a unified format. Key changes: - Add scenario.py utility to extract and merge scenario parameters - Modify throughput.py to detect recipe format and create temp YAML with only llm_api_config section to pass to LLM constructor - Fix dataset field name from output_len to output_tokens in scenario.py - Add tinyllama-simple.yaml test recipe demonstrating recipe usage Recipe format structure: - scenario: test parameters (ISL/OSL/concurrency/num_requests) - env: environment variables to set - llm_api_config: LLM constructor arguments (KV cache, CUDA graphs, etc) With this change, users can now run: trtllm-bench --model <model> throughput \ --extra_llm_api_options <recipe.yaml> Instead of specifying multiple CLI flags for ISL/OSL/concurrency/etc. The recipe format simplifies configuration and enables reusable test configurations. Signed-off-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com>

…cation Add recipe system support to low_latency benchmark command and extract common llm_api_config processing logic to reduce code duplication. Changes: - Add prepare_llm_api_config_for_recipe() utility to scenario.py that extracts llm_api_config section from recipe YAML and creates temp file - Update low_latency.py to use shared utility for recipe processing - Refactor throughput.py to use shared utility instead of inline tempfile logic - Eliminates ~30 lines of duplicated code between benchmark files Both throughput and latency commands now support recipe format with auto-generated datasets and unified behavior. Signed-off-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com>

- Updated `merge_params_with_priority` function to reflect new parameter names in examples. - Modified `generate_bench_command` to include model name and provide detailed command templates for throughput, latency, and build benchmarks. - Renamed validation exceptions for clarity, changing `ValidationError` to `ScenarioValidationError` and `ValidationWarning` to `ScenarioValidationWarning`. - Added a new TinyLlama test recipe for streamlined testing and dataset generation. - Removed outdated recipe files for DeepSeek and GPT-OSS to clean up the repository. These changes enhance usability and maintainability of the benchmarking and recipe systems. Signed-off-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com>

…commands Added process_recipe_scenario() helper in scenario.py to eliminate code duplication between throughput.py and low_latency.py. This consolidates recipe scenario extraction, parameter merging, and dataset auto-generation into a single reusable function. Changes: - Added process_recipe_scenario() to tensorrt_llm/bench/utils/scenario.py - Refactored throughput.py to use new helper (40 lines -> 15 lines) - Refactored low_latency.py to use new helper (40 lines -> 15 lines) - Eliminated ~80 lines of duplicated code - Maintained 100% backward compatibility Tested with e2e recipe perf tests - all passing. Signed-off-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com>

Signed-off-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com>

…i_options Signed-off-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com>

Signed-off-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com>

anish-shanbhag

Great progress here @venkywonka! This is a solid start to the recipe DB

anish-shanbhag · 2025-11-07T18:24:00Z

tensorrt_llm/bench/benchmark/low_latency.py

-    "Path to a YAML file that overwrites the parameters specified by trtllm-bench."
-)
+    "Path to a YAML file that overwrites the parameters specified by trtllm-bench. "
+    "(Deprecated: Use --recipe instead for full scenario support)")


Should we mark this deprecated immediately? I think there would still be users who want to use the LLM API options directly

I had a similar concern -- a lot of folks use this option right now. I think the plan right now is to support this in parallel.

anish-shanbhag · 2025-11-07T18:28:19Z

tensorrt_llm/bench/utils/scenario.py

+        recipe = RecipeConfig(**loaded_data)
+        return recipe.scenario.model_dump()
+
+    except (FileNotFoundError, yaml.YAMLError, KeyError, ValidationError):


I think we should let the error bubble up and cause a failure here instead of silently returning None so that it's obvious to the user that something went wrong

agreed! thanks for the catch

anish-shanbhag · 2025-11-07T18:30:34Z

tensorrt_llm/bench/utils/scenario.py

+    return merged
+
+
+def validate_scenario_params(scenario: Dict[str, Any]) -> None:


Is a separate function necessary for this? Maybe we can directly call ScenarioConfig(**config) instead?

Yeah -- this can be handled at the top level just by instantiating it.

yea great catch, completely agree, this is an artifact of ai-based refactoring (the pydantic refactor was layered on top of some existing POC code lol). will run another cleanup pass

anish-shanbhag · 2025-11-07T18:32:57Z

tensorrt_llm/bench/utils/scenario.py

+        with open(extra_llm_api_options_path, "r") as f:
+            full_recipe = yaml.safe_load(f)
+
+        # Extract only the llm_api_options section


Can we immediately parse into a scenario using ScenarioConfig(**full_recipe) and then use scenario.llm_api_options?

agreed, cleaning that up 🙏🏾

anish-shanbhag · 2025-11-07T18:34:09Z

tensorrt_llm/bench/utils/scenario.py

+        # Extract only the llm_api_options section
+        llm_api_options_only = full_recipe.get("llm_api_options", {})
+
+        # Create temporary file with only llm_api_options


Maybe it would be cleaner to create and pass around an instance of the LlmArgs object instead of creating a temporary LLM API options file, since downstream usage could use the file in unexpected ways

anish-shanbhag · 2025-11-07T18:45:19Z

tensorrt_llm/recipes/schema.py

+    Defines the target workload characteristics for performance testing.
+    """
+
+    model_config = {"extra": "allow"}  # Allow metadata fields like gpu, profile


nit: can we use ConfigDict?

Also - I'm not sure if the comment is 100% accurate since extra="allow" means that fields that are not defined at all within the pydantic model are allowed and ignored (which I do agree we want for now). Whereas e.g. gpu is defined explicitly as an optional field

anish-shanbhag · 2025-11-07T18:47:39Z

tensorrt_llm/recipes/schema.py

+    isl_stdev: int = Field(
+        default=0, ge=0, description="Input sequence length standard deviation (0 = exact)"
+    )
+    osl_stdev: int = Field(
+        default=0, ge=0, description="Output sequence length standard deviation (0 = exact)"
+    )
+    num_requests: int = Field(


Personally I think we should try to avoid mixing in benchmark-related params into the Scenario definition. For example num_requests is very specific to a specific benchmark test and not really related to an inference scenario. I guess this definitely complicates how we integrate with trtllm-bench, so we might have to think more about how to organize this

anish-shanbhag · 2025-11-07T18:49:01Z

tensorrt_llm/recipes/schema.py

+    )
+
+    # Metadata (optional, not validated beyond type)
+    gpu: Optional[str] = Field(default=None, description="GPU type metadata (e.g., 'H100', 'A100')")


Should we start a GPU / GPUType enum for this? How does TRTLLM handle GPU names elsewhere?

yea - i don't think our current POC will be doing any aggressive validation of GPU type obtained from recipe with runtime HW... i figure it would be more useful during the querying/retreival phase

but yea I agree, it'd be more concrete to have a gpu enum let me see if there's already some reusable code anywhere in trtllm doing that

anish-shanbhag · 2025-11-07T18:49:52Z

tensorrt_llm/recipes/schema.py

+    llm_api_options: Dict[str, Any] = Field(
+        default_factory=dict, description="LLM API configuration"
+    )
+    overrides: Optional[Dict[str, Any]] = Field(


I'm probably missing something, but it seems like this field is unused

Same question -- what is this used for?

anish-shanbhag · 2025-11-07T18:52:06Z

tensorrt_llm/bench/utils/scenario.py

@@ -0,0 +1,320 @@
+"""Utilities for extracting and processing recipe scenario parameters.


General q about this file - it seems like some of this logic isn't necessarily specific to trtllm-bench, so should we move some of it to the recipe DB module?

FrankD412 · 2025-11-07T19:21:29Z

tensorrt_llm/recipes/db/tinyllama-test.yaml

+  # Optional: Dataset generation parameters.
+  # This is useful for trtllm-bench to auto-generate dataset, so one can just specify this recipe
+  # to trtllm-bench without prior steps.
+  isl_stdev: 0       # Input sequence length standard deviation (0 = exact)


Is this just for testing? I'm not sure we want this here -- are you thinking that the standard deviation would have a bearing on the recipe we choose?

Same for num_requests

I am aware I'm probably wrongly trying to conflate benchmark params with scenario params - and these dataset gen params should have no bearing on the recipe we choose - maybe ill move it out as this is a problem of the client trying to autogenerate a synthetic dataset...

if we are moving this out, then the user or client calling trtllm-bench will have the burden of generating a synthetic dataset - which i think is okay.

FrankD412 · 2025-11-07T19:39:39Z

tensorrt_llm/bench/benchmark/low_latency.py

+    # Priority: --recipe > --extra_llm_api_options
+    recipe_path = params.get("recipe", None)
+    extra_llm_api_options_path = params.get("extra_llm_api_options", None)
+    config_path = recipe_path if recipe_path else extra_llm_api_options_path


I have concerns about this from a UX perspective. I don't think this is a matter of priority more so that they shouldn't be specified at the same time. See my comment above.

FrankD412 · 2025-11-07T19:40:12Z

tensorrt_llm/bench/benchmark/low_latency.py

@@ -45,13 +47,23 @@
    default=None,
    help="Path to a serialized TRT-LLM engine.",
 )
+@optgroup.option(


If we make a new option group for these, we can classify them as mutually exclusive. I don't think we should allow both to be specified.

I originally thought since recipe contains more information, we allow and prioritize the information in extra_llm_api_options when such a conflict arises - and print a warning clearly telling which source we're prioritizing.

but i agree, its a lot easier to reduce the UX space here by making it mutually exclusive

FrankD412 · 2025-11-07T19:41:45Z

tensorrt_llm/bench/benchmark/throughput.py

@@ -302,6 +314,20 @@ def throughput_command(
    options: GeneralExecSettings = get_general_cli_options(params, bench_env)
    tokenizer = initialize_tokenizer(options.checkpoint_path)

+    # Process recipe scenario if present
+    cli_defaults = {


Couldn't we populate this with what we get from the CLI?

do you mean the cli defaults or cli args expicitly specified by the user?
in that case, the recipe will override the cli args. but we want the cli args provided by user to override the recipe right?
say the user does trtllm-bench --model <model> throughput --target_input_len=1024 --recipe recipe.yaml

basically there are 3 information sources for a given scenario arg (say, isl):

the default if neither CLI nor recipe specifies it

the value user specifies via CLI

the value user indirectly specifies via recipe

Signed-off-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com>

FrankD412 · 2025-11-07T19:52:55Z

tensorrt_llm/bench/utils/scenario.py

+
+        # Parse and validate using Pydantic schema
+        recipe = RecipeConfig(**loaded_data)
+        return recipe.scenario.model_dump()


I agree with @anish-shanbhag below about the exception. This should fail outright if we pass in a path that doesn't exist or if the validation fails. I wouldn't put a try/except here, instead let the higher level caller sort it out. You can add to the docstring the possible errors this can return for developer information.

My question on this line -- is there a reason why we're converting this to a dictionary. I feel like this would be more generally useful if we let the caller decide what to do with it.

I agree, will refactor that. you mean its better to just return the ScenarioConfig object instead of dictifying it right?

venkywonka added 11 commits November 4, 2025 22:00

Scratch work: refactor recipes and add test integration (untested)

1d40bfe

Signed-off-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com>

remove trtllm-configure code to simplify this pr

e0c2b45

Signed-off-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com>

Add --recipe flag to trtllm-bench and rename llm_api_config to llm_ap…

2320616

…i_options Signed-off-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com>

venkywonka self-assigned this Nov 7, 2025

Add Pydantic schema validation for recipes

e65b540

Signed-off-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com>

venkywonka requested review from FrankD412, QiJune, Superjomn, anish-shanbhag, atrifex and juney-nvidia November 7, 2025 01:02

juney-nvidia requested a review from kaiyux November 7, 2025 01:45

anish-shanbhag reviewed Nov 7, 2025

View reviewed changes

FrankD412 reviewed Nov 7, 2025

View reviewed changes

cleanup, add tests, update pydantic and enhance logging

71208df

Signed-off-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com>

FrankD412 reviewed Nov 7, 2025

View reviewed changes

		return merged


		def validate_scenario_params(scenario: Dict[str, Any]) -> None:

		@@ -0,0 +1,320 @@
		"""Utilities for extracting and processing recipe scenario parameters.

[TRTC-1922] [feat] Recipe db and its ingestion implementation for OPS #8990

Are you sure you want to change the base?

[TRTC-1922] [feat] Recipe db and its ingestion implementation for OPS #8990

Uh oh!

Conversation

venkywonka commented Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Description

Test Coverage

PR Checklist

GitHub Bot Help

kill

skip

reuse-pipeline

Uh oh!

anish-shanbhag left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

venkywonka Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

venkywonka Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

venkywonka commented Nov 7, 2025 •

edited

Loading

venkywonka Nov 7, 2025 •

edited

Loading

venkywonka Nov 7, 2025 •

edited

Loading