Skip to content

Support memory profiling in benchmarks #16720

@2010YOUY01

Description

@2010YOUY01

Is your feature request related to a problem or challenge?

Currently in DataFusion's benchmark: it only measures execution time. It would be helpful to also measure the total memory used.

Describe the solution you'd like

Approach 1: Implement memory tracker inside operators, update max RSS for each poll, and collect the stats when the query finished.
Comet has implemented something similar apache/datafusion-comet#1702

Approach 2: Profile memory stats externally in the benchmark script

I'm now slightly leaning toward approach 2, as it doesn't add complexity to the core.

cc @ding-young

Describe alternatives you've considered

No response

Additional context

No response

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions