-
Notifications
You must be signed in to change notification settings - Fork 1.6k
Open
Labels
enhancementNew feature or requestNew feature or request
Description
Is your feature request related to a problem or challenge?
Currently in DataFusion's benchmark: it only measures execution time. It would be helpful to also measure the total memory used.
Describe the solution you'd like
Approach 1: Implement memory tracker inside operators, update max RSS for each poll, and collect the stats when the query finished.
Comet has implemented something similar apache/datafusion-comet#1702
Approach 2: Profile memory stats externally in the benchmark script
I'm now slightly leaning toward approach 2, as it doesn't add complexity to the core.
cc @ding-young
Describe alternatives you've considered
No response
Additional context
No response
ding-young
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request