[DLRMv3] Reference Implementation #2410
Conversation
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
|
@tanvi-mlcommons @pgmpablo157321 to take a look (can ignore the whole HSTU submodules) and help resolve the CLA issues for Linjian. |
|
Hi @mrmhodak , could you help review this PR? All comments of @nvzhihanj have been addressed |
recommendation/dlrm_v3/README.md
Outdated
| ``` | ||
| python accuracy.py --path path/to/mlperf_log_accuracy.json | ||
| ``` | ||
| We use normalized entropy (NE), accuracy, and AUC as the metrics to evaluate the model quality. The accuracy for the reference implementation evaluated on 34,996 requests across 10 inference timestamps are listed below: |
There was a problem hiding this comment.
Are we using all 3 scores for 99%, or just the AUC? (If it's just AUC please help point it out)
Also it seems like the loadgen/mlperf.conf is missing (for latency threshold and sample count)
There was a problem hiding this comment.
Yes we need all 3 scores to be 99% close, I will clarify in the README. I will also add latency threshold to mlperf.conf.
Regarding sample count, could you explain when will we use it? I'm a bit confused as I think performance_sample_count_override
Lines 6 to 30 in 8999c4d
My understanding is that submitters are allowed to change the sample count, and as long as the minimum duration requirement for the benchmark is met, this should be acceptable.
There was a problem hiding this comment.
updated the README and added latency to loadgen/mlperf.conf
|
Hi @pgmpablo157321, can we get this merged? Thanks! |
|
@mrmhodak @hanyunfan FYI to merge |
| bert.Server.target_latency = 130 | ||
| dlrm.Server.target_latency = 60 | ||
| dlrm-v2.Server.target_latency = 60 | ||
| dlrm-v3.Server.target_latency = 80 |
There was a problem hiding this comment.
According to the submission checker it is 60?
There was a problem hiding this comment.
Hi @cfRod , the target latency for dlrmv3 is 80ms. I think we need to update the submission checker
|
Add @arjunsuresh to help take a look at those MLC failing checks. 
|
7 participants
This PR introduces the reference implementation for DLDRMv3.
Instructions for running the benchmark are provided in the README. Please note: The dataset setup is still pending. The README will be updated with download instructions once the dataset becomes available.