Skip to content

Conversation

@mgarrard
Copy link
Contributor

@mgarrard mgarrard commented Jan 6, 2026

Summary:
This updates target trial selection logic in the following ways:

  1. We only consider trials that either (a) have data for all opt config metrics or (b) have data for all metrics -- previously if a trial had data for some opt config metrics it passed the check, but this partial data setup causes issues downstream
  2. If there is no long run trial, we fallback to short run instead of not identifying a target trial
  3. filters out "stale" trials, ie trials that were completed over 10 days ago -- hitting this point would be pretty far down our priority list, but was an idea ItsMrLin initially had that i thought was really interesting
  4. lastly if no trials exist after stale is filtered out, use the stale ones anyway (necessary for benchmarking)
  5. if we still can't find anything, it will return none

Differential Revision:
D90089411

Privacy Context Container: L1307644

@meta-cla meta-cla bot added the CLA Signed Do not delete this pull request or issue due to inactivity. label Jan 6, 2026
@meta-codesync
Copy link

meta-codesync bot commented Jan 6, 2026

@mgarrard has exported this pull request. If you are a Meta employee, you can view the originating Diff in D90089411.

@codecov-commenter
Copy link

codecov-commenter commented Jan 6, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.72%. Comparing base (aa29dd8) to head (65a540f).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4742      +/-   ##
==========================================
- Coverage   96.72%   96.72%   -0.01%     
==========================================
  Files         582      582              
  Lines       60693    60745      +52     
==========================================
+ Hits        58708    58757      +49     
- Misses       1985     1988       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

mgarrard added a commit to mgarrard/Ax that referenced this pull request Jan 6, 2026
… for optimization config metrics (facebook#4742)

Summary:

This updates target trial selection logic in the following ways:
1. We only consider trials that either (a) have data for *all* opt config metrics or (b) have data for *all* metrics -- previously if a trial had data for some opt config metrics it passed the check, but this partial data setup causes issues downstream
2. If there is no long run trial, we fallback to short run instead of not identifying a target trial
3. filters out "stale" trials, ie trials that were completed over 10 days ago -- hitting this point would be pretty far down our priority list, but was an idea ItsMrLin initially had that i thought was really interesting
4. lastly if no trials exist after stale is filtered out, use the stale ones anyway (necessary for benchmarking)
5. if we still can't find anything, it will return none

Reviewed By: ItsMrLin

Differential Revision:
D90089411

Privacy Context Container: L1307644
mgarrard added a commit to mgarrard/Ax that referenced this pull request Jan 7, 2026
… for optimization config metrics (facebook#4742)

Summary:

This updates target trial selection logic in the following ways:
1. We only consider trials that either (a) have data for *all* opt config metrics or (b) have data for *all* metrics -- previously if a trial had data for some opt config metrics it passed the check, but this partial data setup causes issues downstream
2. If there is no long run trial, we fallback to short run instead of not identifying a target trial
3. filters out "stale" trials, ie trials that were completed over 10 days ago -- hitting this point would be pretty far down our priority list, but was an idea ItsMrLin initially had that i thought was really interesting
4. lastly if no trials exist after stale is filtered out, use the stale ones anyway (necessary for benchmarking)
5. if we still can't find anything, it will return none

Reviewed By: ItsMrLin

Differential Revision:
D90089411

Privacy Context Container: L1307644
mgarrard added a commit to mgarrard/Ax that referenced this pull request Jan 7, 2026
… for optimization config metrics (facebook#4742)

Summary:

This updates target trial selection logic in the following ways:
1. We only consider trials that either (a) have data for *all* opt config metrics or (b) have data for *all* metrics -- previously if a trial had data for some opt config metrics it passed the check, but this partial data setup causes issues downstream
2. If there is no long run trial, we fallback to short run instead of not identifying a target trial
3. filters out "stale" trials, ie trials that were completed over 10 days ago -- hitting this point would be pretty far down our priority list, but was an idea ItsMrLin initially had that i thought was really interesting
4. lastly if no trials exist after stale is filtered out, use the stale ones anyway (necessary for benchmarking)
5. if we still can't find anything, it will return none

Reviewed By: ItsMrLin

Differential Revision:
D90089411

Privacy Context Container: L1307644
… for optimization config metrics (facebook#4742)

Summary:

This updates target trial selection logic in the following ways:
1. We only consider trials that either (a) have data for *all* opt config metrics or (b) have data for *all* metrics -- previously if a trial had data for some opt config metrics it passed the check, but this partial data setup causes issues downstream
2. If there is no long run trial, we fallback to short run instead of not identifying a target trial
3. filters out "stale" trials, ie trials that were completed over 10 days ago -- hitting this point would be pretty far down our priority list, but was an idea ItsMrLin initially had that i thought was really interesting
4. lastly if no trials exist after stale is filtered out, use the stale ones anyway (necessary for benchmarking)
5. if we still can't find anything, it will return none

Reviewed By: ItsMrLin

Differential Revision:
D90089411

Privacy Context Container: L1307644
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed Do not delete this pull request or issue due to inactivity. fb-exported meta-exported

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants