Extract machines corresponding to meta models from a train TunedModel machine. #225
Description
Hi there,
Is there any way to obtain the machine of the best meta model that a TunedModel machine has trained through hyperparameter tuning. If my understanding is correct, after evaluating all the meta models through some resampling scheme (ie. some of the training data is left out to evaluate the meta models), the parameters leading to the best measure will then be used to train a final machine of the base model kind but using the tuned hyperparameters and importantly - all the training data without any resampling. I can see how to obtain the final model/machine (i.e the result of machine(model, X, Y) after running fit!() on it). However I would like to obtain the machine of the best meta model - i.e the one trained with a subset of training (due to resampling).
Alternatively, having a index partition object which details which row indices were used as part of the hyperparemeter meta model's training and what indices were used to evaluate them.
For a concrete example, the following code should illuminate what I am trying to acheive:
base_model = XGBoostRegressor(objective="reg:tweedie")
r1 = MLJ.range(base_model, :eta, lower=0.1, upper=0.9)
r2 = MLJ.range(base_model, :tweedie_variance_power, lower=1.05, upper=1.95)
tuned_xgb_model = TunedModel(
model=base_model,
ranges=[r1, r2],
tuning=RandomSearch(rng=123),
resampling=CV(nfolds=10),
measure=RSquared(),
n=32,
acceleration=CPUThreads(),
)
train_idx, test_idx = partition(eachindex(Y), 0.7)
tuned_xgb_machine = machine(tuned_xgb_model, X, Y)
fit!(tuned_xgb_machine, rows=train_idx)Basically I want to save the specific machine which was obtained as the best XGBoost meta model but found using training data which is 9/10 (from the cross validation) of the 70% (train/test split) of the total dataset X.