feat: Improve Acquisition Function optimization in discrete and mixed search spaces

[Sohambasu07]

Issue

• Optimization of acquisition function in NePS BO was rather slow and capped at 30 combinations for search spaces with categorical hyperparameters.

Additions and Deletions

• Removed maximum_allowed_categorical_combinations which previously set a cap on number of allowed categorical combinations in the search space.
• Removed optimize_acqf_mixed()
• Added WrappedAcquisition module in neps.optimizers.acquisition.wrapped_acquisition.
• Added optimize_acqf_discrete_local_search() for purely categorical and mixed search spaces.

How things work now

1. Purely categorical search spaces:

   • Replace optimize_acqf_mixed() with optimize_discrete_local_search() which scales better with increasing number of categorical dimensions and combinations.

2. Mixed search spaces:

   • For mixed search spaces, we first remove the need to generate all possible categorical combinations, which further reduces a lot of computational overhead.
   • Instead we randomly select one combination per iteration of the optimization.
   • We then perform a sequential 2-step optimization:
     • First, keeping the randomly selected categorical combination fixed, we optimize over the continuous features using optimize_acqf().
     • Next, we wrap the acquisition function inside WrappedAcquisition to keep the best seen values of the continuous features fixed and optimize only over the categoricals using optimize_acqf_discrete_local_search(). Finally, we merge the best seen values of both the numerical and categorical features into a single tensor, perform one forward pass over the acquisition function and return the candidate and the score.

Tests

• Tested the optimizers - PriMO, BO and PriorbandBO on the JAHSBench CIFAR10 task.