111 create yaml template for problem submission #121
Conversation
Dvermetten
left a comment
Dvermetten
left a comment
There was a problem hiding this comment.
Maybe it would be useful to have some sort of comments to describe what the fields are / what would be valid input, specifically for the cases where there is a difference between quoted and non-quoted input. Example:
variables: # information about the input variables
types: continuous # can be one of (continuous, integer, binary, mixed)
conditional: 'no' # whether there are conditional dependencies between variables, 'yes' or 'no'
dimensionality: scalable # number of input variables, either as a number (in quotes) or scalable
CIGbalance
left a comment
CIGbalance
left a comment
There was a problem hiding this comment.
Thanks for this! The comments are mostly easy fixes or not that serious.
But I am mostly wondering of how this is supposed to be used? It is called a template, but it has BBOB-specific entries? I think this increases the risk of biasing the results.
I think I would suggest doing a template with a lot more comments (like @Dvermetten suggested) and then we can use this as real data as well as link to it as an example?
| suite/generator/single: suite | ||
| objectives: | ||
| number: '1' | ||
| types: single |
There was a problem hiding this comment.
I would have assumed "types" is continuous etc? What does "single" mean?
| - name: | ||
| short: BBOB | ||
| full: Real-Parameter Black-Box Optimization Benchmarking | ||
| suite/generator/single: suite |
There was a problem hiding this comment.
I still think it would be best to enter problems separately, but I understand the necessity of streamlining this process. So if we want to allow entering suites, maybe we have different templates? Because some values might only make sense for a suite? Otherwise, for a suite, a lot of the input would be "varies" for number of objectives, variables, constraints, dynamic, noise, etc?
| full: Real-Parameter Black-Box Optimization Benchmarking | ||
| suite/generator/single: suite | ||
| objectives: | ||
| number: '1' |
There was a problem hiding this comment.
Could this also be scalable? And why is this a string?
| conditional: 'no' | ||
| dimensionality: scalable | ||
| constraints: | ||
| present: 'no' |
There was a problem hiding this comment.
This is confusing to enter, because they have boundary/box: yes.
I would suggest removing present, it should be clear from the rest of the values.
| implementations: | ||
| - name: COCO | ||
| link: https://github.com/numbbo/coco | ||
| languges: 'C, Python' |
There was a problem hiding this comment.
Technically, COCO has way more interfaces.
Also typo in languages.
| - name: COCO | ||
| link: https://github.com/numbbo/coco | ||
| languges: 'C, Python' | ||
| evaluation time: 'less than a second' |
There was a problem hiding this comment.
This is going to be wild to analyse if we don't give some structure
| link: https://github.com/numbbo/coco | ||
| languges: 'C, Python' | ||
| evaluation time: 'less than a second' | ||
| specific requirements: 'no' |
There was a problem hiding this comment.
what would be an example of a specific requirement?
| evaluation time: 'less than a second' | ||
| specific requirements: 'no' | ||
| source: | ||
| real-world: |
There was a problem hiding this comment.
What are degree and open/closed?
| textual description: | ||
| general info: '' | ||
| motivation: 'evaluate algorithm performance for typical difficulties that occur in continuous problems' | ||
| challenage/key characteristics: '' |
| @@ -0,0 +1,52 @@ | |||
| # Please enter the relevant information. | |||
| # Fields that are not relevant can be left empty. | |||
There was a problem hiding this comment.
Can I not just remove the field?
Draft yaml template for review