Parameters assigned to be fit (with `parameterize`) without corresponding training data result in somewhat esoteric error about gradient keys

[lilyminium]

Describe the bug

A fit recently failed on an IndexError, likely because parameters were assigned to be fit without corresponding training data. We (=Julianne) figured out the cause and how to get around it, but ideally this would not cause an error in Evaluator.

To Reproduce

A reproducing example is here: https://github.com/JHoeflich1/Co_optimization_refit/tree/784d198996a38e9d2e6aa8fde9dc546c0241512b

Output

A full log is here but also pasting below for posterity:

Traceback (most recent call last):
  File "/projects/juho8819/software/anaconda/envs/co-optimization-py3.10/lib/python3.11/site-packages/openff/evaluator/server/server.py", line 697, in _handle_connections
    self._handle_stream(connection, connection.getpeername())
  File "/projects/juho8819/software/anaconda/envs/co-optimization-py3.10/lib/python3.11/site-packages/openff/evaluator/server/server.py", line 678, in _handle_stream
    self._handle_job_submission(connection, address, message_length)
  File "/projects/juho8819/software/anaconda/envs/co-optimization-py3.10/lib/python3.11/site-packages/openff/evaluator/server/server.py", line 617, in _handle_job_submission
    self._launch_batch(batch)
  File "/projects/juho8819/software/anaconda/envs/co-optimization-py3.10/lib/python3.11/site-packages/openff/evaluator/server/server.py", line 541, in _launch_batch
    current_layer.schedule_calculation(
  File "/projects/juho8819/software/anaconda/envs/co-optimization-py3.10/lib/python3.11/site-packages/openff/evaluator/layers/layers.py", line 416, in schedule_calculation
    futures = cls._schedule_calculation(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/projects/juho8819/software/anaconda/envs/co-optimization-py3.10/lib/python3.11/site-packages/openff/evaluator/layers/workflow.py", line 229, in _schedule_calculation
    workflow_graph, provenance = cls._build_workflow_graph(
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/projects/juho8819/software/anaconda/envs/co-optimization-py3.10/lib/python3.11/site-packages/openff/evaluator/layers/workflow.py", line 135, in _build_workflow_graph
    global_metadata = cls._get_workflow_metadata(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/projects/juho8819/software/anaconda/envs/co-optimization-py3.10/lib/python3.11/site-packages/openff/evaluator/layers/workflow.py", line 73, in _get_workflow_metadata
    global_metadata = Workflow.generate_default_metadata(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/projects/juho8819/software/anaconda/envs/co-optimization-py3.10/lib/python3.11/site-packages/openff/evaluator/workflow/workflow.py", line 697, in generate_default_metadata
    relevant_gradient_keys = Workflow._find_relevant_gradient_keys(
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/projects/juho8819/software/anaconda/envs/co-optimization-py3.10/lib/python3.11/site-packages/openff/evaluator/workflow/workflow.py", line 609, in _find_relevant_gradient_keys
    if isinstance(labelled_parameters[0], list):
                  ~~~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range

Could that line maybe be written more carefully to first check the length of labelled_parameters before the type?

Computing environment (please complete the following information):

• Operating system
• Output of running conda list

Additional context

[JHoeflich1]

I am running this on the Colorado University cluster:
NAME="Red Hat Enterprise Linux"
VERSION="8.10 (Ootpa)"

I am not sure why, but this issue does not show up when I use Force Balance to only train non bonded parameters. This issue is only showing up when I try to co-optimize both vdW and valence. I had to manually edit my force_field_tagged.offxml and delete the VDW parameterize tags that did not correspond to [#1:1]-[#6X4], [#1:1]-[#6X3], [#6:1], [#6X4:1]

env.txt