Helper tools for editing yaml files

[TeresiaOlsson]

I start a separate discussion on this topic to not discuss too many things at once in #2 .

The idea is:

• If we have a json schema describing the configuration we can use existing tools to help the users to write and debug the configuration.

I have tested:

1. Adding the schema to VS code
2. Using the MetaConfigurator

The MetaConfigurator is a tool developed at University of Stuttgart to help people write json/yaml files which follow some schema but also to edit or develop a schema.

Github:
https://github.com/MetaConfigurator/meta-configurator

Talk at deRSE25:
https://events.hifis.net/event/1741/contributions/13330/

Test case:

I made a very simple test case with some pydantic models.

import json
from pydantic import BaseModel, Field
from typing import Optional

class DeviceModel(BaseModel):
    """Device model class"""
    setpoint: str
    readback: str
    unit: str

class LinearConvModel(BaseModel):
    """Unit conversion model class"""
    factor: float
    offset: float

class QuadrupoleModel(BaseModel):
    """Quadrupole model class"""
    name: str = Field(description="Name of the magnet")
    hardware: Optional[DeviceModel] = Field(default=None, description="Hardware device associated with the magnet")
    unitconv: Optional[LinearConvModel] = Field(default=None, description="Unit conversion associated with the magnet")

I then extracted the schema for the quadrupole.

model_schema = QuadrupoleModel.model_json_schema()
with open("schema.json", "w") as f:
    json.dump(model_schema, f, indent=2)

It looked like this:

{
  "$defs": {
    "DeviceModel": {
      "description": "Device model class",
      "properties": {
        "setpoint": {
          "title": "Setpoint",
          "type": "string"
        },
        "readback": {
          "title": "Readback",
          "type": "string"
        },
        "unit": {
          "title": "Unit",
          "type": "string"
        }
      },
      "required": [
        "setpoint",
        "readback",
        "unit"
      ],
      "title": "DeviceModel",
      "type": "object"
    },
    "LinearConvModel": {
      "description": "Unit conversion model class",
      "properties": {
        "factor": {
          "title": "Factor",
          "type": "number"
        },
        "offset": {
          "title": "Offset",
          "type": "number"
        }
      },
      "required": [
        "factor",
        "offset"
      ],
      "title": "LinearConvModel",
      "type": "object"
    }
  },
  "description": "Quadrupole model class",
  "properties": {
    "name": {
      "description": "Name of the magnet",
      "title": "Name",
      "type": "string"
    },
    "hardware": {
      "anyOf": [
        {
          "$ref": "#/$defs/DeviceModel"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "Hardware device associated with the magnet"
    },
    "unitconv": {
      "anyOf": [
        {
          "$ref": "#/$defs/LinearConvModel"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "Unit conversion associated with the magnet"
    }
  },
  "required": [
    "name"
  ],
  "title": "QuadrupoleModel",
  "type": "object"
}

[TeresiaOlsson]

First I tested VS code:

• I installed the YAML extension and added a schema to the yaml file by putting # yaml-language-server: $schema=schema.json at the top of the file. I think there is also some global way to do it but I didn't test that yet.

Then problem: VS code doesn't understand the difference between writing a model and a schema. So when it suggests what to add it also suggests fields that are only descriptions in the pydantic model and not editable fields. For example the description field. Very bad and not useful.

Maybe there is some way to tell VS code that I don't want to edit the schema. I will see if I figure that out.

[MJGaughran]

I've not had any issues with 'editing metadata fields' or similar. Feel free to send a screenshot if it continues.

[TeresiaOlsson]

Okay, that's good. Then I will try some more now when I know that there is hope that it should work. Weird things were also going on with the YAML extension for me for a while where it didn't even wanted to check the syntax... And I had copilot activated so AI also started to interfere and make suggestions. Bad copilot that rather crashed than helped me.

[TeresiaOlsson]

VS code is a mystery to me... But I think my editor might be doing the same as yours. It suggest something that it recalls from before but for me it is very detailed because it remembers what copilot created before I turned it off. It does this very elaborate suggestion:

But this helped. Those type of suggestions are gone. And also that it attempts to add a description field so that could potentially also have been something the AI came up with.

Try this in settings:

{
    "[yaml]": {
        "editor.wordBasedSuggestions": "currentDocument"
    }
}

I find it no longer suggests arbitrary strings at the top level, but still lets you recall fields from before. It doesn't seem to suggest based on field type, though.

You can also switch to "off" if you only want the schema field suggestions.

Only problem now is that auto-completion for hardware is null and not the keywords for DeviceModel. It doesn't seem to handle the auto-completion for the references well which is strange.

    "hardware": {
      "anyOf": [
        {
          "$ref": "#/$defs/DeviceModel"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "Hardware device associated with the magnet"

[MJGaughran]

I believe that's because those fields are optional, so null is a valid (and default) value. You can still autocomplete all properties by using autocomplete on the next line (indented).

The exact keys you press are dependent on your editor config, but in a test project I can:

• On an empty line, use ctrl-space to bring up suggestions
• Select 'unitconv' and use tab to select
• Backspace to delete null (the default)
• Press enter to get a new (indented) line
• ctrl-space to view suggestions
• Select 'LinearConvModel' to autofill all properties of the object, or select individual properties

If you find copilot isn't helpful for yaml specifically, I believe you can disable it for particular file types. Without testing, something like:

"[yaml]": {
    "github.copilot.editor.enableAutoCompletions": false
}

[MJGaughran]

You can also use "editor.acceptSuggestionsOnEnter" as either on or smart for yaml files. I find this automatically deletes the null that gets auto-suggested, so there's no need to press backspace. Obviously there are other ways of configuring this further.

It's probably worth considering that VS Code is very widely used, as are yaml files. There will be ways to customise its behaviour to meet most people's requirements.

[TeresiaOlsson]

Then I tested the MetaConfigurator.

• I went to this page: https://www.metaconfigurator.org, opened my schema file and switched to yaml.
• Then I could edit the fields either using the GUI or the editor. It was very easy. The GUI will tell me exactly which options I have to choose from.
• If I want to edit the schema instead of the models I can switch from "Data Editor" to "Schema Editor". Then I also get a visualisation of how the different data models connect together.

I really liked it.

[MJGaughran]

I assume, despite talking about 'editing the schema', you are only interested in displaying it in a friendlier format?

[TeresiaOlsson]

In this case yes. You can also use the tool to generate a schema and modify it but for pyAML I imagine that the schema should be extracted from the code and then the user would only use the "Data Editor" functionality to put together their configuration in the format described by the schema.

Then the only useful part of the "Schema Editor" mode would be to see the figure of how the different data models are connected together. Maybe that could be useful for some users who are not familiar with data models? At least for me it helps.