Proposal for main pyaml structure

[JeanLucPons]

Here is my implementation for the top level structure:

type: pyaml.pyaml
instrument: sr
simulators:
  - type: pyaml.lattice.simulator
    lattice: lattices/ring.mat
    name: model
data_folder: /data/store
arrays:
  - type: pyaml.lattice.array
    name: HCORR
    elements:
      - SH1-C01A-H
      - SH1-C02A-H
  - type: pyaml.lattice.array
    name: VCORR
    elements:
      - SH1-C01A-V
      - SH1-C02A-V
devices:
  - sr/quadrupoles/QF1C01A.yaml
  - sr/quadrupoles/SH1_C01A.yaml
  - sr/quadrupoles/SH1_C02A.yaml

Associated with its pydantic schema:

class ConfigModel(BaseModel):

    model_config = ConfigDict(arbitrary_types_allowed=True)

    instrument: str
    """Instrument name"""
    control: ControlSystem = None
    """control system"""
    simulators: list[Simulator] = None
    """Simulator list"""
    data_folder: str
    """Data folder"""
    arrays: list[Array] = None
    """Element family"""
    devices: list[Element]
    """Element list"""

Usage:

from pyaml.pyaml import pyaml
from pyaml.configuration import set_root_folder
set_root_folder("tests/config")

sr = pyaml("sr.yaml")
...

It is rather close to the original spec done by @simoneliuzzo.
Any tips, comments are welcome.

[simoneliuzzo]

Dear @JeanLucPons,

I like the simple usage! It looks very similar to the User interface specification indeed!

"root folder" could be part of the configuration file?

For the array names, should some arrays names be compulsory? for example Quadrupoles, Sextupoles, Dipoles, HorSteerers, VerSteerers, SkewQuadrupoles, CTs, BPMs, BLMs, etc... (The list is in the Spec document).

In order to keep the configuration scalable, we could default the above compulsory arrays to be empty and if defined fill them with the user definition given in the configuration file.

pyaml is becoming real very fast!

[JeanLucPons]

model.errors are models.

[TeresiaOlsson]

@JeanLucPons You are not worried about performance issues if the whole config needs to be reloaded for every operational mode? I thought performance was an issue you had for FCC?
@simoneliuzzo Automatically calculating files are nice but I think this solution will only work for machines that have good agreement between model and machine. In our case we need to determine all the response matrices experimentally. It also won't work when a new mode has been found experimentally so no lattice file exists.

@TeresiaOlsson do you switch between different operation modes daily at HZB? At SOLEIL this is only done on machine Mondays in preparation for the user operation for the week (if next week is in a different operation mode).

At MLS we actually have a dedicated tool. You just click a button and the machine will do the energy and optics ramps automatically. It can ramp both up and down in energy and between several different optics. My dream is to eventually have this functionality in pyAML so the same procedure easily can be simulated, be used with a shadow etc. I'm also guessing that the software won't live forever so it might be easier to maintain long-term if it was connected to pyAML. And I would also like to have the functionality at BESSY II because at the moment switching to low alpha there is a manual process which takes hours and requires an experienced accelerator physicists to do. With the tool at MLS, the operators can switch the mode. I think they probably also switch the mode mostly weekly but since MLS operates in decay mode it is also used for ramping down (with current) for injections and then ramping up again. If you are operating in low-alpha mode for example you need to do both energy ramp + optics ramp every injection to not lose the beam.

[simoneliuzzo]

model.errors are models.

We can have some functions to set the errors (but I do not see it as urgent), or let the user do this job and specify the location of lattices with errors.
An error is raised if the lattices are not compatible with the model.design lattice file, for example the number of elements is different.

[simoneliuzzo]

@JeanLucPons You are not worried about performance issues if the whole config needs to be reloaded for every operational mode? I thought performance was an issue you had for FCC?
@simoneliuzzo Automatically calculating files are nice but I think this solution will only work for machines that have good agreement between model and machine. In our case we need to determine all the response matrices experimentally. It also won't work when a new mode has been found experimentally so no lattice file exists.

@TeresiaOlsson do you switch between different operation modes daily at HZB? At SOLEIL this is only done on machine Mondays in preparation for the user operation for the week (if next week is in a different operation mode).

At MLS we actually have a dedicated tool. You just click a button and the machine will do the energy and optics ramps automatically. It can ramp both up and down in energy and between several different optics. My dream is to eventually have this functionality in pyAML so the same procedure easily can be simulated, be used with a shadow etc. I'm also guessing that the software won't live forever so it might be easier to maintain long-term if it was connected to pyAML. And I would also like to have the functionality at BESSY II because at the moment switching to low alpha there is a manual process which takes hours and requires an experienced accelerator physicists to do. With the tool at MLS, the operators can switch the mode. I think they probably also switch the mode mostly weekly but since MLS operates in decay mode it is also used for ramping down (with current) for injections and then ramping up again. If you are operating in low-alpha mode for example you need to do both energy ramp + optics ramp every injection to not lose the beam.

How fast is this ramp in energy/optics? milliseconds or minutes?

[TeresiaOlsson]

@JeanLucPons You are not worried about performance issues if the whole config needs to be reloaded for every operational mode? I thought performance was an issue you had for FCC?
@simoneliuzzo Automatically calculating files are nice but I think this solution will only work for machines that have good agreement between model and machine. In our case we need to determine all the response matrices experimentally. It also won't work when a new mode has been found experimentally so no lattice file exists.

@TeresiaOlsson do you switch between different operation modes daily at HZB? At SOLEIL this is only done on machine Mondays in preparation for the user operation for the week (if next week is in a different operation mode).

At MLS we actually have a dedicated tool. You just click a button and the machine will do the energy and optics ramps automatically. It can ramp both up and down in energy and between several different optics. My dream is to eventually have this functionality in pyAML so the same procedure easily can be simulated, be used with a shadow etc. I'm also guessing that the software won't live forever so it might be easier to maintain long-term if it was connected to pyAML. And I would also like to have the functionality at BESSY II because at the moment switching to low alpha there is a manual process which takes hours and requires an experienced accelerator physicists to do. With the tool at MLS, the operators can switch the mode. I think they probably also switch the mode mostly weekly but since MLS operates in decay mode it is also used for ramping down (with current) for injections and then ramping up again. If you are operating in low-alpha mode for example you need to do both energy ramp + optics ramp every injection to not lose the beam.

How fast is this ramp in energy/optics? milliseconds or minutes?

Now you are asking too complicated questions ;) I need to ask a colleague for the details, but I would say minutes. It's not like a booster ramp. It relies on a set of ramping tables which makes a path through the tune space which has been experimentally found to best maintain the current.

[simoneliuzzo]

Dear @JeanLucPons,

In my opinion, I think we can add the energy to the configuration file as a configuration of pyaml.lattice.simulator. It will be useful in my opinion when we will use pyaml for boosters. It may be also useful for machines that may modify their dipoles or that could measure precisely their energy, the new gradients will recompute for the correct energy, as specified in the configuration file.

In the pyaml simulator code after loading the lattice we will issue an atsetenergy to make sure the value of the configuration file is used.

[simoneliuzzo]

Dear @TeresiaOlsson,
I think energy and momentum compaction are rather different parameters.
Energy is an input for lattice computation. The same lattice file works for all energies. So to me it makes sense to have it as an input, in the configuration.
Momentum compaction instead is a parameter computed from the optics. The lattice has only one momentum compaction that may not be changed by the user at wish. So it is a parameter that has to be computed from the lattice, and eventually stored somewhere for fast access.

[TeresiaOlsson]

Whatever the solution is I think it needs to work for these cases:

Energy: ramped machines and linacs. In MML the energy is specified in the config per operational mode. So if we want it in the config I think it needs to be a mode specific parameter and we need a way to load it without reloading the whole config.

Experimentally found lattices: sometimes a new lattice is found in the control room so no lattice model exists. This we for example did a while ago with a negative alpha mode in BESSY II. In this case we measure the momentum compaction factor and want to put the experimentally found value into pyAML so we can measure the chromaticity despite having no model yet.

[simoneliuzzo]

Ok, I get it. This is something that never happens for us (we do not have a measurement of the beam energy).
I think it could work as for tune, ORM, chroma responses. You can have the simulated ones or measured ones.

folder BESSY

• folder standard

1. standard.yaml
2. alpha.csv % generated by pyaml if missing based on model
3. alphameasured.csv % measured by some tuning tool "momentumcompaction measurement"
4. alphanew.csv % value inferred with some other tools not part of pyAML

the file standard.yaml will be modified to point to either of the files. No need to change optics, or lattice if this is not requested. If the optics become operational or you need to switch among the different alpha optics, then a new operation configuration is created copy pasting standard.yaml to for example newalpha.yaml and modifying where needed.

folder BESSY

• folder standard

1. standard.yaml
2. alpha.csv % generated by pyaml if missing based on model
3. alphameasured.csv % measured by some tuning tool "momentumcompaction measurement"
4. alphanew.csv % value inferred with some other tools not part of pyAML

• folder newalpha

1. newalpha.yaml
2. alphanew.csv % value inferred with some other tools not part of pyAML

This second option is my favorite as it allows to to revert easily to standard operation settings.

[simoneliuzzo]

concerning the energy parameter I have no problems to update it as needed after instantiation of pyaml.

import pyaml
booster = pyaml(./booster/standard.yaml)
E = booster.energy.get()
6e9 eV
booster.quadrupole.current.get()
100 A
booster.rf.voltage.get()
1000 V
booster.energy.set(3e9)
booster.quadrupole.current.get()
50 A
booster.rf.voltage.get()
500 V

For linacs the energy is intended as the maximum energy. And it can be changed, with impact both on RF and magnet settings.

The energy.set() command will require quite some work if we want it. For us for example, I think it is not needed.

I have a question for MML users: MML drives the booster ramp? For us this is driven by some extremely complex PS. What we can act on is the curve (a file), but this is unlikely to be made machine agnostic.

[TeresiaOlsson]

I haven't used the booster ramp in MML myself yet but I think there might be some labs who have experience of it. There is an example for the ALS booster in the MML git repository.

[TeresiaOlsson]

I think it would be nice if we could list what is static config data and what is dynamic config data. If we have that it might be easier to figure out how to load the different parts? For me it feels like the most elegant solution would be one where the static data is only loaded once and the dynamic data is more like an update like @simoneliuzzo suggested.

I think @Sulimankhail probably has ideas for how that could be implemented because we have the exact same issue with different operational modes in our twin and have discussed static vs dynamic config before and how to deal with it.

[simoneliuzzo]

For me configuration file informations are static, as a calibration curve or and hardware name.
If some parameter must change rather frequently (dynamic) then it should have a tunable field in pyaml as for example strength for the quadrupoles.

Let's imagine a chroma measurement high level application in pyaml.
The momentum compaction could be:
.1 a property written in configuration,
.2 an attribute editable by the user (for example with measured data) and with default some property in the configuration file.

the second option, I think is better for a chroma measurement application.

[TeresiaOlsson]

I think it might also be worth spending some time to look at how they do it in hydra. I have only looked at it briefly so far because I didn't have any more time but it seems like a package where they try to solve the same problem as us.

If it doesn't work to use it directly we can maybe at least get inspiration from how they do it.

https://hydra.cc/

[gubaidulinvadim]

@TeresiaOlsson @Sulimankhail Please make a simple example of how you would solve the configuration of different operation modes with static and dynamic data categorisation (with hydra or without), then it can be discussed. It's not very practical for everyone to read the Hydra documentation and try to implement something. And I think you are the only ones with experience in using many different operation modes and changing them on the fly.

[TeresiaOlsson]

@gubaidulinvadim I agree. We will do it and test out some different options but it will likely take some weeks before we can come back with a suggestion. Many other things we also need to do at the moment :(