pyaml presentation

[JeanLucPons]

Please find my proposal for pyaml architecture (Maintainers meeting 25/072025):

PyAML prototype.pdf

More details on EBS combined function magnets can be found here, source code and calibration curves for magnetic models can be found here.
UnitConv will be renamed to MagnetModel as suggested by Yoshi.
A unit conversion module dedicated to conversion from non standard unit (SI) can be added.
pySC dependency is still under investigation.
Asynchronous calls (DOOCS and EPICS) for fast family access is under investigation.

[gubaidulinvadim]

I would say that a general unit conversion that Yoshi was talking about (even if for most non-magnet elements it's a simple multiplication factor) is useful. Even if TANGO already allows to do this. I didn't quite follow what @yhidaka meant by his BPM example though. If it is a conversion between different type of signals (Vertical difference, vs diagonal difference) or a constant offset, I would not say that this falls under unit conversion concept.

[yhidaka]

An example I mentioned was a fictitious case of x = raw_sig_1 + raw_sig_2 and y = raw_sig_1 - raw_sig_2. We don't have this kind of BPM at NSLS-II, but we do actually have this type of orbit steerer (i.e., each channel provides diagonal kicks).

If pyAML has the capability of generic unit conversion, we can let that handle BPM calibration as well, using a 2x2 matrix to apply gain and rotation errors. Here, the units for raw_x & raw_y are the same as those for calibrated_x & calibrated_y, but they are both equally useful signals to have.

There are many more use cases for having such a generic conversion capability, a subset of which aphla already had from Day 1. Of course, simple multiplication factor and offset can be handled as well.

In PAMILA, I implemented "unit converters" as universal converters, without any restriction on what types of signals (i.e., properties associated with magnets, BPMs, RF, etc.) can be used or whether the units actually even have to change after conversion. All the converters simply accept (any) input arguments and return the output after (any) conversion to signal objects. I believe this idea is similar to Waheed's translator / liaison architecture in that it treats all signals the same way.

[JeanLucPons]

I see. Do you expect that this kind of conversion object can always be invertible or at least that it admits a usable pseudo inversion ?
I'm also wondering how to make it user friendly. For instance for a BPM tilt I would like that a user has a well defined and typed tilt() method on the BPM object, so the auto completion work and the documentation appears in tooltip. May be using pyi ?

[JeanLucPons]

Here is for instance what i would like to target. This greatly improve user experience.

[TeresiaOlsson]

Maybe we can look at how it's done in pytac? If I understand the suggestion correctly I think pytac also had generic unit conversion like that which you specified in the configuration. Here are some examples of how the user would use it:
https://diamondlightsource.github.io/pytac/main/user/tutorials/basic-tutorial.html#unit-conversions

[TeresiaOlsson]

I would expect the unit conversion to always be invertible but I have understood that it is difficult to do for the ESRF magnets. Maybe there could be an option to specify if a unit conversion is invertible or not to solve that problem? Like you for signals can specify if it's read and write or read only signal?

[TeresiaOlsson]

@JeanLucPons I'm wondering how you plan in your proposal that the automatic generation of metadata and saving the results in a standardized structure will be done? If I understood correctly you are not planning to include something like a measurement execution engine that should handle it? Can you perhaps explain a bit your thoughts about the part in the slides that says "user pyaml scripts"? Is the idea that this part should be implemented using something like plans/flows like @yhidaka and how will the metadata and data be saved?

One requirement that is very important for us is that this should not be implemented on an application basis because then it will be like in MML where data management is only done for the standard application while the scripts the physicists write themselves to do a quick measurement during machine time will have poor data management and they have to waste time trying to figure out a good format to save it in, headings etc.

The second requirement is that we want to be able to redo analysis without having to redo measurements. That is also a problem in MML where many scripts don't have separation between the different steps. So if you realise the analysis had problem you need to spend a lot of time trying to extract the measurement data from the saved data and write a separate script to redo the analysis. I think that was one of the reasons behind the flows @yhidaka suggests.

[TeresiaOlsson]

Okay, then I understand that. Thank you :) A second question I'm wondering about now. At what step do you think the part where applications should be possible to run in either hardware and physics units without modifications come in then? I thought this would be in the experiment orchestration part but since it might have a significant impact on the architecture we need to consider it already now?

Currently I don't understand in your proposal how the user will write an application without having to choose if it should be done in hardware or physics units. Can you perhaps explain what your idea is for that? Will there maybe be some general get and set function like in MML which does it in the units you have specified? Or it will be more like in ophyd where it returns several attributes at once when you read a device so the user always get both hardware and physics? Or maybe something else?

[JeanLucPons]

For the time being it is like in this tune correction example:

quadForTuneDesign = sr.design.get_magnets("QForTune")

# Hardware units (not always possible depending on underlying magnet models and may throw an exception)
currents = quadForTuneDesign.hardwares.get()
currents += np.matmul(correctionmat,[0.1,0.05]) # Ask for correction [dqx,dqy]
quadForTuneDesign.hardwares.set(currents)

# Physics units
strs = quadForTuneDesign.strengths.get()
strs += np.matmul(correctionmat,[0.1,0.05]) # Ask for correction [dqx,dqy]
quadForTuneDesign.strengths.set(strs)

And we can implement something like I proposed here:

sr.design.set_hardware_mode()
vars = quadForTuneDesign.get()
vars += np.matmul(correctionmat,[0.1,0.05]) # Ask for correction [dqx,dqy]
quadForTuneDesign.set(vars)

Of course here it is an example and the correctionmat has to be selected by the user, but this selection will be handle automatically later in Tune object of the tuning tools package. If you need to access currents (or voltages) of a complex magnet model that do not support hardware units, you have to access them through the magnet model as described above.

[TeresiaOlsson]

For me the first example is what we don't want because then the unit is hardcoded into the script and we need to maintain two versions for the users to be able to choose.

I think we want the second example. Together with the possibility to not only specify which unit to use globally but also be able to overwrite that default in the get call if desired. Like it can be done in pytac. I think the unit conversion worked well there. I think pytac also has a way to specify if you want to get the set point or the readback which we perhaps could take inspiration from?

[JeanLucPons]

I don't see why you want to specify unit to be used in the get call ?
Either you use quadForTuneDesign.set(vars) or quadForTuneDesign.hardwares.set(vars) or quadForTuneDesign.strenghts.set(vars).

Note that it also possible to write:

quads = quadForTuneDesign.hardwares if mode=='hardware' else quadForTuneDesign.strengths
something = quads.get()
quads.set(something)

The set_hardware_mode() or set_physics_mode() will likely do something similar internally.

[TeresiaOlsson]

Okay, it can be done that way too. What I wanted was just the functionality to globally define what the default unit should be and be able to overwrite that default if you have a part of your script which needs to always use specific units and where the user shouldn't be able to choose.