Proposal for configuration format #2
Replies: 17 comments 91 replies
-
I really like that the format is keywords only. I think that makes it easy for new users to understand and to convert between different formats. I also like the use of package and module names for the types. Before I was thinking that the types that are part of pyAML could be just a keyword, so |
Beta Was this translation helpful? Give feedback.
-
I feel the unit conversion module needs to be more generic, and should be re-usable for any device, not just for magnets. I think the three generic converters, "identity", "poly1d" (from By the way, I've already implemented a working module with these as This module should be easily integrated into pyAML, as it was deliberately designed that way by removing any specificity. Any unit conversion can be defined through the class FunctionSpec(BaseModel):
name: str
args: List[Any] = Field(default_factory=list)
kwargs: Dict[str, Any] = Field(default_factory=dict)
description: str = ""
input_repr_ranges: Dict[
str, Tuple[float | None, float | None] | List[float | None]
] = Field(default_factory=dict) class UnitConvSpec(BaseModel):
src_units: str | List[str]
dst_units: str | List[str]
func_spec: FunctionSpec
func: Callable | None = Field(
default=None, exclude=True
) # Exclude func during serialization As an example, a custom unit conversion module can be defined like this (https://github.com/NSLS2/pamila/blob/main/examples/conv_plugins/ID23d_repr_convs.py): from pamila.device.conversion.plugin_manager import register
@register(name="ID23d_repr_convs.from_I1_I2_gap_to_x", is_factory_function=False)
def from_I1_I2_gap_to_x(I1: float, I2: float, gap: float):
# I1, I2: [A]
# gap: [mm]
# x: [urad]
x = (I1 + I2) * max([(1e2 - gap), 0.0]) / 20.0
return [x]
... |
Beta Was this translation helpful? Give feedback.
-
Thanks Yoshi for sharing your work. It is true that the unit conversion that I presented above refers to magnets as it can get multiple strenght(s) and produce multiple current(s) or vice/versa using a random model. At ESRF we used to work using SI at the device level and then Tango has an embedded unit conversion configuration for displaying with the unit you want. I would prefer to add this kind of unit conversion at the Do you know if it is possible to override the pydantinc BaseModel constructor in order to avoid duplicate type checking for the Factory ? What tool do you use to edit your config file ? Do you take advantages of pydantinc schema for editing and if yes how do you handle external module ? Thanks. |
Beta Was this translation helpful? Give feedback.
-
Each lab has their own magnet models and users who want to use PyAML using unitconv will have to write their own modules to fit in PyAML. We will provide only basic one for PyAML with examples to write advanced magnet models. For instance, EBS sextupole model will not be in PyAML unless we share it with DESY. So it is important that the interface to write a magnet model is as intuitive and as simple as possible. |
Beta Was this translation helpful? Give feedback.
-
Thanks Teresia. This looks not so bad and rather intuitive. I'm now wondering how to make it recursive to handle nested objects. In your example the User class has be to able to build an arbitrary object from the config file. The goal is also to try to use all provided shema (internal or external to pyaml) with an editor (vscode may have pydantic plugin) otherwise no real added value to use pydantic for our needs. I really would like to avoid as much as possible args and kwargs. The Factory constructor just pass what is in the config file to the constructor by unpacking the dict, it is a nice and simple approach. We also have to check performances for large ring that have thousands of magnets. |
Beta Was this translation helpful? Give feedback.
-
Please keep in mind that we have to provide a simple way to add random magnet models otherwise pyaml will simply not be used. - type: pyaml.magnet.CombinedFunctionMagnet
name: SHC01E
H: SHC01E-H # Virutal single function magnet to be used in various magnet families
V: SHC01E-V # Virutal single function magnet to be used in various magnet families
SQ: SHC01E-SQ # Virutal single function magnet to be used in various magnet families
unitconv:
type: ebs.magnet.SH3UnitConv
units: ["rad","rad","m-1"]
hcurve: /path/curve/sh3_h.yaml
vcurve: /path/curve/sh3_v.yaml
sqcurve: /path/curve/sh3_sq.yaml
matrix: /path/curve/sh3_matrix.yaml # matrix handling cross dependencies between functions
calibration_factor: 1.00656
calibration_offset: 0.0034
powerconverter:
-type: pyaml.control.device
setpoint: sr/ps-sh3/c01-e-1/current
readback: sr/ps-sh3/c01-e-1/current
unit: A
-type: pyaml.control.device
setpoint: sr/ps-sh3/c01-e-2/current
readback: sr/ps-sh3/c01-e-2/current
unit: A
-type: pyaml.control.device
setpoint: sr/ps-sh3/c01-e-3/current
readback: sr/ps-sh3/c01-e-3/current
unit: A |
Beta Was this translation helpful? Give feedback.
-
I just take note there of possible design improvements to remember. Try to use dict backtracking in order to provide a more comfortable frame for the implemention of internal and external modules. backtracking should allow to have: class Quadrupole(Magnet):
"""Quadrupole class"""
def __init__(self, name:str = None, hardware : Device = None, unitconv: UnitConv = None): instead of: class Quadrupole(Magnet):
"""Quadrupole class"""
def __init__(self, name:str = None, hardware:dict = None, unitconv: dict = None):
super().__init__(name) Where |
Beta Was this translation helpful? Give feedback.
-
The backtracking of the dict works, that's great ! But unfortunatly python3 does not check abstract type so the The factory becomes: """Build an object from the dict"""
def buildObject(d:dict):
typeStr = d["type"]
if typeStr is None:
raise Exception("No type specified for " + str(d))
del d["type"]
p = importlib.import_module(typeStr)
return p.factory_constructor(d)
"""Main factory function"""
def deepFirstBuild(d:dict):
# Depth-first factory
for key, value in d.items():
if isinstance(value,dict):
obj = deepFirstBuild(value)
#Replace the inner dict by the object itself
d[key]=obj
# we are now on a leaf (no nested object), we can construt
obj = buildObject(d)
return obj |
Beta Was this translation helpful? Give feedback.
-
Here is new proposal for module implementation. I didn't know if pydantic can help us here ? Any tips are wlecome. from .Magnet import Magnet
from .UnitConv import UnitConv
from pyaml.control import Abstract
from pyaml.control.Device import Device
from pyaml.lattice.RWFloat import RWFloat
from pyaml.configuration.Factory import checkType
class Quadrupole(Magnet):
"""Quadrupole class"""
def __init__(self, name:str, hardware:Device = None, unitconv: UnitConv = None):
super().__init__(name)
# Check input type
checkType(hardware,Device,"Quadrupole %s" % (name));
checkType(unitconv,UnitConv,"Quadrupole %s" % (name))
if(hardware is not None):
# Direct access to a magnet device that supports strength/current conversion
# TODO
raise Exception("Quadrupole %s, hardware access not implemented" % (name))
elif(unitconv is not None):
self.unitconv = unitconv
else:
raise Exception("Quadrupole %s, no control system or model defined" % (name))
self.strength : Abstract.ReadWriteFloatScalar = RWFloat(self.unitconv)
def __repr__(self):
return "%s(name=%s, unitconv=%s)" % (
self.__class__.__name__, self.name, self.unitconv)
def factory_constructor(config: dict) -> Quadrupole:
"""Construct a Quadrupole from Yaml config file"""
return Quadrupole(**config) In case you make wrong link in your files, you get such an error:
Line number and file name for error management are not yet implemented. |
Beta Was this translation helpful? Give feedback.
-
List of points to address (not complete):
|
Beta Was this translation helpful? Give feedback.
-
@JeanLucPons Did you in your suggestion already get it to work nicely with reading many yaml files? I was planning to test This is what I'm planning to look into: https://hydra.cc/docs/advanced/compose_api/ |
Beta Was this translation helpful? Give feedback.
-
I didn't try 'hydra' yet but i'm affraid it will loose line numbers for error reporting. The problem is always the same, you can have a fully valid set of yaml files but with an invalid structure or an invalid keyword somewhere. When you construct your objects, you have to be able to cleary tell to the user from where the error comes. |
Beta Was this translation helpful? Give feedback.
-
If we decide to say no file number and file name, then the config loader and the factory are done. For ~150 lines of python without extra complex dependecies, it answers to requirements. No overhead due to the duplication of the constructors signature induced by pydantic in each object. |
Beta Was this translation helpful? Give feedback.
-
At the ESRF when we change a magnet in the ring by a spare one, we change its serial number in the DB which allows the software to retrieve good calibration factors stored in csv files (it contains also spare mangets). How to handle this in our scheme ? Changing manually the UnitConv calibration factor ? Implement such a mechanism using serial number ? |
Beta Was this translation helpful? Give feedback.
-
I recall that I will not code all pyAML myself !
|
Beta Was this translation helpful? Give feedback.
-
@JeanLucPons I will test the schema + editor in VS code today. Is there a way to extract a json schema from the classes in the |
Beta Was this translation helpful? Give feedback.
-
@JeanLucPons @TeresiaOlsson I made an example where I converted you can find it in the branch called You can try putting some extra fields in the configuration files to see the validation fail :) |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Hello,
I restart the discussion around the file format here and the factory design.
Nothing yet about pydantic scheme but any tips are welcome.
This scheme is incomplete and some fields may be added later. For instance in the examples bellow, there is nothing in the powerconverter object to get state and status in order to report faults to the user. Asynchronous stuff, such as event mechanism, is not yet there as well. Each field containing a link to a yaml file is automatically expanded.
For EBS magnets unit conversion, we use similar architecture (in C++) that you can access here and complex combined function magnets are supported. A simple example of a combined function magnet is SH3, a sextupole with poles forming a rectangle able to perform H,V,SQ strengths. SH5 can perform H,V,SQ,S.
A classic quad configuration:
A classic quad configuration, with unitconv description specified in a separate file:
A quad configuration that bypass the unitconv (control system allows to work with strength):
A quad with an experimental unitconv object:
The type refers to the name of a python module that must containts a factory_constructor.
An example of a simple implementation can be seen below:
UnitConv is an abstract class that the underlying unitconv object must implement.
Beta Was this translation helpful? Give feedback.
All reactions