Snakemake

[TeresiaOlsson]

I'm wondering if anyone has experience of snakemake and what you think of it?

https://snakemake.github.io/

It sounds like a tool that might be of interest for simulated commissioning since that is a workflow which people might want to both run on their own computer and later scale to a cluster with many seeds. @kparasch maybe you have already looked at it?

[kparasch]

It's the first time I hear of it but it does look interesting. I have some worries about it though. Would it be the worth the investment if every new user needs to learn another language? Then in the introduction of their paper they mention 23 other packages that can do something similar and I am afraid we start relying on something that will become stale in a couple of years.

In any case, I think it could be worth to give it a try and see if it can be useful without too much effort.

[TeresiaOlsson]

As I have understood it, snakemake is a pretty standard tool to use in other fields where it is common to use workflow managers so I don't think we need to worry about it becoming stale. Overall I think workflow managers will develop into becoming a standard tool for creating reproducable results. In some fields it already is (like bioinformatics) but accelerator physics seems to be a bit behind this trend.

I thought the syntax was just a yaml file where you describe your input, output and which python script to run? To me it sounded similar to ansible which we use at HZB to manage the software in our control room and cluster and that I have found to be very easy to learn and use. And actually also similar to how your write a CI/CD pipeline since that is also a yaml file describing a workflow.

I'm curious to test how well the cluster integration works. According to the documentation you should be able to integrate it with your workload manager (like SLURM) and then directly submit the jobs. I know that some labs run simulated commission where they either need to write the cluster integration part themselves or copy files back-and-forth and that is just not an efficient workflow in my experience. It would be very nice if this could work so you just run the workflow on your computer and snakemake will directly submit the jobs to the cluster queue.