Main script doesn't stop when template has no result.json

[lcdumort]

When a 'subjob' generated from a template fails to produce a result.json file due to an error, the main script does not terminate. Although the error is reported in the output, the main script continues running even though it can't proceed.

This issue is relatively minor. The main concerns are the unnecessary consumption of computational resources to keep the failed main job running and the confusion caused by having to manually cancel the main script to prevent an accumulation of 'dead' jobs.

[tovrstra]

Good point. Both ways of working can be useful: let the script keep going as far as it can, or stop as soon as something fails. It depends on whether that error means everything is probably wrong, in which case it better stops completely. The other scenario is that just a minor thing is fails while the rest is fine, in which case it makes sense to keep going, fix the minor issue and rerun to complete the remaining parts.

It would make sense to stop in case of a failed job by default and to have an option to switch on the "keep going" mode parman has now.

[lcdumort]

I'm currently in a situation where I need to handle failed jobs or errors. Currently, I try to catch the errors that result in the absence of a result.json file in the job scripts. In those cases, I write the string 'None' to the result.json file. I then check for this in processes/jobs that rely on the 'failed' job.

I believe it would make the most sense to add an optional onfail argument to the job constructor. This argument would take a function to be executed if there is no result.json file. This allows the user to write simple functions or lambda expressions in the main run script (e.g., returning None if it fails) or more complex ones to ensure the subsequent part of the workflow remains compatible (e.g., writing an empty file if the next function concatenates multiple files in a list).

If onfail isn't specified, the default behavior should be throwing a runtime error that stops the main script if any job fails to deliver a result.json file. This approach seems the most Pythonic to me.

I won't assign this to myself just yet, but if I can find some time this weekend I'll have a look to see if I can implement this and create a pull-request.

[lcdumort]

On second thought, it makes more sense to add this onfail optional argument to the runner instance rather than the job, as not having a result.json file is outside the scope of a job instance to handle.