Consider moving to Apache airflow as workflow framework https://airflow.apache.org/docs/apache-airflow/stable/index.html