diff --git a/README-fr.md b/README-fr.md index 80efdc8..4f903e5 100644 --- a/README-fr.md +++ b/README-fr.md @@ -16,25 +16,72 @@ Il est possible de faire appel à des jours de consulting pour l'implémentation * Visual TOM 7.1.2 ou plus * Python 3.x ou plus * Azure Data Factory resource - * Installer les packages python requis avec pip: + * Création d'un environnement virtuel venv : + + Se positionner dans le répertoire projet (qui contient les scripts et le fichier requirements.txt) : + * python → ton interpréteur + * .venv → nom du dossier qui contiendra l’environnement virtuel (bonne pratique : toujours dans le projet). + + En ligne de commande (Windows ou Linux) : ```bash + python -m venv .venv + ``` + Cela crée une arborescence : + + mon_projet/ + * .venv/ <- environnement virtuel + * azureDataFactory.py + * requirements.txt + + * Activation de l'environnement virtuel : + + Sous Windows : + ``` + mon_projet\.venv\Scripts\activate.bat + ``` + Sous Linux : + ``` + source mon_projet/.venv/bin/activate + ``` + Quand il est activé, l'invite de commande affiche généralement (venv) ou (.venv) devant → tous les pip install se feront dans cet environnement. + + Installer les packages python requis dans votre environnement virtuel : + ``` pip install -r requirements.txt ``` - * Agent Unix (l'utilisation sous Windows sera disponible plus tard) + + * Installation et paramétrage des queues Windows et Linux : + * Agent Unix : tom_submit.azdatafactory + * Agent Windows : submit_queue_azdatafactory.bat + + Renseigner la variable **PROJECT_PATH** dans la queue batch (Windows ou Linux) pour indiquer votre répertoire projet. + + Exemples : + ``` + set PROJECT_PATH=%TOM_HOME%\SCRIPTS\AzureDataFactory\ + ``` + ou + ``` + project_path=/var/lib/absyss/visual-tom/scripts/azure/az-datafactory + ``` # Consignes - * Créer une application Azure et définir les variables d'environnement suivantes dans config.py dans le même dossier (un template est disponible dans le dépôt): + * Créer une application Azure et définir les variables d'environnement suivantes dans un objet Contexte dans Visual TOM : * `AZURE_SUBSCRIPTION_ID`: Subscription ID de votre Azure subscription * `AZURE_TENANT_ID`: Tenant ID de votre Azure Active Directory * `AZURE_CLIENT_ID`: Client ID de votre application Azure Active Directory * `AZURE_CLIENT_SECRET`: Secret client de votre application Azure Active Directory * `AZURE_DATA_FACTORY_RESOURCE_GROUP`: Resource group de votre Azure Data Factory + + L'utilisation des Ressources Secrets est recommandée : + ![Custom application screenshot](screenshots/AzureDataFactory_Context.png?raw=true) + * Créer dans Visual TOM une connexion "Custom Application" avec la définition suivante ou importer le fichier MyApplication-AzureDataFactory.xml: ```bash vtimport -x -f MyApplication-AzureDataFactory.xml ``` - ![Custom application screenshot](screenshots/Azure_DataFactory_CustomApplication.png?raw=true) + ![Custom application screenshot](screenshots/AzureDataFactory_CustomApp_WebInterface.png?raw=true) * Créer la queue batch sur les Agents et mettre à jour le submitter avec le chemin réel de azureDataFactory.py Description des paramètres: diff --git a/README.md b/README.md index a7f37e5..38cc481 100644 --- a/README.md +++ b/README.md @@ -16,25 +16,72 @@ Consultings days can be requested to help for the implementation. * Visual TOM 7.1.2 or greater * Python 3.x or greater * Azure Data Factory resource - * Install the required python packages using pip: + * Creation of a virtual environment (venv): + + Go to the project directory (which contains the scripts and the requirements.txt file): + * python → your interpreter + * .venv → the folder that will contain the virtual environment (best practice: always inside the project repository). + + From the command line (Windows or Linux): ```bash + python -m venv .venv + ``` + + This creates the following structure: + + my_project/ + * .venv/ <- vitual environment + * azureDataFactory.py + * requirements.txt + + * Activating the virtual environment : + + On Windows : + ``` + my_project\.venv\Scripts\activate.bat + ``` + On Linux : + ``` + source my_project/.venv/bin/activate + ``` + When activated, the command prompt usually displays (venv) or (.venv) → all pip install commands will install packages inside this environment. + + Install the required Python packages in your virtual environment: + ``` pip install -r requirements.txt ``` - * Unix Agent (Windows usage will be available later) + * Installation and configuration of Windows and Linux queues : + * Unix Agent : tom_submit.azdatafactory + * Windows Agent : submit_queue_azdatafactory.bat + + Set the PROJECT_PATH variable in the batch queue (Windows or Linux) to indicate your project directory. + + Examples : + ``` + set PROJECT_PATH=%TOM_HOME%\SCRIPTS\AzureDataFactory\ + ``` + or + ``` + project_path=/var/lib/absyss/visual-tom/scripts/azure/az-datafactory + ``` # Instructions - * Create an Azure Application and set the following environment variables in config.py in the same folder (a template is available in the repository): + * Create an Azure Application and set the following environment variables in a object Context Visual TOM : * `AZURE_SUBSCRIPTION_ID`: Subscription ID of your Azure subscription * `AZURE_TENANT_ID`: Tenant ID of your Azure Active Directory * `AZURE_CLIENT_ID`: Client ID of your Azure Active Directory application * `AZURE_CLIENT_SECRET`: Client secret of your Azure Active Directory application * `AZURE_DATA_FACTORY_RESOURCE_GROUP`: Resource group of your Azure Data Factory + + Resources Secret are recommanded : + + ![Custom application screenshot](screenshots/AzureDataFactory_Context.png?raw=true) * Create in Visual TOM a "Custom Application" connection with the following definition or import the file MyApplication-AzureDataFactory.xml: ```bash vtimport -x -f MyApplication-AzureDataFactory.xml ``` - ![Custom application screenshot](screenshots/Azure_DataFactory_CustomApplication.png?raw=true) + ![Custom application screenshot](screenshots/AzureDataFactory_CustomApp_WebInterface.png?raw=true) * Create the batch queue on the Agents and update the submitter with actual path of azureDataFactory.py Description of the parameters: diff --git a/azureDataFactory.py b/azureDataFactory.py index 0a61c2e..43ba19f 100644 --- a/azureDataFactory.py +++ b/azureDataFactory.py @@ -4,7 +4,6 @@ import argparse import json import os -from config import * ##################################################### ### Function to print messages to the standard output @@ -29,6 +28,19 @@ def load_json_param(param): except json.JSONDecodeError: raise ValueError("Invalid JSON string or file path provided") +##################################################### +### Load Azure configuration from environment variables +##################################################### +AZURE_SUBSCRIPTION_ID = os.getenv("AZURE_SUBSCRIPTION_ID") +AZURE_TENANT_ID = os.getenv("AZURE_TENANT_ID") +AZURE_CLIENT_ID = os.getenv("AZURE_CLIENT_ID") +AZURE_CLIENT_SECRET = os.getenv("AZURE_CLIENT_SECRET") +AZURE_DATA_FACTORY_RESOURCE_GROUP = os.getenv("AZURE_DATA_FACTORY_RESOURCE_GROUP") + +# Checking security +if not all([AZURE_SUBSCRIPTION_ID, AZURE_TENANT_ID, AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, AZURE_DATA_FACTORY_RESOURCE_GROUP]): + raise EnvironmentError("⚠️ Some variables are not correctly defined.") + # Argument parsing parser = argparse.ArgumentParser(description='Trigger and monitor an Azure Data Factory pipeline.') parser.add_argument('--factory', type=str, required=True, help='Name of the Azure Data Factory') @@ -89,4 +101,4 @@ def load_json_param(param): if activity_run.status in ["Failed", "Cancelled"]: printFormat("ERROR", f"Activity {activity_run.activity_name} failed with error: {json.dumps(activity_run.error, indent=4)}") - exit(99) \ No newline at end of file + exit(99) diff --git a/config.py.template b/config.py.template deleted file mode 100644 index 4901215..0000000 --- a/config.py.template +++ /dev/null @@ -1,8 +0,0 @@ -#Variables for Azure Credentials -AZURE_SUBSCRIPTION_ID="" -AZURE_TENANT_ID="" -AZURE_CLIENT_ID="" -AZURE_CLIENT_SECRET="" - -#Variables for Azure Data Factory -AZURE_DATA_FACTORY_RESOURCE_GROUP="" \ No newline at end of file diff --git a/screenshots/AzureDataFactory_Context.png b/screenshots/AzureDataFactory_Context.png new file mode 100644 index 0000000..7509b53 Binary files /dev/null and b/screenshots/AzureDataFactory_Context.png differ diff --git a/screenshots/AzureDataFactory_CustomApp_WebInterface.png b/screenshots/AzureDataFactory_CustomApp_WebInterface.png new file mode 100644 index 0000000..90e79e4 Binary files /dev/null and b/screenshots/AzureDataFactory_CustomApp_WebInterface.png differ diff --git a/submit_queue_azdatafactory.bat b/submit_queue_azdatafactory.bat new file mode 100644 index 0000000..567fb92 --- /dev/null +++ b/submit_queue_azdatafactory.bat @@ -0,0 +1,60 @@ +@echo OFF + +call submit_aff.bat %* +echo _______________________________________________________________________ +echo Debut de l'execution du script... +date /T +time /T +echo _______________________________________________________________________ + +rem Mode TEST +if "%TOM_JOB_EXEC%" == "TEST" ( + echo Job execute en mode TEST + %ABM_BIN%\tsend -sT -r0 -m"Traitement termine (mode TEST)" + %ABM_BIN%\vtgestlog + goto FIN +) + + +set PROJECT_PATH=%TOM_HOME%\SCRIPTS\AzureDataFactory\ + +REM Vérifie si 3 arguments sont passés +IF "%~3"=="" ( + REM Execute without params + echo "%PROJECT_PATH%\.venv\Scripts\python %PROJECT_PATH%\azureDataFactory.py --factory %1 --pipeline %2" + %PROJECT_PATH%\.venv\Scripts\python %PROJECT_PATH%\azureDataFactory.py --factory %1 --pipeline %2 + set RETCODE=%ERRORLEVEL% +) ELSE ( + REM Execute with params + echo "%PROJECT_PATH%\.venv\Scripts\python %PROJECT_PATH%\azureDataFactory.py --factory %1 --pipeline %2 --params %3" + %PROJECT_PATH%\.venv\Scripts\python %PROJECT_PATH%\azureDataFactory.py --factory %1 --pipeline %2 --params %3 + set RETCODE=%ERRORLEVEL% +) + +if %RETCODE% equ 0 goto TERMINE +goto ERREUR + +:ERREUR +%ABM_BIN%\tsend -sE -r%RETCODE% -m"Traitement en erreur (%RETCODE%)" +%ABM_BIN%\vtgestlog +echo _______________________________________________________________________ +echo Fin d'execution du script +date /T +time /T +echo Exit [%RETCODE%] +echo _______________________________________________________________________ +if not "%TOM_LOG_ACTION%"==" " call Gestlog_wnt.bat +exit %RETCODE% + +:TERMINE +%ABM_BIN%\tsend -sT -r%RETCODE% -m"Traitement termine (%RETCODE%)" +%ABM_BIN%\vtgestlog +echo _______________________________________________________________________ +echo Fin d'execution du script +date /T +time /T +echo Exit [%RETCODE%] +if not "%TOM_LOG_ACTION%"==" " call Gestlog_wnt.bat +exit %RETCODE% + +:FIN diff --git a/tom_submit.azdatafactory b/tom_submit.azdatafactory index 6d91800..fc2baa9 100644 --- a/tom_submit.azdatafactory +++ b/tom_submit.azdatafactory @@ -23,13 +23,15 @@ if [ -n ${TOM_JOB_ID:-0} ] ; then exit 0 fi + project_path=/var/lib/absyss/visual-tom/scripts/azure/az-datafactory + #if param 3 is empty then parameter params is not set if [ $# -eq 3 ]; then if [ "'$3'" = "''" ]; then - python3 -u /var/lib/absyss/visual-tom/scripts/azure/az-datafactory/azureDataFactory.py --factory $1 --pipeline $2 + ${project_path}/.venv/bin/python3 -u ${project_path}/azureDataFactory.py --factory $1 --pipeline $2 stat_fin_job=$? else - python3 -u /var/lib/absyss/visual-tom/scripts/azure/az-datafactory/azureDataFactory.py --factory $1 --pipeline $2 --params $3 + ${project_path}/.venv/bin/python3 -u ${project_path}/azureDataFactory.py --factory $1 --pipeline $2 --params $3 stat_fin_job=$? fi else