SwanProjects extension for jupyter lab 3

README.md

@@ -12,4 +12,5 @@ Repository that stores all the Jupyter extensions for SWAN.
 * [SwanNotebookViewer](SwanNotebookViewer) - Read-only mode for opening notebooks (used from the Sharing interface) inside Jupyter Notebooks
 * [SwanNotifications](SwanNotifications) - Extension to display notifications to users
 * [SwanOauthRenew](SwanOauthRenew) - Extension that fetches the latest oAuth tokens from JupyterHub and writes to the file observed by EOS
-* [SwanShare](SwanShare) - Jupyter Notebooks/CERNBox sharing integration used by SwanContents
+* [SwanShare](SwanShare) - Jupyter Notebooks/CERNBox sharing integration used by SwanContents
+* [SwanProjects](SwanProjects) - Jupyter Lab extension with backend and dialogs to Create/Edit projects, also have a customized KerneSpecManager to handle kernels in multiple environments.

SwanProjects/.bumpversion.cfg

@@ -0,0 +1,11 @@
+[bumpversion]
+current_version = 0.1.0
+commit = True
+tag = True
+tag_name = SwanProjects/v{new_version}
+message = SwanProjects v{new_version}
+
+[bumpversion:file:swanprojects/_version.py]
+
+[bumpversion:file:package.json]
+

SwanProjects/MANIFEST.in

@@ -0,0 +1,29 @@
+include LICENSE
+include README.md
+include SECURITY.md
+include pyproject.toml
+recursive-include jupyter-config *.json
+
+include swanprojects/kernelmanager/resources/*
+include swanprojects/stacks.json
+include swanprojects/static/index.html
+include package.json
+include install.json
+include ts*.json
+include yarn.lock
+
+graft swanprojects/labextension
+
+# Javascript files
+graft src
+graft style
+prune **/node_modules
+prune lib
+prune binder
+
+# Patterns to exclude from any directory
+global-exclude *~
+global-exclude *.pyc
+global-exclude *.pyo
+global-exclude .git
+global-exclude .ipynb_checkpoints

SwanProjects/README.md

@@ -0,0 +1,35 @@
+# SwanProjects
+
+Server and Lab extension that provides:
+* In the backend, the endpoints to:
+  * Create and edit projects
+  * Get project information
+  * Get software stack information
+  * Customized Kernel Spec Manager  to handle kernel metadata.
+* In the Lab extension:
+  * React dialogs with a set of components that allows to create and edit projects
+  * LabIcons required for the dialogs
+
+## Requirements
+
+JupyterLab~=3.0 and SwanContents
+
+## Install
+
+Install the package and the lab extension:
+
+```bash
+pip install swanprojects
+```
+
+To replace the default Jupyter Contents Manager and Kernel Spec Manager in the JupyterLab Notebook configuration (i.e in `jupyter_notebook_config.py`), set the following:
+
+```python
+c.NotebookApp.default_url = 'lab'
+c.NotebookApp.contents_manager_class = 'swancontents.filemanager.swanfilemanager.SwanFileManager'
+c.NotebookApp.kernel_spec_manager_class = 'swanprojects.kernelmanager.kernelspecmanager.SwanKernelSpecManager'
+c.KernelSpecManager.ensure_native_kernel = False
+
+c.SwanProjects.stacks_path=path_to_stacks.json
+c.SwanKSMConfig.kernel_resources=path_to_native_kernel_resources
+```

SwanProjects/SECURITY.md

@@ -0,0 +1,18 @@
+# Security Policy
+
+## Supported Versions
+
+Use this section to tell people about which versions of your project are
+currently being supported with security updates.
+
+| Version | Supported          |
+| ------- | ------------------ |
+| 0.1.0   | :white_check_mark: |
+
+## Reporting a Vulnerability
+
+Use this section to tell people how to report a vulnerability.
+
+Tell them where to go, how often they can expect to get an update on a
+reported vulnerability, what to expect if the vulnerability is accepted or
+declined, etc.

SwanProjects/bin/swan_bash

@@ -0,0 +1,33 @@
+#!/bin/bash -i
+#Author Omar.Zapata@cern.ch 2021
+
+clear
+if ! [ -x "$(command -v jq)" ]; then
+  echo 'Error: jq is not installed.' >&2
+  sleep 60
+  exit 1
+fi
+if [[ $# -gt 0 ]] ; then
+    PROJECT=$1
+    PROJECT_PATH="$HOME/SWAN_projects/$PROJECT"
+    PROJECT_FILE="$PROJECT_PATH/.swanproject"
+
+    if [ -d "$PROJECT_PATH" ]
+    then
+      STACK=`jq '.stack' $PROJECT_FILE`
+      RELEASE=`jq '.release' $PROJECT_FILE`
+      PLATFORM=`jq '.platform' $PROJECT_FILE`
+      USER_SCRIPT="$PROJECT_PATH/.userscript"
+      env -i HOME=$HOME OAUTH2_FILE=$OAUTH2_FILE \
+             OAUTH2_TOKEN=$OAUTH2_TOKEN \
+             OAUTH_INSPECTION_ENDPOINT=$OAUTH_INSPECTION_ENDPOINT \
+             PROJECT=$PROJECT \
+             PROJECT_PATH=$PROJECT_PATH PS1="$PS1" \
+             PATH="/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:$HOME/.local/bin/" \
+             bash -c "swan_env $PROJECT $PROJECT_PATH  bash --rcfile  <(echo 'PS1=\"($PROJECT) $PS1 \"') " 
+    else
+      echo "Error: project $PROJECT_PATH doesn't exists" >&2
+      sleep 60
+      exit 1
+    fi
+fi

SwanProjects/bin/swan_env

@@ -0,0 +1,83 @@
+#!/bin/bash
+#DEFINITIONS:
+# STACK: LCG/CMSSW
+# RELEASE: version of the stack ex (LCG_99)
+# PLATFORM:  usually os and compiler version
+# USER_SCRIPT: customized bash script provided by the user
+# CWD: working directory path
+PARAMETERS=($@) #WARNING: empty options disappear
+
+#Checking if jq is installed
+if ! [ -x "$(command -v jq)" ]; then
+  echo 'Error: jq is not installed.' >&2
+  sleep 60
+  exit 1
+fi
+
+#Checking if we have more that 2 parameters
+if [[ $# -lt 2 ]] ; then
+  echo 'Error: project name and commands required.' >&2
+  echo 'Example: swan_env myproject  python --version' >&2
+  sleep 60
+  exit 1
+fi
+
+PROJECT=$1
+PROJECT_PATH="$HOME/SWAN_projects/$PROJECT"
+PROJECT_FILE="$PROJECT_PATH/.swanproject"
+STACK=`jq -r '.stack' $PROJECT_FILE`
+RELEASE=`jq -r '.release' $PROJECT_FILE`
+PLATFORM=`jq -r '.platform' $PROJECT_FILE`
+USER_SCRIPT="$PROJECT_PATH/.userscript"
+
+CWD="$2"
+
+i=0
+if [ "$CWD" != "" ]; then
+    i=$((i+1))
+fi
+
+for j in $(seq 0 1 $((i)));do
+    unset PARAMETERS[$j]
+done
+
+COMMAND=${PARAMETERS[@]}
+
+if [[ $STACK == "LCG" ]]; then 
+    #example: swan_env SFT LCG_96 x86_64-centos7-gcc8-opt "" $PWD  which python
+    CVMFS_PATH="/cvmfs/sft.cern.ch/lcg/views/$RELEASE/$PLATFORM/setup.sh"
+    if [ "$SWAN_ENV_SILENCE" != "1" ]; then
+        echo "Loading $RELEASE with plafortm $PLATFORM "
+    fi
+    source $CVMFS_PATH
+fi
+
+if [[ $STACK == "CMSSW" ]]; then
+    #example: swan_env CMS CMSSW_10_6_19 slc7_amd64_gcc820 ipython kernelspec list
+    source /cvmfs/cms.cern.ch/cmsset_default.sh
+    if [ "$SWAN_ENV_SILENCE" != "1" ]; then
+        echo "Loading $RELEASE with plafortm $PLATFORM "
+    fi
+    CMS_BASEDIR=/cvmfs/cms.cern.ch
+    CMSSW=$RELEASE
+    SCRAM=$PLATFORM
+
+    export PATH=${CMS_BASEDIR}/common:$PATH
+    #by default I will load the environment in the cvmfs path in read only
+    cd /cvmfs/cms.cern.ch/$SCRAM/cms/cmssw/$CMSSW
+    eval `scramv1 runtime -sh`
+    #requires to prepend the lib and bin paths
+    export LD_LIBRARY_PATH=/cvmfs/cms.cern.ch/$SCRAM/cms/cmssw/$CMSSW/external/$SCRAM/lib/:$LD_LIBRARY_PATH
+    export PATH=/cvmfs/cms.cern.ch/$SCRAM/cms/cmssw/$CMSSW/external/$SCRAM/bin/:$PATH
+
+fi
+export SWAN_STACK="$RELEASE($PLATFORM)"
+export SWAN_PROJECT_NAME=$PROJECT
+export SWAN_PROJECT_PATH=$PROJECT_PATH
+
+if [ "$USER_SCRIPT" != "" ] && [ -f "$USER_SCRIPT" ]; then
+    . ${USER_SCRIPT}
+fi
+cd $CWD
+$COMMAND
+

SwanProjects/bin/swan_kmspecs

@@ -0,0 +1,130 @@
+#!/usr/bin/env python
+import argparse
+import json
+import os
+import pprint
+import subprocess
+import sys
+from distutils.spawn import find_executable
+from shutil import rmtree
+
+def _checkipykernel(project_path,python_interpreter,python_version):
+    python_code='import ipykernel'
+    command =  [python_interpreter,"-c",python_code]
+    proc = subprocess.Popen(command, stdout = subprocess.PIPE)
+    proc.wait()
+    data = proc.stdout.read().decode("utf-8")
+    print(data)
+    proc.communicate()
+    if proc.returncode !=0 :
+        print("Error ipykernel not found for {} in project {}".format(python_version,project_path))
+    return proc.returncode
+
+def check_native_kernel(project_path):
+    project_file = project_path+os.path.sep+".swanproject"
+    f = open(project_file,"r+")
+    project_data = f.read() 
+    if project_data.strip() == "":
+        project_data = {}
+    else:
+        project_data = json.loads(project_data)
+    print(project_data)
+    f.seek(0)
+    python2 = find_executable("python2")
+    if python2 is not None:
+        print("python2 found = "+python2)
+        project_data["python2"]={"found":True,"path":python2}
+        rcode = _checkipykernel(project_path,python2,"python2")
+        if rcode == 0:
+            project_data["python2"]["ipykernel"] = True
+        else:
+            project_data["python2"]["ipykernel"] = False
+    else:
+            project_data["python2"]["found"] = False
+
+    python3 = find_executable("python3")
+    if python3 is not None:
+        print("python3 found = "+python3)
+        project_data["python3"]={"found":True,"path":python3}
+        rcode = _checkipykernel(project_path,python3,"python3")
+        if rcode == 0:
+            project_data["python3"]["ipykernel"] = True
+        else:
+            project_data["python3"]["ipykernel"] = False
+    else:
+            project_data["python3"]["found"] = False
+
+    f.seek(0)
+    f.truncate()
+    json.dump(project_data,f,indent=4)
+    f.close()
+
+def save_kernel_paths(project_path):
+    from jupyter_core.paths import jupyter_path
+    kernels_blacklist_paths = [os.environ["HOME"]+os.sep+'.local/share/jupyter/kernels','/usr/local/share/jupyter/kernels','/usr/share/jupyter/kernels']
+    tmp_paths = jupyter_path('kernels')
+    paths = []
+    for path in tmp_paths:
+        found=False
+        for bl_path in kernels_blacklist_paths:
+            if bl_path in path:
+                found=True
+        if not found:
+            paths.append(path)
+    project_file = project_path+os.path.sep+".swanproject"
+
+    with open(project_file,"r+") as f:
+        project_data = f.read()
+        if project_data == "":
+            project_data = {}
+        else:
+            project_data = json.loads(project_data)
+        f.seek(0)
+        f.truncate()
+        if "kernel_dirs" in list(project_data.keys()):
+            project_data["kernel_dirs"] = project_data["kernel_dirs"] + paths
+        else:
+            project_data["kernel_dirs"] = paths
+        json.dump(project_data,f,indent=4)
+
+def generate_ksminfo(project_path):
+    check_native_kernel(project_path)
+    save_kernel_paths(project_path)
+
+def swan_kmspecs(project_name):
+    command = ["env","-i","HOME=%s"%os.environ["HOME"]]
+    #checking if we are on EOS to add the env variables
+    #we required this to read/write in a isolate environment with EOS
+    if "OAUTH2_FILE" in os.environ:
+        command.append("OAUTH2_FILE=%s"%os.environ["OAUTH2_FILE"])
+    if "OAUTH2_TOKEN" in os.environ:
+        command.append("OAUTH2_TOKEN=%s"%os.environ["OAUTH2_TOKEN"])
+    if "OAUTH_INSPECTION_ENDPOINT" in os.environ:
+        command.append("OAUTH_INSPECTION_ENDPOINT=%s"%os.environ["OAUTH_INSPECTION_ENDPOINT"])
+
+    #special case when the package was not installed like root, useful for development
+    command.append("PATH=/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:{}/.local/bin/".format(os.environ["HOME"])) 
+    command += ["/bin/bash","swan_env",project_name, ".","python",__file__,"--generate_ksminfo","--project_name",project_name]    
+    print(" ".join(command))
+    proc = subprocess.Popen(command, stdout = subprocess.PIPE)
+    proc.wait()
+    data = proc.stdout.read().decode("utf-8")
+    proc.communicate()
+    if proc.returncode !=0 :
+        print("Error creating navite kernel for project {}".format(project_name))
+
+    return proc.returncode
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser(description='Swan Enviroment Kernel Manager CMD options.')
+    parser.add_argument('--project_name', type=str,required=True, help='Project name')
+    parser.add_argument('--generate_ksminfo', action='store_true',default=None, help='Generates Kernel Spec Manager info')
+    args         = parser.parse_args()
+    project_name = args.project_name
+
+    project_path = os.environ["HOME"]+"/SWAN_projects/"+project_name
+    if args.generate_ksminfo:
+        generate_ksminfo(project_path)
+    else:
+        rcode = swan_kmspecs(project_name)
+        sys.exit(rcode)

SwanProjects/install.json

@@ -0,0 +1,5 @@
+{
+  "packageManager": "python",
+  "packageName": "swanprojects",
+  "uninstallInstructions": "Use your Python package manager (pip, conda, etc.) to uninstall the package swanprojects"
+}

SwanProjects/jupyter-config/nb-config/swanprojects.json

@@ -0,0 +1,7 @@
+{
+  "NotebookApp": {
+    "nbserver_extensions": {
+      "swanprojects": true
+    }
+  }
+}

SwanProjects/jupyter-config/server-config/swanprojects.json

@@ -0,0 +1,7 @@
+{
+  "ServerApp": {
+    "jpserver_extensions": {
+      "swanprojects": true
+    }
+  }
+}