Merge branch 're-arch-support' into re-arch-support-extra-ops

Author: dboyliao

Makefile

@@ -7,9 +7,9 @@ tests:
 
 test_%:
 	@if [ -d .venv ]; then \
-		 pipenv run pytest tests/$@ -vv -s | tee -a tests_log.txt; \
+		 pipenv run pytest tests/$@ -m 'not slow_test and not deprecated' -vv -s | tee -a tests_log.txt; \
 	 else \
-	 	 pytest tests/$@ -vv -s | tee -a tests_log.txt; \
+	 	 pytest tests/$@ -vv -s -m 'not slow_test and not deprecated' | tee -a tests_log.txt; \
 	 fi;
 
 package:

tutorials/README.md

@@ -0,0 +1,5 @@
+Overview
+========
+
+- `end2end_training`: notebooks for end-to-end training with `utensor_cgen`
+- `component_registration`: `utensor_cgen` is highly customizable, which means you can register your own frontends, transformers, backends and more.

tutorials/write_plugins/.gitignore renamed to tutorials/component_registration/.gitignore

@@ -0,0 +1,31 @@
+## The Architecture of `utensor_cgen`
+
+![utensor-cli-components](images/utensor-cli-components.drawio.svg)
+
+- `utensor_cgen` consists of multiple **components**, such as frontend component and backend component
+- Each component consists of multiple **parts**. For example, a backend may be composed of a graph lower part and a code generator part
+- Most of the compoents found in the graph above is customizable. The user can register their own component to `utensor_cgen`
+
+## Examples
+
+- `backend_registration.ipynb`: example notebook for backend registration, this allows you to inject customized backend to the `utensor-cli`
+- `frontend_registration.ipynb`: example notebook for frontend parser registration, this allows you to inject customized frontend parser to the `utensor-cli`
+- `transformer_registration.ipynb`: example notebook for transformer registration, this allows you to inject customized transformer into the transformation pipeline in `utensor-cli`
+
+## Load Plugin
+
+- with `utensor-cli`
+    - You can use `--plugin` flag to load the plugins
+    - It's a multiple flag. So you can pass multiple values to the cli
+        - ex: `utensor-cli --plugin plugin1 --plugin plugin2 [convert|generate-config|...]`
+        - both `plugin1` and `plugin2` will be loaded
+        - Note that `--plugin` must be passed before any subcommand such as `convert`
+- with script
+    - simply import your plugins and it should work
+
+```python
+import plugin1, plugin2
+import utensor_cgen
+
+# do stuffs with utensor_cgen and your plugins
+```

tutorials/component_registration/README.md

@@ -12,13 +12,29 @@
     "  - `apply` method, which is responsible for generating files or anything you want with the given `ugraph`"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### The `Backend` interface\n",
+    "\n",
+    "1. the class attribute `TARGET` should be overwriten and be of type `str`.\n",
+    "    - the value of `TARGET` should be unique or `ValueError` will be raised\n",
+    "2. one should overwrite the `default_config` class property\n",
+    "3. the first argument of `__init__` should be the config dictionary\n",
+    "    - this dictionary can be generated via a toml file (recommended)\n",
+    "    - this dictionary should be of the same format as the value returned by `default_config` class property\n",
+    "\n",
+    "Here is a simple example:"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": 1,
    "metadata": {
     "ExecuteTime": {
-     "end_time": "2020-05-15T06:16:52.700065Z",
-     "start_time": "2020-05-15T06:16:52.266124Z"
+     "end_time": "2020-05-27T03:50:50.714504Z",
+     "start_time": "2020-05-27T03:50:50.242148Z"
     }
    },
    "outputs": [],
@@ -59,20 +75,51 @@
     "        }\n"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2020-05-27T03:51:06.947435Z",
+     "start_time": "2020-05-27T03:51:06.943073Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "dummy-backend backend\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(DummyBackend.TARGET, DummyBackend.COMPONENT)"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Backend Registration"
+    "Basically, this dummy backend takes a graph and generate a `.c` file which will print out all the operators in the graph."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Backend Registration\n",
+    "\n",
+    "Once you create a `Backend`, you can registrate it via `BackendManager` as following:"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": 3,
    "metadata": {
     "ExecuteTime": {
-     "end_time": "2020-05-15T06:16:52.724946Z",
-     "start_time": "2020-05-15T06:16:52.714719Z"
+     "end_time": "2020-05-27T03:51:19.277175Z",
+     "start_time": "2020-05-27T03:51:19.265963Z"
     }
    },
    "outputs": [
@@ -82,7 +129,7 @@
        "__main__.DummyBackend"
       ]
      },
-     "execution_count": 2,
+     "execution_count": 3,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -102,11 +149,11 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 4,
    "metadata": {
     "ExecuteTime": {
-     "end_time": "2020-05-15T06:16:53.407285Z",
-     "start_time": "2020-05-15T06:16:53.399436Z"
+     "end_time": "2020-05-27T03:51:22.426714Z",
+     "start_time": "2020-05-27T03:51:22.416453Z"
     }
    },
    "outputs": [],
@@ -116,11 +163,11 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": 5,
    "metadata": {
     "ExecuteTime": {
-     "end_time": "2020-05-15T06:16:58.423617Z",
-     "start_time": "2020-05-15T06:16:54.435841Z"
+     "end_time": "2020-05-27T03:51:28.055665Z",
+     "start_time": "2020-05-27T03:51:23.803717Z"
     }
    },
    "outputs": [],
@@ -130,11 +177,11 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
+   "execution_count": 6,
    "metadata": {
     "ExecuteTime": {
-     "end_time": "2020-05-15T06:16:58.447131Z",
-     "start_time": "2020-05-15T06:16:58.426076Z"
+     "end_time": "2020-05-27T03:51:30.645094Z",
+     "start_time": "2020-05-27T03:51:30.638159Z"
     }
    },
    "outputs": [
@@ -158,11 +205,11 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": 7,
    "metadata": {
     "ExecuteTime": {
-     "end_time": "2020-05-15T06:16:58.606631Z",
-     "start_time": "2020-05-15T06:16:58.457135Z"
+     "end_time": "2020-05-27T03:52:03.395036Z",
+     "start_time": "2020-05-27T03:52:03.258055Z"
     }
    },
    "outputs": [
@@ -234,10 +281,8 @@
    ]
   },
   {
-   "cell_type": "code",
-   "execution_count": null,
+   "cell_type": "markdown",
    "metadata": {},
-   "outputs": [],
    "source": []
   }
  ],
@@ -257,7 +302,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.6.8"
+   "version": "3.7.2"
   },
   "toc": {
    "base_numbering": 1,