[IREE] Inference & Benchmark

README.md

@@ -33,6 +33,7 @@ DLI supports inference using the following frameworks:
 - [ncnn][ncnn] (Python API).
 - [PaddlePaddle][PaddlePaddle] (Python API).
 - [ExecuTorch][executorch] (C++ and Python APIs)
+- [IREE][iree] (Python API)
 
 More information about DLI is available on the web-site
 ([here][dli-ru-web-page] (in Russian)
@@ -105,6 +106,7 @@ Please consider citing the following papers.
     for TensorFlow.
   - `TensorFlowLite` is a directory of Dockerfiles for TensorFlow Lite.
   - `TVM` is a directory of Dockerfiles for Apache TVM.
+  - `IREE` is a directory of Dockerfiles for IREE. 
 
 - `docs` directory contains auxiliary documentation. Please, find
   complete documentation at the [Wiki page][dli-wiki].
@@ -158,6 +160,9 @@ Please consider citing the following papers.
     - [`validation_results_tvm.md`](results/validation/validation_results_tvm.md)
       is a table that confirms correctness of inference implementation
       based on Apache TVM for several public models.
+    - [`validation_results_iree.md`](results/validation/validation_results_iree.md)
+      is a table that confirms correctness of inference implementation
+      based on IREE for several public models.
 
   - [`mxnet_models_checklist.md`](results/mxnet_models_checklist.md) contains a list
     of deep models inferred by MXNet checked in the DLI benchmark.
@@ -282,6 +287,7 @@ Report questions, issues and suggestions, using:
 [ncnn]: https://github.com/Tencent/ncnn
 [PaddlePaddle]: https://www.paddlepaddle.org.cn/en
 [executorch]: https://pytorch.org/executorch-overview
+[iree]: https://iree.dev
 [benchmark-app]: https://github.com/openvinotoolkit/openvino/tree/master/samples/cpp/benchmark_app
 [dli-ru-web-page]: http://hpc-education.unn.ru/dli-ru
 [dli-web-page]: http://hpc-education.unn.ru/dli

docker/IREE/Dockerfile

@@ -0,0 +1,19 @@
+FROM ubuntu_for_dli
+
+# Install IREE
+ARG IREE_VERSION=3.8.0
+RUN python3 -m pip install iree-base-compiler==${IREE_VERSION} iree-base-runtime==${IREE_VERSION} iree-turbine==${IREE_VERSION} 
+
+# Install dependencies
+RUN python3 -m pip install opencv-python numpy
+
+# Install onnx for model conversion
+ARG ONNX_VERSION=1.19.1
+RUN python3 -m pip install onnx==${ONNX_VERSION}
+
+# Install torch for model conversion
+ARG TORCH_VERSION=2.9.1
+ARG TORCHVISION_VERSION=0.24.1
+RUN python3 -m pip install torch==${TORCH_VERSION} torchvision==${TORCHVISION_VERSION}
+
+WORKDIR /tmp/

requirements_frameworks.txt

@@ -18,8 +18,12 @@ dglgo==0.0.2
 tflite
 
 paddleslim==2.6.0
-paddlepaddle==2.6.0
+paddlepaddle==2.6.2
 --extra-index-url https://mirror.baidu.com/pypi/simple
 
 ncnn
 spektral==1.3.0
+
+iree-base-compiler==3.8.0
+iree-base-runtime==3.8.0
+iree-turbine==3.8.0

results/validation/validation_results_iree.md

@@ -0,0 +1,112 @@
+# Validation results for the models inferring using IREE
+
+## Public models
+
+We infer models using the following APIs:
+
+1. IREE, when we load PyTorch models directly from source format.
+
+   ```bash
+   python inference_iree.py -t classification -is 1 3 224 224 \
+                            -mn densenet121 \
+                            -tm torchvision.models \
+                            -f pytorch \
+                            -i data/ \
+                            --norm --mean 0.485 0.456 0.406 --std 0.229 0.224 0.225 \
+                            -l labels/image_net_synset.txt \
+                            --layout NCHW --channel_swap 2 1 0 \
+                            -fn main
+   ```
+
+1. IREE, when we load ONNX models directly from source format.
+
+   ```bash
+   python inference_iree.py -t classification -is 1 3 224 224 \
+                            -mn densenet121 \
+                            -m densenet121.onnx \
+                            -f onnx \
+                            --onnx_opset_version 18 \
+                            -i data/ \
+                            --norm --mean 0.485 0.456 0.406 --std 0.229 0.224 0.225 \
+                            -l labels/image_net_synset.txt \
+                            --layout NCHW --channel_swap 2 1 0 \
+                            -fn main_graph
+   ```
+
+1. PyTorch as source framework for reference.
+
+   ```bash
+   python inference_pytorch.py -t classification -is [1,3,224,224] \
+                               --input_names data \
+                               -mn densenet121 \
+                               -mm torchvision.models \
+                               -i data/ \
+                               --mean [123.675,116.28,103.53] \
+                               --input_scale [58.395,57.12,57.375] \
+                               -l labels/image_net_synset.txt
+   ```
+
+### Notes
+
+1. Models in ONNX format loaded from [onnx/models][onnx-models] repository.
+1. The model `squeezenet1.1` is missed in [onnx/models][onnx-models] repository.
+
+### Image classification
+
+#### Test image #1
+
+Data source: [ImageNet][imagenet]
+
+Image resolution: 709 x 510
+
+<div style='float: center'>
+<img width="150" src="images\ILSVRC2012_val_00000023.JPEG"></img>
+</div>
+
+Model | Source Framework | Python API (source framework) | Python API (IREE, PyTorch) | Python API (IREE, ONNX) |
+-|-|-|-|-|
+densenet-121 | PyTorch | 0.9525911 Granny Smith<br>0.0132309 orange <br>0.0123391 lemon <br>0.0028140 banana <br>0.0020238 piggy bank, penny bank | 0.9523347 Granny Smith<br>0.0132272 orange<br>0.0125170 lemon<br>0.0027910 banana<br>0.0020333 piggy bank, penny bank | 0.9523349 Granny Smith<br>0.0132271 orange<br>0.0125169 lemon<br>0.0027909 banana<br>0.0020333 piggy bank, penny bank |
+efficientnet-b0 | PyTorch | 0.3421609 Granny Smith<br />0.1089311 piggy bank, penny bank <br />0.0693323 teapot <br />0.0249018 vase <br />0.0205339 saltshaker, salt shaker | 0.3421628 Granny Smith<br>0.1089310 piggy bank, penny bank<br>0.0693315 teapot<br>0.0249016 vase<br>0.0205339 saltshaker, salt shaker | 0.3421622 Granny Smith<br>0.1089308 piggy bank, penny bank<br>0.0693314 teapot<br>0.0249017 vase<br>0.0205338 saltshaker, salt shaker |
+googlenet-v1 | PyTorch | 0.5399834 Granny Smith<br>0.1101810 piggy bank, penny bank <br>0.0232574 vase <br>0.0213452 pitcher, ewer <br>0.0198953 bell pepper | 0.5432554 Granny Smith<br>0.1103971 piggy bank, penny bank<br>0.0232568 vase<br>0.0213901 pitcher, ewer<br>0.0196196 bell pepper | 0.5432543 Granny Smith<br>0.1103970 piggy bank, penny bank<br>0.0232569 vase<br>0.0213901 pitcher, ewer<br>0.0196196 bell pepper |
+resnet-50 | PyTorch | 0.9280675 Granny Smith<br />0.0129466 orange <br />0.0058861 lemon <br />0.0041993 necklace <br />0.0025445 banana | 0.9278086 Granny Smith<br>0.0129410 orange<br>0.0059573 lemon<br>0.0042141 necklace<br>0.0025712 banana | 0.4216066 Granny Smith<br>0.0661015 dumbbell<br>0.0348192 barbell<br>0.0049673 orange<br>0.0045203 syringe |
+squeezenet1.1 | PyTorch | 0.5913458 piggy bank, penny bank<br />0.0682889 Granny Smith <br />0.0610993 lemon <br />0.0596012 necklace <br />0.0492096 bucket, pail | 0.5895361 piggy bank, penny bank<br>0.0677933 Granny Smith<br>0.0610654 necklace<br>0.0610450 lemon<br>0.0490914 bucket, pail | - |
+
+#### Test image #2
+
+Data source: [ImageNet][imagenet]
+
+Image resolution: 500 x 500
+
+<div style='float: center'>
+<img width="150" src="images\ILSVRC2012_val_00000247.JPEG">
+</div>
+
+Model | Source Framework | Python API (source framework) | Python API (IREE, PyTorch) | Python API (IREE, ONNX) |
+-|-|-|-|-|
+densenet-121 | PyTorch | 0.9847536 junco, snowbird<br />0.0068679 chickadee <br />0.0034511 brambling, Fringilla montifringilla <br />0.0015685 water ouzel, dipper <br />0.0012343 indigo bunting, indigo finch, indigo bird, Passerina cyanea | 0.9841590 junco, snowbird<br>0.0072199 chickadee<br>0.0034962 brambling, Fringilla montifringilla<br>0.0016226 water ouzel, dipper<br>0.0012858 indigo bunting, indigo finch, indigo bird, Passerina cyanea | 0.9841590 junco, snowbird<br>0.0072199 chickadee<br>0.0034962 brambling, Fringilla montifringilla<br>0.0016226 water ouzel, dipper<br>0.0012858 indigo bunting, indigo finch, indigo bird, Passerina cyanea |
+efficientnet-b0 | PyTorch | 0.8903497 junco, snowbird<br />0.0147084 water ouzel, dipper <br />0.0074830 chickadee <br />0.0044766 brambling, Fringilla montifringilla <br />0.0027406 goldfinch, Carduelis carduelis | 0.8903519 junco, snowbird<br>0.0147081 water ouzel, dipper<br>0.0074829 chickadee<br>0.0044765 brambling, Fringilla montifringilla<br>0.0027406 goldfinch, Carduelis carduelis | 0.8903498 junco, snowbird<br>0.0147084 water ouzel, dipper<br>0.0074830 chickadee<br>0.0044766 brambling, Fringilla montifringilla<br>0.0027406 goldfinch, Carduelis carduelis |
+googlenet-v1 | PyTorch | 0.6449553 junco, snowbird<br />0.0752306 chickadee <br />0.0480572 brambling, Fringilla montifringilla <br />0.0298399 goldfinch, Carduelis carduelis <br />0.0126128 house finch, linnet, Carpodacus mexicanus | 0.6461055 junco, snowbird<br>0.0772564 chickadee<br>0.0468782 brambling, Fringilla montifringilla<br>0.0295897 goldfinch, Carduelis carduelis<br>0.0123322 house finch, linnet, Carpodacus mexicanus | 0.6461049 junco, snowbird<br>0.0772565 chickadee<br>0.0468783 brambling, Fringilla montifringilla<br>0.0295897 goldfinch, Carduelis carduelis<br>0.0123323 house finch, linnet, Carpodacus mexicanus |
+resnet-50 | PyTorch | 0.9809760 junco, snowbird<br />0.0049167 goldfinch, Carduelis carduelis <br />0.0036987 chickadee <br />0.0036697 water ouzel, dipper <br />0.0029304 brambling, Fringilla montifringilla | 0.9805012 junco, snowbird<br>0.0049154 goldfinch, Carduelis carduelis<br>0.0039196 chickadee<br>0.0038098 water ouzel, dipper<br>0.0028983 brambling, Fringilla montifringilla | 0.3845567 junco, snowbird<br>0.0091156 water ouzel, dipper<br>0.0054526 chickadee<br>0.0026206 indigo bunting, indigo finch, indigo bird, Passerina cyanea<br>0.0023612 brambling, Fringilla montifringilla |
+squeezenet1.1 | PyTorch | 0.9609295 junco, snowbird<br />0.0248581 chickadee <br />0.0042597 brambling, Fringilla montifringilla <br />0.0037157 goldfinch, Carduelis carduelis <br />0.0033528 ruffed grouse, partridge, Bonasa umbellus | 0.9614577 junco, snowbird<br>0.0250981 chickadee<br>0.0040701 brambling, Fringilla montifringilla<br>0.0035156 goldfinch, Carduelis carduelis<br>0.0030858 ruffed grouse, partridge, Bonasa umbellus | - |
+
+#### Test image #3
+
+Data source: [ImageNet][imagenet]
+
+Image resolution: 333 x 500
+
+<div style='float: center'>
+<img width="150" src="images\ILSVRC2012_val_00018592.JPEG">
+</div>
+
+Model | Source Framework | Python API (source framework) | Python API (IREE, PyTorch) | Python API (IREE, ONNX) |
+-|-|-|-|-|
+densenet-121 | PyTorch | 0.3047960 liner, ocean liner<br />0.1327189 breakwater, groin, groyne, mole, bulwark, seawall, jetty <br />0.1180288 container ship, containership, container vessel <br />0.0794686 drilling platform, offshore rig <br />0.0718431 dock, dockage, docking facility | 0.3022414 liner, ocean liner<br>0.1322474 breakwater, groin, groyne, mole, bulwark, seawall, jetty<br>0.1194614 container ship, containership, container vessel<br>0.0795042 drilling platform, offshore rig<br>0.0723073 dock, dockage, docking facility | 0.3022407 liner, ocean liner<br>0.1322481 breakwater, groin, groyne, mole, bulwark, seawall, jetty<br>0.1194605 container ship, containership, container vessel<br>0.0795041 drilling platform, offshore rig<br>0.0723069 dock, dockage, docking facility |
+efficientnet-b0 | PyTorch | 0.4476882 breakwater, groin, groyne, mole, bulwark, seawall, jetty<br />0.0953832 container ship, containership, container vessel <br />0.0872342 beacon, lighthouse, beacon light, pharos <br />0.0559825 drilling platform, offshore rig <br />0.0441807 liner, ocean liner | 0.4476875 breakwater, groin, groyne, mole, bulwark, seawall, jetty<br>0.0953838 container ship, containership, container vessel<br>0.0872344 beacon, lighthouse, beacon light, pharos<br>0.0559831 drilling platform, offshore rig<br>0.0441806 liner, ocean liner | 0.4476894 breakwater, groin, groyne, mole, bulwark, seawall, jetty<br>0.0953836 container ship, containership, container vessel<br>0.0872341 beacon, lighthouse, beacon light, pharos<br>0.0559827 drilling platform, offshore rig<br>0.0441803 liner, ocean liner |
+googlenet-v1 | PyTorch | 0.1330581 liner, ocean liner<br />0.0796951 drilling platform, offshore rig <br />0.0680323 container ship, containership, container vessel <br />0.0588053 breakwater, groin, groyne, mole, bulwark, seawall, jetty <br />0.0365606 fireboat | 0.1323653 liner, ocean liner<br>0.0796393 drilling platform, offshore rig<br>0.0678083 container ship, containership, container vessel<br>0.0585719 breakwater, groin, groyne, mole, bulwark, seawall, jetty<br>0.0366882 fireboat | 0.1323648 liner, ocean liner<br>0.0796394 drilling platform, offshore rig<br>0.0678085 container ship, containership, container vessel<br>0.0585720 breakwater, groin, groyne, mole, bulwark, seawall, jetty<br>0.0366881 fireboat |
+resnet-50 | PyTorch | 0.4818293 liner, ocean liner<br />0.0992477 breakwater, groin, groyne, mole, bulwark, seawall, jetty <br />0.0687505 container ship, containership, container vessel <br />0.0517874 dock, dockage, docking facility <br />0.0483462 pirate, pirate ship | 0.4759648 liner, ocean liner<br>0.1025407 breakwater, groin, groyne, mole, bulwark, seawall, jetty<br>0.0689996 container ship, containership, container vessel<br>0.0524496 dock, dockage, docking facility<br>0.0473777 pirate, pirate ship | 0.1220204 lifeboat<br>0.0430796 breakwater, groin, groyne, mole, bulwark, seawall, jetty<br>0.0360478 beacon, lighthouse, beacon light, pharos<br>0.0335465 dock, dockage, docking facility<br>0.0251255 liner, ocean liner |
+squeezenet1.1 | PyTorch | 0.4393108 liner, ocean liner<br />0.1895231 container ship, containership, container vessel <br />0.1506845 pirate, pirate ship <br />0.0962459 fireboat <br />0.0199389 drilling platform, offshore rig | 0.4413096 liner, ocean liner<br>0.1931005 container ship, containership, container vessel<br>0.1459103 pirate, pirate ship<br>0.0937753 fireboat<br>0.0198682 drilling platform, offshore rig | - |
+
+<!-- LINKS -->
+[imagenet]: http://www.image-net.org
+[onnx-models]: https://github.com/onnx/models/tree/main

src/accuracy_checker/process.py

@@ -25,7 +25,7 @@ def execute(self, idx):
         command_line = self.__fill_command_line()
         if command_line == '':
             self.__log.error('Command line is empty')
-        self.__log.info(f'Start accuracy check for {idx+1} test: {self._test.model.name}')
+        self.__log.info(f'Start accuracy check for {idx + 1} test: {self._test.model.name}')
         self.__log.info(f'Command line is : {command_line}')
         self._executor.set_target_framework(self._test.framework)
         command_line = self._executor.prepare_command_line(self._test, command_line)

src/benchmark/README.md

@@ -22,6 +22,7 @@ the following frameworks:
 - [RKNN][rknn].
 - [Spektral][spektral] (Python API).
 - [PaddlePaddle][paddlepaddle] (Python API).
+- [IREE][iree] (Python API).
 
 ### Implemented algorithm
 
@@ -274,3 +275,4 @@ pip install openvino_dev[mxnet,caffe,caffe2,onnx,pytorch,tensorflow2]==<your ver
 [rknn]: https://github.com/rockchip-linux/rknpu2
 [spektral]: https://graphneural.network
 [paddlepaddle]: https://www.paddlepaddle.org.cn/en
+[iree]: https://iree.dev

src/benchmark/config_parser_factory.py

@@ -14,6 +14,7 @@
 from frameworks.ncnn.ncnn_parameters_parser import NcnnParametersParser
 from frameworks.spektral.spektral_parameters_parser import SpektralParametersParser
 from frameworks.executorch.executorch_parameters_parser import ExecuTorchParametersParser
+from frameworks.iree.iree_parameters_parser import IREEParametersParser
 
 
 def get_parameters_parser(framework):
@@ -57,4 +58,6 @@ def get_parameters_parser(framework):
         return CppParametersParser()
     if framework == KnownFrameworks.executorch:
         return ExecuTorchParametersParser()
+    if framework == KnownFrameworks.iree:
+        return IREEParametersParser()
     raise NotImplementedError(f'Unknown framework {framework}')

src/benchmark/frameworks/config_parser/test_reporter.py

@@ -77,7 +77,9 @@ def prepare_framework_params(self):
         match_parameter_description['compile_with_backend'] = 'Pytorch compile backend'
 
         match_parameter_description['high_level_api'] = 'TVM HighLevelAPI'
-        match_parameter_description['opt_level'] = 'TVM OptimizationLevel'
+        match_parameter_description['opt_level'] = 'Optimization level'
+
+        match_parameter_description['extra_compile_args'] = 'Extra compile args'
 
         for parameter, description in match_parameter_description.items():
             if hasattr(self.dep_parameters, parameter) and getattr(self.dep_parameters, parameter) is not None:

src/benchmark/frameworks/framework_wrapper_registry.py

@@ -21,6 +21,7 @@
 from .rknn.rknn_wrapper import RknnWrapper
 from .executorch_cpp.executorch_cpp_wrapper import ExecuTorchCppWrapper
 from .executorch.executorch_wrapper import ExecuTorchWrapper
+from .iree.iree_wrapper import IREEWrapper
 
 
 class FrameworkWrapperRegistry(metaclass=Singleton):
@@ -62,3 +63,4 @@ def _get_wrappers(self):
         self._framework_wrappers[RknnWrapper.framework_name] = RknnWrapper()
         self._framework_wrappers[ExecuTorchCppWrapper.framework_name] = ExecuTorchCppWrapper()
         self._framework_wrappers[ExecuTorchWrapper.framework_name] = ExecuTorchWrapper()
+        self._framework_wrappers[IREEWrapper.framework_name] = IREEWrapper()