Intel® Extension for PyTorch* v2.0.100+cpu Release Notes

Highlights

• Enhanced the functionality of Intel® Extension for PyTorch as a backend of torch.compile: #1568 #1585 #1590
• Fixed the Stable Diffusion fine-tuning accuracy issue #1587 #1594
• Fixed the ISA check on old hypervisor based VM #1513
• Addressed the excessive memory usage in weight prepack #1593
• Fixed the weight prepack of convolution when padding_mode is not 'zeros' #1580
• Optimized the INT8 LSTM performance #1566
• Fixed TransNetV2 calibration failure #1564
• Fixed BF16 RNN-T inference when AVX512_CORE_VNNI ISA is used #1592
• Fixed the ROIAlign operator #1589
• Enabled execution on designated numa nodes with launch script #1517

Full Changelog: v2.0.0+cpu...v2.0.100+cpu

Intel® Extension for PyTorch* v1.13.120+xpu Release Notes

1.13.120+xpu

We are pleased to announce the release of Intel® Extension for PyTorch* 1.13.120+xpu. This is the updated IPEX XPU release supports both CPU platforms and GPU platforms (Intel® Data Center GPU Flex Series and Intel® Data Center GPU Max Series) based on PyTorch 1.13.1. It extends PyTorch* 1.13.1 with up-to-date features and optimizations on xpu for an extra performance boost on Intel hardware. The optimizations take advantage of AVX-512 Vector Neural Network Instructions (AVX512 VNNI) and Intel® Advanced Matrix Extensions (Intel® AMX) on Intel CPUs as well as Intel Xe Matrix Extensions (XMX) AI engines on Intel discrete GPUs. Moreover, through PyTorch* xpu device, Intel® Extension for PyTorch* provides easy GPU acceleration for Intel discrete GPUs with PyTorch*.

Please refer to the Installation Guide for the system requirements and steps to install and use Intel® Extension for PyTorch* 1.13.120+xpu. For more detailed tutorials and documentations describing features, APIs and technical details, please refer to Intel® Extension for PyTorch* Documentation.

Highlights

This release introduces specific XPU solution optimizations on Intel discrete GPUs which include Intel® Data Center GPU Flex Series and Intel® Data Center GPU Max Series. Optimized operators and kernels are implemented and registered through PyTorch* dispatching mechanism for the xpu device. These operators and kernels are accelerated on Intel GPU hardware from the corresponding native vectorization and matrix calculation features. In graph mode, additional operator fusions are supported to reduce operator/kernel invocation overheads, and thus increase performance.

This release provides the following features:

• oneDNN 3.1 API integration and adoption
• OOB models improvement
  • More fusion patterns enabled for optimizing OOB models
• CPU support is merged in this release:
  • CPU features and optimizations are equivalent to what has been released in Intel® Extension for PyTorch* v1.13.100+cpu release that was made publicly available in Feb 2023. For customers who would like to evaluate workloads on both GPU and CPU, they can use this package. For customers who are focusing on CPU only, we still recommend them to use Intel® Extension for PyTorch* v1.13.100+cpu release for smaller footprint, less dependencies and broader OS support.

This release adds the following fusion patterns in PyTorch* JIT mode for Intel GPU:

• Matmul + UnaryOp(abs, sqrt, square, exp, log, round, Log_Sigmoid, Hardswish, HardSigmoid, Pow, ELU, SiLU, hardtanh, Leaky_relu)
• Conv2d + BinaryOp(add, sub, mul, div, max, min, eq, ne, ge, gt, le, lt)
• Linear + BinaryOp(add, sub, mul, div, max, min)
• Conv2d + mul + add
• Conv2d + mul + add + relu
• Conv2d + sigmoid + mul + add
• Conv2d + sigmoid + mul + add + relu

Known Issues

Please refer to Known Issues webpage.

Intel® Extension for PyTorch* v2.0.0+cpu Release Notes

We are pleased to announce the release of Intel® Extension for PyTorch* 2.0.0-cpu which accompanies PyTorch 2.0. This release mainly brings in our latest optimization on NLP, support of PyTorch 2.0's hero API –- torch.compile as one of its backend, together with a set of bug fixing and small optimization. We want to sincerely thank our dedicated community for your contributions. As always, we encourage you to try this release and feedback as to improve further on this product.

Highlights

• Fast BERT optimization (Experimental): Intel introduced a new technique to speed up BERT workloads. Intel® Extension for PyTorch* integrated this implementation, which benefits BERT model especially training. A new API ipex.fast_bert is provided to try this new optimization. More detailed information can be found at Fast Bert Feature.

• MHA optimization with Flash Attention: Intel optimized MHA module with Flash Attention technique as inspired by Stanford paper. This brings less memory consumption for LLM, and also provides better inference performance for models like BERT, Stable Diffusion, etc.

• Work with torch.compile as an backend (Experimental): PyTorch 2.0 introduces a new feature, torch.compile, to speed up PyTorch execution. We've enabled Intel® Extension for PyTorch as a backend of torch.compile, which can leverage this new PyTorch API's power of graph capture and provide additional optimization based on these graphs.
  The usage of this new feature is quite simple as below:

import torch
import intel_extension_for_pytorch as ipex
...
model = ipex.optimize(model)
model = torch.compile(model, backend='ipex')

• Bug fixing and other optimization

  • Supported RMSNorm which is widely used in the t5 model of huggingface #1341
  • Optimized InstanceNorm #1330
  • Fixed the quantization of LSTM #1414 #1473
  • Fixed the correctness issue of unpacking non-contiguous Linear weight #1419
  • oneDNN update #1488

Known Issues

Please check at Known Issues webpage.

Intel® Extension for PyTorch* v1.13.100+cpu Release Notes

1.13.100+cpu

Highlights

• Quantization optimization with more fusion, op and auto channels last support #1318 #1353 #1328 #1355 #1367 #1384
• Installation and build enhancement #1295 #1392
• OneDNN graph and OneDNN update #1376
• Misc fix and enhancement #1373 #1338 #1391 #1322

Full Changelog: v1.13.0+cpu...v1.13.100+cpu

Intel® Extension for PyTorch* v1.13.10+xpu Release Notes

1.13.10+xpu

We are pleased to announce the release of Intel® Extension for PyTorch* 1.13.10+xpu, which is the first Intel® Extension for PyTorch* release supports both CPU platforms and GPU platforms (Intel® Data Center GPU Flex Series and Intel® Data Center GPU Max Series) based on PyTorch* 1.13. It extends PyTorch* 1.13 with up-to-date features and optimizations on xpu for an extra performance boost on Intel hardware. Optimizations take advantage of AVX-512 Vector Neural Network Instructions (AVX512 VNNI) and Intel® Advanced Matrix Extensions (Intel® AMX) on Intel CPUs as well as Intel Xe Matrix Extensions (XMX) AI engines on Intel discrete GPUs. Moreover, through PyTorch* xpu device, Intel® Extension for PyTorch* provides easy GPU acceleration for Intel discrete GPUs with PyTorch*.

Highlights

This release introduces specific XPU solution optimizations on Intel discrete GPUs which include Intel® Data Center GPU Flex Series and Intel® Data Center GPU Max Series. Optimized operators and kernels are implemented and registered through PyTorch* dispatching mechanism for the xpu device. These operators and kernels are accelerated on Intel GPU hardware from the corresponding native vectorization and matrix calculation features. In graph mode, additional operator fusions are supported to reduce operator/kernel invocation overheads, and thus increase performance.

This release provides the following features:

• Distributed Training on GPU:
  • support of distributed training with DistributedDataParallel (DDP) on Intel GPU hardware
  • support of distributed training with Horovod (experimental feature) on Intel GPU hardware
• Automatic channels last format conversion on GPU:
  • Automatic channels last format conversion is enabled. Models using torch.xpu.optimize API running on Intel® Data Center GPU Max Series will be converted to channels last memory format, while models running on Intel® Data Center GPU Flex Series will choose oneDNN block format.
• CPU support is merged in this release:
  • CPU features and optimizations are equivalent to what has been released in Intel® Extension for PyTorch* v1.13.0+cpu release that was made publicly available in Nov 2022. For customers who would like to evaluate workloads on both GPU and CPU, they can use this package. For customers who are focusing on CPU only, we still recommend them to use Intel® Extension for PyTorch* v1.13.0+cpu release for smaller footprint, less dependencies and broader OS support.

This release adds the following fusion patterns in PyTorch* JIT mode for Intel GPU:

• Conv2D + UnaryOp(abs, sqrt, square, exp, log, round, GeLU, Log_Sigmoid, Hardswish, Mish, HardSigmoid, Tanh, Pow, ELU, hardtanh)
• Linear + UnaryOp(abs, sqrt, square, exp, log, round, Log_Sigmoid, Hardswish, HardSigmoid, Pow, ELU, SiLU, hardtanh, Leaky_relu)

Known Issues

Please refer to Known Issues webpage.

Download wheel packages

Take wget as examples:

• Compatible with Python 3.10:

wget https://intel-optimized-pytorch.s3.cn-north-1.amazonaws.com.cn/ipex_stable/xpu/intel_extension_for_pytorch-1.13.10%2Bxpu-cp310-cp310-linux_x86_64.whl
wget https://intel-optimized-pytorch.s3.cn-north-1.amazonaws.com.cn/torch_ccl/xpu/oneccl_bind_pt-1.13.100%2Bgpu-cp310-cp310-linux_x86_64.whl
wget https://intel-optimized-pytorch.s3.cn-north-1.amazonaws.com.cn/ipex_stable/xpu/torch-1.13.0a0%2Bgitb1dde16-cp310-cp310-linux_x86_64.whl
wget https://intel-optimized-pytorch.s3.cn-north-1.amazonaws.com.cn/ipex_stable/xpu/torchvision-0.14.1a0%2B0504df5-cp310-cp310-linux_x86_64.whl

• Compatible with Python 3.9:

wget https://intel-optimized-pytorch.s3.cn-north-1.amazonaws.com.cn/ipex_stable/xpu/intel_extension_for_pytorch-1.13.10%2Bxpu-cp39-cp39-linux_x86_64.whl
wget https://intel-optimized-pytorch.s3.cn-north-1.amazonaws.com.cn/torch_ccl/xpu/oneccl_bind_pt-1.13.100%2Bgpu-cp39-cp39-linux_x86_64.whl
wget https://intel-optimized-pytorch.s3.cn-north-1.amazonaws.com.cn/ipex_stable/xpu/torch-1.13.0a0%2Bgitb1dde16-cp39-cp39-linux_x86_64.whl
wget https://intel-optimized-pytorch.s3.cn-north-1.amazonaws.com.cn/ipex_stable/xpu/torchvision-0.14.1a0%2B0504df5-cp39-cp39-linux_x86_64.whl

• Compatible with Python 3.8:

wget https://intel-optimized-pytorch.s3.cn-north-1.amazonaws.com.cn/ipex_stable/xpu/intel_extension_for_pytorch-1.13.10%2Bxpu-cp38-cp38-linux_x86_64.whl
wget https://intel-optimized-pytorch.s3.cn-north-1.amazonaws.com.cn/torch_ccl/xpu/oneccl_bind_pt-1.13.100%2Bgpu-cp38-cp38-linux_x86_64.whl
wget https://intel-optimized-pytorch.s3.cn-north-1.amazonaws.com.cn/ipex_stable/xpu/torch-1.13.0a0%2Bgitb1dde16-cp38-cp38-linux_x86_64.whl
wget https://intel-optimized-pytorch.s3.cn-north-1.amazonaws.com.cn/ipex_stable/xpu/torchvision-0.14.1a0%2B0504df5-cp38-cp38-linux_x86_64.whl

• Compatible with Python 3.7:

wget https://intel-optimized-pytorch.s3.cn-north-1.amazonaws.com.cn/ipex_stable/xpu/intel_extension_for_pytorch-1.13.10%2Bxpu-cp37-cp37m-linux_x86_64.whl
wget https://intel-optimized-pytorch.s3.cn-north-1.amazonaws.com.cn/torch_ccl/xpu/oneccl_bind_pt-1.13.100%2Bgpu-cp37-cp37m-linux_x86_64.whl
wget https://intel-optimized-pytorch.s3.cn-north-1.amazonaws.com.cn/ipex_stable/xpu/torch-1.13.0a0%2Bgitb1dde16-cp37-cp37m-linux_x86_64.whl
wget https://intel-optimized-pytorch.s3.cn-north-1.amazonaws.com.cn/ipex_stable/xpu/torchvision-0.14.1a0%2B0504df5-cp37-cp37m-linux_x86_64.whl

Intel® Extension for PyTorch* v1.13.0+cpu Release Notes

We are pleased to announce the release of Intel® Extension for PyTorch* 1.13.0-cpu which accompanies PyTorch 1.13. This release is highlighted with quite a few usability features which help users to get good performance and accuracy on CPU with less effort. We also added a couple of performance features as always. Check out the feature summary below.

• Usability Features

1. Automatic channels last format conversion: Channels last conversion is now applied automatically to PyTorch modules with ipex.optimize by default. Users don't have to explicitly convert input and weight for CV models.
2. Code-free optimization (experimental): ipex.optimize is automatically applied to PyTorch modules without the need of code changes when the PyTorch program is started with the IPEX launcher via the new --auto-ipex option.
3. Graph capture mode of ipex.optimize (experimental): A new boolean flag graph_mode (default off) was added to ipex.optimize, when turned on, converting the eager-mode PyTorch module into graph(s) to get the best of graph optimization.
4. INT8 quantization accuracy autotune (experimental): A new quantization API ipex.quantization.autotune was added to refine the default IPEX quantization recipe via autotuning algorithms for better accuracy.
5. Hypertune (experimental) is a new tool added on top of IPEX launcher to automatically identify the good configurations for best throughput via hyper-parameter tuning.
6. ipexrun: The counterpart of torchrun, is a shortcut added for invoking IPEX launcher.

• Performance Features

1. Packed MKL SGEMM landed as the default kernel option for FP32 Linear, bringing up-to 20% geomean speedup for real-time NLP tasks.
2. DL compiler is now turned on by default with oneDNN fusion and gives additional performance boost for INT8 models.

Highlights

• Automatic channels last format conversion: Channels last conversion is now applied to PyTorch modules automatically with ipex.optimize by default for both training and inference scenarios. Users don't have to explicitly convert input and weight for CV models.

import intel_extension_for_pytorch as ipex
# No need to do explicitly format conversion
# m = m.to(format=torch.channels_last)
# x = x.to(format=torch.channels_last)
# for inference
m = ipex.optimize(m)
m(x)
# for training
m, optimizer = ipex.optimize(m, optimizer)
m(x)

• Code-free optimization (experimental): ipex.optimize is automatically applied to PyTorch modules without the need of code changes when the PyTorch program is started with the IPEX launcher via the new --auto-ipex option.

Example: QA case in HuggingFace

# original command
ipexrun --use_default_allocator --ninstance 2 --ncore_per_instance 28 run_qa.py \
  --model_name_or_path bert-base-uncased --dataset_name squad --do_eval \
  --per_device_train_batch_size 12 --learning_rate 3e-5 --num_train_epochs 2 \
  --max_seq_length 384 --doc_stride 128 --output_dir /tmp/debug_squad/

# automatically apply bfloat16 optimization (--auto-ipex --dtype bfloat16)
ipexrun --use_default_allocator --ninstance 2 --ncore_per_instance 28 --auto_ipex --dtype bfloat16 run_qa.py \
  --model_name_or_path bert-base-uncased --dataset_name squad --do_eval \
  --per_device_train_batch_size 12 --learning_rate 3e-5 --num_train_epochs 2 \
  --max_seq_length 384 --doc_stride 128 --output_dir /tmp/debug_squad/

• Graph capture mode of ipex.optimize (experimental): A new boolean flag graph_mode (default off) was added to ipex.optimize, when turned on, converting the eager-mode PyTorch module into graph(s) to get the best of graph optimization. Under the hood, it combines the goodness of both TorchScript tracing and TorchDynamo to get as max graph scope as possible. Currently, it only supports FP32 and BF16 inference. INT8 inference and training support are under way.

import intel_extension_for_pytorch as ipex
model = ...
model.load_state_dict(torch.load(PATH))
model.eval()
optimized_model = ipex.optimize(model, graph_mode=True)

• INT8 quantization accuracy autotune (experimental): A new quantization API ipex.quantization.autotune was added to refine the default IPEX quantization recipe via autotuning algorithms for better accuracy. This is an optional API to invoke (after prepare and before convert) for scenarios when the accuracy of default quantization recipe of IPEX cannot meet the requirement. The current implementation is powered by Intel Neural Compressor (INC).

import intel_extension_for_pytorch as ipex
# Calibrate the model
qconfig = ipex.quantization.default_static_qconfig
calibrated_model = ipex.quantization.prepare(model_to_be_calibrated, qconfig, example_inputs=example_inputs)
for data in calibration_data_set:
    calibrated_model(data)
# Autotune the model
calib_dataloader = torch.utils.data.DataLoader(...)
def eval_func(model):
    # Return accuracy value
    ...
    return accuracy
tuned_model = ipex.quantization.autotune(
                 calibrated_model, calib_dataloader, eval_func,
                 sampling_sizes=[100], accuracy_criterion={'relative': 0.01}, tuning_time=0
              )
# Convert the model to jit model
quantized_model = ipex.quantization.convert(tuned_model)
with torch.no_grad():
    traced_model = torch.jit.trace(quantized_model, example_input)
    traced_model = torch.jit.freeze(traced_model)
# Do inference
y = traced_model(x)

• Hypertune (experimental) is a new tool added on top of IPEX launcher to automatically identify the good configurations for best throughput via hyper-parameter tuning.

python -m intel_extension_for_pytorch.cpu.launch.hypertune --conf_file <your_conf_file> <your_python_script> [args]

Known Issues

Please check at Known Issues webpage.

Intel® Extension for PyTorch* v1.10.200+gpu Release Notes

Intel® Extension for PyTorch* v1.10.200+gpu extends PyTorch* 1.10 with up-to-date features and optimizations on XPU for an extra performance boost on Intel Graphics cards. XPU is a user visible device that is a counterpart of the well-known CPU and CUDA in the PyTorch* community. XPU represents an Intel-specific kernel and graph optimizations for various “concrete” devices. The XPU runtime will choose the actual device when executing AI workloads on the XPU device. The default selected device is Intel GPU. XPU kernels from Intel® Extension for PyTorch* are written in DPC++ that supports SYCL language and also a number of DPC++ extensions.

Highlights

This release introduces specific XPU solution optimizations on Intel® Data Center GPU Flex Series 170. Optimized operators and kernels are implemented and registered through PyTorch* dispatching mechanism for the XPU device. These operators and kernels are accelerated on Intel GPU hardware from the corresponding native vectorization and matrix calculation features. In graph mode, additional operator fusions are supported to reduce operator/kernel invocation overheads, and thus increase performance.

This release provides the following features:

• Auto Mixed Precision (AMP)
  • support of AMP with BFloat16 and Float16 optimization of GPU operators
• Channels Last
  • support of channels_last (NHWC) memory format for most key GPU operators
• DPC++ Extension
  • mechanism to create PyTorch* operators with custom DPC++ kernels running on the XPU device
• Optimized Fusion
  • support of SGD/AdamW fusion for both FP32 and BF16 precision

This release supports the following fusion patterns in PyTorch* JIT mode:

• Conv2D + ReLU
• Conv2D + Sum
• Conv2D + Sum + ReLU
• Pad + Conv2d
• Conv2D + SiLu
• Permute + Contiguous
• Conv3D + ReLU
• Conv3D + Sum
• Conv3D + Sum + ReLU
• Linear + ReLU
• Linear + Sigmoid
• Linear + Div(scalar)
• Linear + GeLu
• Linear + GeLu_
• T + Addmm
• T + Addmm + ReLu
• T + Addmm + Sigmoid
• T + Addmm + Dropout
• T + Matmul
• T + Matmul + Add
• T + Matmul + Add + GeLu
• T + Matmul + Add + Dropout
• Transpose + Matmul
• Transpose + Matmul + Div
• Transpose + Matmul + Div + Add
• MatMul + Add
• MatMul + Div
• Dequantize + PixelShuffle
• Dequantize + PixelShuffle + Quantize
• Mul + Add
• Add + ReLU
• Conv2D + Leaky_relu
• Conv2D + Leaky_relu_
• Conv2D + Sigmoid
• Conv2D + Dequantize
• Softplus + Tanh
• Softplus + Tanh + Mul
• Conv2D + Dequantize + Softplus + Tanh + Mul
• Conv2D + Dequantize + Softplus + Tanh + Mul + Quantize
• Conv2D + Dequantize + Softplus + Tanh + Mul + Quantize + Add

Known Issues

• [CRITICAL ERROR] Kernel 'XXX' removed due to usage of FP64 instructions unsupported by the targeted hardware

  FP64 is not natively supported by the Intel® Data Center GPU Flex Series platform. If you run any AI workload on that platform and receive this error message, it means a kernel requiring FP64 instructions is removed and not executed, hence the accuracy of the whole workload is wrong.

• symbol undefined caused by _GLIBCXX_USE_CXX11_ABI

  ImportError: undefined symbol: _ZNK5torch8autograd4Node4nameB5cxx11Ev

  DPC++ does not support _GLIBCXX_USE_CXX11_ABI=0, Intel® Extension for PyTorch* is always compiled with _GLIBCXX_USE_CXX11_ABI=1. This symbol undefined issue appears when PyTorch* is compiled with _GLIBCXX_USE_CXX11_ABI=0. Update PyTorch* CMAKE file to set _GLIBCXX_USE_CXX11_ABI=1 and compile PyTorch* with particular compiler which supports _GLIBCXX_USE_CXX11_ABI. We recommend using gcc version 9.4.0 on ubuntu 20.04.

• Can't find oneMKL library when build Intel® Extension for PyTorch* without oneMKL

  /usr/bin/ld: cannot find -lmkl_sycl
  /usr/bin/ld: cannot find -lmkl_intel_ilp64
  /usr/bin/ld: cannot find -lmkl_core
  /usr/bin/ld: cannot find -lmkl_tbb_thread
  dpcpp: error: linker command failed with exit code 1 (use -v to see invocation)

  When PyTorch* is built with oneMKL library and Intel® Extension for PyTorch* is built without oneMKL library, this linker issue may occur. Resolve it by setting:

  export USE_ONEMKL=OFF
  export MKL_DPCPP_ROOT=${PATH_To_Your_oneMKL}/__release_lnx/mkl

  Then clean build Intel® Extension for PyTorch*.

• undefined symbol: mkl_lapack_dspevd. Intel MKL FATAL ERROR: cannot load libmkl_vml_avx512.so.2 or libmkl_vml_def.so.2

  This issue may occur when Intel® Extension for PyTorch* is built with oneMKL library and PyTorch* is not build with any MKL library. The oneMKL kernel may run into CPU backend incorrectly and trigger this issue. Resolve it by installing MKL library from conda:

  conda install mkl
  conda install mkl-include

  then clean build PyTorch*.

• OSError: libmkl_intel_lp64.so.1: cannot open shared object file: No such file or directory

  Wrong MKL library is used when multiple MKL libraries exist in system. Preload oneMKL by:

  export LD_PRELOAD=${MKL_DPCPP_ROOT}/lib/intel64/libmkl_intel_lp64.so.1:${MKL_DPCPP_ROOT}/lib/intel64/libmkl_intel_ilp64.so.1:${MKL_DPCPP_ROOT}/lib/intel64/libmkl_sequential.so.1:${MKL_DPCPP_ROOT}/lib/intel64/libmkl_core.so.1:${MKL_DPCPP_ROOT}/lib/intel64/libmkl_sycl.so.1

  If you continue seeing similar issues for other shared object files, add the corresponding files under ${MKL_DPCPP_ROOT}/lib/intel64/ by LD_PRELOAD. Note that the suffix of the libraries may change (e.g. from .1 to .2), if more than one oneMKL library is installed on the system.

Intel® Extension for PyTorch* v1.12.300-cpu Release Notes

Highlights

• Optimize BF16 MHA fusion to avoid transpose overhead to boost BERT-* BF16 performance #992
• Remove 64bytes alignment constraint for FP32 and BF16 AddLayerNorm fusion #992
• Fix INT8 RetinaNet accuracy issue #1032
• Fix Cat.out issue that does not update the out tensor (#1053) #1074

Full Changelog: v1.12.100...v1.12.300

Intel® Extension for PyTorch* v1.12.100-cpu Release Notes

This is a patch release to fix the AVX2 issue that blocks running on non-AVX512 platforms.

Intel® Extension for PyTorch* v1.12.0-cpu Release Notes

We are excited to bring you the release of Intel® Extension for PyTorch* 1.12.0-cpu, by tightly following PyTorch 1.12 release. In this release, we matured the automatic int8 quantization and made it a stable feature. We stabilized runtime extension and brought about a MultiStreamModule feature to further boost throughput in offline inference scenario. We also brought about various enhancements in operation and graph which are positive for the performance of broad set of workloads.

• Automatic INT8 quantization became a stable feature baking into a well-tuned default quantization recipe, supporting both static and dynamic quantization and a wide range of calibration algorithms.
• Runtime Extension, featured MultiStreamModule, became a stable feature, could further enhance throughput in offline inference scenario.
• More optimizations in graph and operations to improve performance of broad set of models, examples include but not limited to wave2vec, T5, Albert etc.
• Pre-built experimental binary with oneDNN Graph Compiler tuned on would deliver additional performance gain for Bert, Albert, Roberta in INT8 inference.

Highlights

• Matured automatic INT8 quantization feature baking into a well-tuned default quantization recipe. We facilitated the user experience and provided a wide range of calibration algorithms like Histogram, MinMax, MovingAverageMinMax, etc. Meanwhile, We polished the static quantization with better flexibility and enabled dynamic quantization as well. Compared to the previous version, the brief changes are as follows. Refer to tutorial page for more details.

v1.11.0-cpu

v1.12.0-cpu

import intel_extension_for_pytorch as ipex # Calibrate the model qconfig = ipex.quantization.QuantConf(qscheme=torch.per_tensor_affine) for data in calibration_data_set: with ipex.quantization.calibrate(qconfig): model_to_be_calibrated(x) qconfig.save('qconfig.json') # Convert the model to jit model conf = ipex.quantization.QuantConf('qconfig.json') with torch.no_grad(): traced_model = ipex.quantization.convert(model, conf, example_input) # Do inference y = traced_model(x)

import intel_extension_for_pytorch as ipex # Calibrate the model qconfig = ipex.quantization.default_static_qconfig # Histogram calibration algorithm and calibrated_model = ipex.quantization.prepare(model_to_be_calibrated, qconfig, example_inputs=example_inputs) for data in calibration_data_set: calibrated_model(data) # Convert the model to jit model quantized_model = ipex.quantization.convert(calibrated_model) with torch.no_grad(): traced_model = torch.jit.trace(quantized_model, example_input) traced_model = torch.jit.freeze(traced_model) # Do inference y = traced_model(x)

• Runtime Extension, featured MultiStreamModule, became a stable feature. In this release, we enhanced the heuristic rule to further enhance throughput in offline inference scenario. Meanwhile, we also provide the ipex.cpu.runtime.MultiStreamModuleHint to custom how to split the input into streams and concat the output for each steam.

v1.11.0-cpu

v1.12.0-cpu

import intel_extension_for_pytorch as ipex # Create CPU pool cpu_pool = ipex.cpu.runtime.CPUPool(node_id=0) # Create multi-stream model multi_Stream_model = ipex.cpu.runtime.MultiStreamModule(model, num_streams=2, cpu_pool=cpu_pool)

import intel_extension_for_pytorch as ipex # Create CPU pool cpu_pool = ipex.cpu.runtime.CPUPool(node_id=0) # Optional multi_stream_input_hint = ipex.cpu.runtime.MultiStreamModuleHint(0) multi_stream_output_hint = ipex.cpu.runtime.MultiStreamModuleHint(0) # Create multi-stream model multi_Stream_model = ipex.cpu.runtime.MultiStreamModule(model, num_streams=2, cpu_pool=cpu_pool, multi_stream_input_hint, # optional multi_stream_output_hint ) # optional

• Polished the ipex.optimize to accept the input shape information which would conclude the optimal memory layout for better kernel efficiency.

v1.11.0-cpu	v1.12.0-cpu
import intel_extension_for_pytorch as ipex model = ... model.load_state_dict(torch.load(PATH)) model.eval() optimized_model = ipex.optimize(model, dtype=torch.bfloat16)	import intel_extension_for_pytorch as ipex model = ... model.load_state_dict(torch.load(PATH)) model.eval() optimized_model = ipex.optimize(model, dtype=torch.bfloat16, sample_input=input)

• Provided a pre-built experimental binary with oneDNN Graph Compiler turned on, which would deliver additional performance gain for Bert, Albert, and Roberta in INT8 inference.

• Provided more optimizations in graph and operations

  • Fuse Adam to improve training performance #822
  • Enable Normalization operators to support channels-last 3D #642
  • Support Deconv3D to serve most models and implement most fusions like Conv
  • Enable LSTM to support static and dynamic quantization #692
  • Enable Linear to support dynamic quantization #787
  • Fusions.
    • Fuse Add + Swish to accelerate FSI Riskful model #551
    • Fuse Conv + LeakyReLU #589
    • Fuse BMM + Add #407
    • Fuse Concat + BN + ReLU #647
    • Optimize Convolution1D to support channels last memory layout and fuse GeLU as its post operation. #657
    • Fuse Einsum + Add to boost Alphafold2 #674
    • Fuse Linear + Tanh #711

Known Issues

• RuntimeError: Overflow when unpacking long when a tensor's min max value exceeds int range while performing int8 calibration. Please customize QConfig to use min-max calibration method.

• Calibrating with quantize_per_tensor, when benchmarking with 1 OpenMP* thread, results might be incorrect with large tensors (find more detailed info here. Editing your code following the pseudocode below can workaround this issue, if you do need to explicitly set OMP_NUM_THREAEDS=1 for benchmarking. However, there could be a performance regression if oneDNN graph compiler prototype feature is utilized.

  Workaround pseudocode:

  # perform convert/trace/freeze with omp_num_threads > 1(N)
  torch.set_num_threads(N)
  prepared_model = prepare(model, input)
  converted_model = convert(prepared_model)
  traced_model = torch.jit.trace(converted_model, input)
  freezed_model = torch.jit.freeze(traced_model)
  # run freezed model to apply optimization pass
  freezed_model(input)
  
  # benchmarking with omp_num_threads = 1
  torch.set_num_threads(1)
  run_benchmark(freezed_model, input)
  

• Low performance with INT8 support for dynamic shapes
  The support for dynamic shapes in Intel® Extension for PyTorch* INT8 integration is still work in progress. When the input shapes are dynamic, for example inputs of variable image sizes in an object detection task or of variable sequence lengths in NLP tasks, the Intel® Extension for PyTorch* INT8 path may slow down the model inference. In this case, use stock PyTorch INT8 functionality.
  Note: Using Runtime Extension feature if batch size cannot be divided by number of streams, because mini batch size on each stream are not equivalent, scripts run into this issues.

• BF16 AMP(auto-mixed-precision) runs abnormally with the extension on the AVX2-only machine if the topology contains Conv, Matmul, Linear, and BatchNormalization

• Runtime extension of MultiStreamModule doesn't support DLRM inference, since the input of DLRM (EmbeddingBag specifically) can't be simplely batch split.

• Runtime extension of MultiStreamModule has poor performance of RNNT Inference comparing with native throughput mode. Only part of the RNNT models (joint_net specifically) can be jit traced into graph. However, in one batch inference, joint_net is invoked multi times. It increases the overhead of MultiStreamModule as input batch split, thread synchronization and output concat.

• Incorrect Conv and Linear result if the number of OMP threads is changed at runtime
  The oneDNN memory layout depends on the number of OMP threads, which requires the caller to detect the changes for the # of OMP threads while this release has not implemented it yet.

• Low throughput with DLRM FP32 Tra...