Add onnxruntime as wasi-nn backend #4485

dongsheng28849455 · 2025-07-14T03:40:46Z

Steps to verify:

Install the onnx runtime (official release), assuming in /opt/onnxruntime
Build iwasm with WAMR_BUILD_WASI_NN_ONNX enabled
Using an onnx model of ssd-mobilenetv1 from ONNX Model Zoo
Generate the data file of input_tensor.bin, from origin picture for wasi-nn (with shape [1, 383, 640, 3])
Use nn-cli for test, eg.

--load-graph=file=./ssd_mobilenet_v1.onnx,id=graph1,encoding=1 
--init-execution-context=graph-id=graph1,id=exec0 
--set-input=file=./input_tensor.bin,context-id=exec0,dim=1,dim=383,dim=640,dim=3,type=3 
--compute=context-id=exec0 
--get-output=context-id=exec0,file=output.bin

Generate output.bin, with shape [1, 100, 4] and f32 type, which contents match the sample's output

yamt

what's the relationship with #4304?

yamt · 2025-07-14T06:04:44Z

core/iwasm/libraries/wasi-nn/src/wasi_nn_onnx.cpp

+#endif
+        default:
+            NN_WARN_PRINTF("Unsupported ONNX tensor type: %d", ort_type);
+            return fp32; // Default to fp32


why?
is there anything relying on this?

Follow up:
Type converter btw wasi-nn and onnx runtime returns bool instead of type

yamt · 2025-07-14T06:11:47Z

core/iwasm/libraries/wasi-nn/src/wasi_nn_onnx.cpp

+    std::lock_guard<std::mutex> lock(g_ort_ctx.mutex);
+
+    if (g_ort_ctx.is_initialized) {
+        *onnx_ctx = &g_ort_ctx;


why do you use globals?

i guess resources like graphs are not expected to be shared among unrelated instances.

I think it is not for sharing among instances? just a cache for next running

as g_graphs is a global, any instances can access any graphs.
ditto for g_exec_ctxs.

why to cache? is it very expensive to build these objects?

yamt · 2025-07-14T06:34:35Z

core/iwasm/libraries/wasi-nn/src/wasi_nn_onnx.cpp

+                   "total: %zu bytes",
+                   tensor_size, element_size, output_size_bytes);
+
+    if (*out_buffer_size < output_size_bytes) {


out_buffer_size is output-only these days.

can you apply recent changes?
#4411
#4468

Follow up:
out_buffer_size will not hold the expected size.

yamt · 2025-07-14T06:37:39Z

core/iwasm/libraries/wasi-nn/src/wasi_nn_onnx.cpp

+    status = ort_ctx->ort_api->CreateTensorWithDataAsOrtValue(
+        exec_ctx->memory_info, input_tensor->data.buf,
+        get_tensor_element_size(static_cast<tensor_type>(input_tensor->type))
+            * total_elements,


do you really need to calculate the size by yourself?
isn't input_tensor->data.size enough?

Follow up:
onnx runtime will not calculate input_tenser size

dongsheng28849455 · 2025-07-14T07:01:25Z

what's the relationship with #4304?

1, Adapt to latest wasi-nn arch and support WAMR_BUILD_WASI_EPHEMERAL_NN
2, Test with models and nn-cli

1, type converter btw wasi-nn and onnx runtime returns bool instead of type 2, out_buffer_size does not hold the expected size. 3, onnx runtime does not need calculate input_tenser size.

yamt · 2025-07-23T02:17:34Z

core/iwasm/libraries/wasi-nn/include/wasi_nn.h

@@ -21,7 +21,8 @@
 #else
 #define WASI_NN_IMPORT(name) \
    __attribute__((import_module("wasi_nn"), import_name(name)))
-#warning You are using "wasi_nn", which is a legacy WAMR-specific ABI. It's deperecated and will likely be removed in future versions of WAMR. Please use "wasi_ephemeral_nn" instead. (For a WASM module, use the wasi_ephemeral_nn.h header instead. For the runtime configurations, enable WASM_ENABLE_WASI_EPHEMERAL_NN/WAMR_BUILD_WASI_EPHEMERAL_NN.)
+#warning \
+    "You are using \"wasi_nn\", which is a legacy WAMR-specific ABI. It's deprecated and will likely be removed in future versions of WAMR. Please use \"wasi_ephemeral_nn\" instead. (For a WASM module, use the wasi_ephemeral_nn.h header instead. For the runtime configurations, enable WASM_ENABLE_WASI_EPHEMERAL_NN/WAMR_BUILD_WASI_EPHEMERAL_NN.)"


it's simpler to move unrelated changes to separate PRs.

1, clang-format change the line
2, agree with unrelated, can it be OK for this time?

what version of clang-format are you using?
wamr is currently using clang-format-14.
it didn't change the line for me.

yamt · 2025-07-23T02:18:07Z

core/iwasm/libraries/wasi-nn/include/wasi_nn_types.h

@@ -27,7 +27,7 @@ extern "C" {
 #define WASI_NN_TYPE_NAME(name) WASI_NN_NAME(type_##name)
 #define WASI_NN_ENCODING_NAME(name) WASI_NN_NAME(encoding_##name)
 #define WASI_NN_TARGET_NAME(name) WASI_NN_NAME(target_##name)
-#define WASI_NN_ERROR_TYPE WASI_NN_NAME(error);
+#define WASI_NN_ERROR_TYPE WASI_NN_NAME(error)


ditto
Agree with unrelated, can it be OK for this time?

yamt · 2025-07-23T02:18:15Z

core/iwasm/libraries/wasi-nn/src/wasi_nn.c

@@ -21,7 +21,8 @@
 #include "wasm_export.h"

 #if WASM_ENABLE_WASI_EPHEMERAL_NN == 0
-#warning You are using "wasi_nn", which is a legacy WAMR-specific ABI. It's deperecated and will likely be removed in future versions of WAMR. Please use "wasi_ephemeral_nn" instead. (For a WASM module, use the wasi_ephemeral_nn.h header instead. For the runtime configurations, enable WASM_ENABLE_WASI_EPHEMERAL_NN/WAMR_BUILD_WASI_EPHEMERAL_NN.)
+#warning \
+    "You are using \"wasi_nn\", which is a legacy WAMR-specific ABI. It's deprecated and will likely be removed in future versions of WAMR. Please use \"wasi_ephemeral_nn\" instead. (For a WASM module, use the wasi_ephemeral_nn.h header instead. For the runtime configurations, enable WASM_ENABLE_WASI_EPHEMERAL_NN/WAMR_BUILD_WASI_EPHEMERAL_NN.)"


yamt · 2025-07-23T02:20:33Z

core/iwasm/libraries/wasi-nn/src/wasi_nn_onnx.cpp

+    std::lock_guard<std::mutex> lock(g_ort_ctx.mutex);
+
+    if (g_ort_ctx.is_initialized) {
+        *onnx_ctx = &g_ort_ctx;


as g_graphs is a global, any instances can access any graphs.
ditto for g_exec_ctxs.

why to cache? is it very expensive to build these objects?

yamt · 2025-07-23T02:22:48Z

core/iwasm/libraries/wasi-nn/src/wasi_nn_onnx.cpp

+
+    size_t input_tensor_size = 1;
+    for (size_t i = 0; i < input_tensor->dimensions->size; ++i)
+        input_tensor_size *= input_tensor->dimensions->buf[i];


model_tensor_size and input_tensor_size are now unused?

yamt · 2025-07-29T08:45:17Z

core/iwasm/libraries/wasi-nn/src/wasi_nn_onnx.cpp

+#endif
+        default:
+            NN_WARN_PRINTF("Unsupported wasi-nn tensor type: %d", type);
+            return false; // Default to float


stale comment?

yamt · 2025-07-29T08:47:53Z

core/iwasm/libraries/wasi-nn/src/wasi_nn_onnx.cpp

+    wasi_nn_error err = success;
+    OrtStatus *status = nullptr;
+    OnnxRuntimeContext *ctx = nullptr;
+    ctx = new OnnxRuntimeContext();


we typically use wasm_runtime_malloc for memory allocation.
i don't think wamr in general is prepared for C++ exceptions.

also, malloc/free is used in this file. it's better to be consistent.

just considering the std::xxx object wrapped in them, new will trigger the creation for them together and release automatically. and seems wasi_nn_tensorflowlite also does that...

yamt · 2025-07-29T08:55:08Z

core/iwasm/libraries/wasi-nn/src/wasi_nn_onnx.cpp

+{
+    OnnxRuntimeContext *ort_ctx = (OnnxRuntimeContext *)onnx_ctx;
+
+    if (g >= MAX_GRAPHS || !ort_ctx->graphs[g].is_initialized) {


it's simpler to make these checks under the lock.

yamt · 2025-07-29T08:55:18Z

core/iwasm/libraries/wasi-nn/src/wasi_nn_onnx.cpp

+{
+    OnnxRuntimeContext *ort_ctx = (OnnxRuntimeContext *)onnx_ctx;
+
+    if (ctx >= MAX_CONTEXTS || !ort_ctx->exec_ctxs[ctx].is_initialized) {


it's simpler to make these checks under the lock.

yamt · 2025-07-29T08:56:55Z

core/iwasm/libraries/wasi-nn/src/wasi_nn_onnx.cpp

+{
+    OnnxRuntimeContext *ort_ctx = (OnnxRuntimeContext *)onnx_ctx;
+
+    if (ctx >= MAX_CONTEXTS || !ort_ctx->exec_ctxs[ctx].is_initialized) {


it's simpler to make these checks under the lock.

yamt · 2025-07-29T08:57:08Z

core/iwasm/libraries/wasi-nn/src/wasi_nn_onnx.cpp

+{
+    OnnxRuntimeContext *ort_ctx = (OnnxRuntimeContext *)onnx_ctx;
+
+    if (ctx >= MAX_CONTEXTS || !ort_ctx->exec_ctxs[ctx].is_initialized) {


it's simpler to make these checks under the lock.

yamt · 2025-07-29T09:01:11Z

core/iwasm/libraries/wasi-nn/src/wasi_nn_onnx.cpp

+            NN_ERR_PRINTF("Failed to get input name");
+            return err;
+        }
+        exec_ctx->input_names.push_back(input_name);


can push_back raise an exception?

shall we catch that?

Add onnxruntime as wasi-nn backend

2273302

dongsheng28849455 requested review from loganek, lum1n0us, no1wudi, TianlongLiang, wenyongh, xujuntwt95329 and yamt as code owners July 14, 2025 03:40

lum1n0us added the new feature Determine if this Issue request a new feature or this PR introduces a new feature. label Jul 14, 2025

yamt reviewed Jul 14, 2025

View reviewed changes

yamt added the wasi-nn label Jul 14, 2025

dongsheng28849455 added 2 commits July 17, 2025 14:20

follow up some review comments

98fb6ce

1, type converter btw wasi-nn and onnx runtime returns bool instead of type 2, out_buffer_size does not hold the expected size. 3, onnx runtime does not need calculate input_tenser size.

clang-format

70dbd1b

dongsheng28849455 requested a review from yamt July 17, 2025 06:49

yamt reviewed Jul 23, 2025

View reviewed changes

dongsheng28849455 force-pushed the feature/support_onnx_for_wasi-nn branch from aa88085 to 1fb25ad Compare July 29, 2025 08:09

dongsheng28849455 requested a review from yamt July 29, 2025 08:13

dongsheng28849455 force-pushed the feature/support_onnx_for_wasi-nn branch from 1fb25ad to 1e60909 Compare July 29, 2025 08:15

remove global context

29e4dd5

dongsheng28849455 force-pushed the feature/support_onnx_for_wasi-nn branch from 1e60909 to 29e4dd5 Compare July 29, 2025 08:17

yamt reviewed Jul 29, 2025

View reviewed changes

dongsheng28849455 requested a review from yamt July 29, 2025 09:06

Add onnxruntime as wasi-nn backend #4485

Are you sure you want to change the base?

Add onnxruntime as wasi-nn backend #4485

Conversation

dongsheng28849455 commented Jul 14, 2025

Uh oh!

yamt left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongsheng28849455 commented Jul 14, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!