update mnnsr

Author: tumuyan

README.md

@@ -34,7 +34,14 @@ RealSR-NCNN-Android-GUI could load extra models from sdcard automatily in ver 1.
 
 ![](Screenshot_models.jpg)
 
-### Convert pth models by yourself
+### Convert models to mnn format
+✨ ver 1.11 support [mnn](https://github.com/alibaba/MNN) models. MNN could support more models than ncnn, but it is slower in my experiment.  
+refer to https://mnn-docs.readthedocs.io/en/latest/tools/convert.html , you could convert ONNX, TFLITE, TorchScript, Tensorflow models to mnn format.
+1. `pip install mnn`
+2. (for example, convert onnx to mnn) `MNNConvert -f ONNX  --modelFile "{onnx_path}" --MNNModel "{mnn_path}"  --bizCode biz --fp16  --info  --detectSparseSpeedUp`
+
+
+### Convert pth models to ncnn format by yourself
 Also you could convert ESRGAN pth moddls by yourself.
 1. Download ESRGAN pytorch models from [OpenModelDB](https://openmodeldb.info/) and unzip it to somewhere.
 2. Download [cupscale](https://github.com/n00mkrad/cupscale) and unzip it in your PC.
@@ -178,13 +185,13 @@ Most of the C code is copied from Nihui, cause of the directory structure had to
 
 ## Others Open-Source Code Used
 -   [https://github.com/Tencent/ncnn](https://github.com/Tencent/ncnn)  for fast neural network inference on ALL PLATFORMS
+-   [https://github.com/alibaba/MNN](https://github.com/alibaba/MNN) A lightweight deep learning framework, battle-tested by business-critical use cases in Alibaba.  
 -   [https://github.com/nothings/stb](https://github.com/nothings/stb)  for decoding and encoding image on Linux / MacOS
 -   [https://github.com/tronkko/dirent](https://github.com/tronkko/dirent)  for listing files in directory on Windows
 -   [https://github.com/webmproject/libwebp](https://github.com/webmproject/libwebp) for encoding and decoding Webp images on ALL PLATFORMS
 -   [https://github.com/avaneev/avir](https://github.com/avaneev/avir) AVIR image resizing algorithm designed by Aleksey Vaneev
 -   [https://github.com/ImageMagick/ImageMagick6](https://github.com/ImageMagick/ImageMagick6) Use ImageMagick® to resize/convert images.
 -   [https://github.com/MolotovCherry/Android-ImageMagick7](https://github.com/MolotovCherry/Android-ImageMagick7) The repository has been archived, I use this fork [https://github.com/Miseryset/Android-ImageMagick7](https://github.com/Miseryset/Android-ImageMagick7)
 
-
 ## Others packaged models
 - Real-ESRGAN model [Nomos8kSC](https://github.com/Phhofm/models/tree/main/4xNomos8kSC) trained by Phhofm.  

README_CHS.md

@@ -6,7 +6,7 @@
 随着移动互联网的快速发展，智能设备逐渐普及到生活的每个角落。随之而来的是大量的图像数据。有的图片本身分辨率就比较低，难以看清楚细节；有的在存储和传输的过程中被反复压缩和劣化，最终不再清晰。  
 为了获得更加高质量的视觉体验，或者出于更为基本的目的看清楚图片，图像恢复/超分辨率算法应运而生。而手机作为目前我们生活中最常使用的智能设备，显然有使用这一技术的迫切需求。  
 
-这个仓库正是为安卓设备构建的一个图像超分辨率的应用。具有如下特点：  
+这个仓库正是为安卓设备构建的一个图像超分辨率的应用。具有如下特点：
   ✅ 内置超分算法和模型多。最初使用了[RealSR-NCNN](https://github.com/nihui/realsr-ncnn-vulkan)和[Real-ESRGAN](https://github.com/xinntao/Real-ESRGAN)的成果，后来又添加了[SRMD-NCNN](https://github.com/nihui/srmd-ncnn-vulkan)和[RealCUGAN-NCNN](https://github.com/nihui/realcugan-ncnn-vulkan), [Anime4KCPP](https://github.com/TianZerL/Anime4KCPP)。同时也内置了[waifu2x-ncnn](https://github.com/nihui/waifu2x-ncnn-vulkan)（但是没有内置模型和预设命令，如有需求自行下载并添加）  
   ✅ 兼顾传统插值算法。包括常见的nearest、bilinear、bicubic算法，以及imagemagick的二十多种filter。  
   ✅ 内置缩小算法。除使用用户指定倍率和算法的缩小方式外，resize-ncnn设计了一种自动缩小的算法de-nearest。参见[笔记](https://note.youdao.com/s/6XlIFbWt)  
@@ -16,6 +16,7 @@
   ✅ 自定义优先选用的超分算法和模型。    
   ✅ 自定义预设命令。  
   ✅ 图片处理过程完全在本地运行，无需担心隐私泄漏、服务器排队、服务收费；处理耗时取于决选择的模型、图片大小以及设备的性能。  
+  
 
 ### 下载地址 
 [Github Release](https://github.com/tumuyan/RealSR-NCNN-Android/releases)  
@@ -44,7 +45,15 @@
 ![](ScreenshotCHS.jpg)
 
 
-## 为 RealSR-NCNN-Android-GUI 增加更多模型
+
+## 为 RealSR-NCNN-Android-GUI 增加更多mnn模型
+✨ ver 1.11 或者更高版本支持  [mnn](https://github.com/alibaba/MNN)  模型。在我的测试中，mnn的速度要比ncnn慢，但是可以兼容更多模型。 
+转换模型的方法可以参考 https://mnn-docs.readthedocs.io/en/latest/tools/convert.html 模型转换工具能够将其他格式的模型（如：ONNX, TFLITE, TorchScript, Tensorflow等）转换为MNN模型
+1. （首先确认你已经安装了Python环境）`pip install mnn`
+2. （举例转换onnx模型到mnn格式）直接输入命令 `MNNConvert -f ONNX  --modelFile "{onnx_path}" --MNNModel "{mnn_path}"  --bizCode biz --fp16  --info  --detectSparseSpeedUp`
+
+
+## 为 RealSR-NCNN-Android-GUI 增加更多ncnn模型
 RealSR-NCNN-Android-GUI 在 ver 1.7.6 以上的版本可以自动加载自定义模型。
 你可以从 https://huggingface.co/tumuyan2/realsr-models 下载更多模型：
 1. 在文件管理器里新建一个目录
@@ -196,6 +205,7 @@ RealSR-NCNN-Android-GUI\app\src\main\assets\
 
 ## 使用的其他开源项目
 - [https://github.com/Tencent/ncnn](https://github.com/Tencent/ncnn)  for fast neural network inference on ALL PLATFORMS
+- [https://github.com/alibaba/MNN](https://github.com/alibaba/MNN) A lightweight deep learning framework, battle-tested by business-critical use cases in Alibaba.  
 - [https://github.com/nothings/stb](https://github.com/nothings/stb)  for decoding and encoding image on Linux / MacOS
 - [https://github.com/tronkko/dirent](https://github.com/tronkko/dirent)  for listing files in directory on Windows
 - [https://github.com/webmproject/libwebp](https://github.com/webmproject/libwebp) for encoding and decoding Webp images on ALL PLATFORMS

RealSR-NCNN-Android-CLI/MNN-SR/src/main/jni/CMakeLists.txt

@@ -1,6 +1,8 @@
-project(mnnsr)
+project(mnnsr-ncnn)
 
 cmake_minimum_required(VERSION 3.4.1)
+#set(CMAKE_CXX_STANDARD 20)
+#set(CMAKE_CXX_STANDARD_REQUIRED ON)
 set(CMAKE_BUILD_TYPE Debug)
 #set(CMAKE_BUILD_TYPE Release)
 
@@ -54,9 +56,9 @@ else()  # Android Studio
     set_target_properties(MNN PROPERTIES IMPORTED_LOCATION
             ${mnn_DIR}/${ANDROID_ABI}/libMNN.so)
 
-    add_library(c++_shared SHARED IMPORTED)
-    set_target_properties(c++_shared PROPERTIES IMPORTED_LOCATION
-            ${mnn_DIR}/${ANDROID_ABI}/libc++_shared.so)
+#    add_library(c++_shared SHARED IMPORTED)
+#    set_target_properties(c++_shared PROPERTIES IMPORTED_LOCATION
+#            ${mnn_DIR}/${ANDROID_ABI}/libc++_shared.so)
 
     add_library(MNN_CL SHARED IMPORTED)
     set_target_properties(MNN_CL PROPERTIES IMPORTED_LOCATION
@@ -73,11 +75,17 @@ else()  # Android Studio
 endif()
 
 message(STATUS "TARGET_ARCH: ${TARGET_ARCH}")
+
+
+find_package(OpenCV REQUIRED)
+include_directories(${OpenCV_DIR}/include)
+message(STATUS "find opencv result: ${OpenCV_LIBS}")
+
+
 include_directories(${mnn_DIR}/include)
 #include_directories(${mnn_DIR}/${ANDROID_ABI}/include/MNN)
 
 
-
 if(EXISTS ${MNN_LIB})
     message(STATUS "find mnn: ${MNN_LIB}")
 else()
@@ -102,8 +110,6 @@ else()
 endif()
 message(STATUS "find ncnn result: ${NCNN_LIB}")
 
-find_package(OpenCV REQUIRED)
-include_directories(${OpenCV_DIR}/include)
 
 set(libwebp_DIR ${CMAKE_CURRENT_SOURCE_DIR}/../../../../../3rdparty/libwebp)
 
@@ -136,9 +142,9 @@ if(NOT USE_SYSTEM_WEBP)
 endif()
 
 
-add_executable(${PROJECT_NAME}  main.cpp  mnnsr.cpp)
+add_executable(${PROJECT_NAME} main.cpp mnnsr.cpp)
 
-target_link_libraries(${PROJECT_NAME} webp ${MNN_LIB} ${OpenCV_LIBS}  ${NCNN_LIB})
+target_link_libraries(${PROJECT_NAME} webp  ${OpenCV_LIBS} ${MNN_LIB} ${NCNN_LIB})
 
 add_custom_command(TARGET ${PROJECT_NAME}  POST_BUILD
         COMMAND ${CMAKE_COMMAND} -E copy_directory

RealSR-NCNN-Android-CLI/MNN-SR/src/main/jni/main.cpp

@@ -29,15 +29,12 @@
 #define STBI_NO_PIC
 #define STBI_NO_STDIO
 
-#include "stb_image.h"
 
 #define STB_IMAGE_WRITE_IMPLEMENTATION
 
-#include "stb_image_write.h"
 
 #endif // _WIN32
 
-#include "webp_image.h"
 #include <chrono>
 
 using namespace std::chrono;
@@ -107,11 +104,6 @@ static std::vector<int> parse_optarg_int_array(const char *optarg) {
 
 #endif // _WIN32
 
-// ncnn
-//#include "cpu.h"
-//#include "gpu.h"
-//#include "platform.h"
-
 
 #include "mnnsr.h"
 #include "filesystem_utils.h"
@@ -217,7 +209,6 @@ void *load(void *args) {
     const LoadThreadParams *ltp = (const LoadThreadParams *) args;
     const int count = ltp->input_files.size();
     const int scale = ltp->scale;
-//    const bool check = ltp->check_threshold > 0;
 
 #pragma omp parallel for schedule(static, 1) num_threads(ltp->jobs_load)
     for (int i = 0; i < count; i++) {
@@ -272,9 +263,9 @@ void *load(void *args) {
             }
             merge(channels.data(), 3, inimage);
             v.inimage = inimage;
-        } else if(c==3){
+        } else if (c == 3) {
             v.inimage = image;
-        } else{
+        } else {
             fprintf(stderr, "[error] channel=%d, %s\n", image.channels(), imagepath.c_str());
             continue;
         }
@@ -346,54 +337,6 @@ class SaveThreadParams {
 
 };
 
-float compareNcnnMats(const ncnn::Mat &mat1, const ncnn::Mat &mat2) {
-    fprintf(stderr, "check result: mat1 %dx%dx%d, mat2 %dx%dx%d, elempack: %d, elemsize: %zu",
-            mat1.w, mat1.h, mat1.c, mat2.w, mat2.h, mat2.c, mat1.elempack, mat1.elemsize);
-
-    // 检查两个Mat对象的维度是否相同
-    if (mat1.w != mat2.w || mat1.h != mat2.h) {
-        return -1;
-    }
-
-    // 检查两个Mat对象的数据类型是否相同
-    if (mat1.elemsize != mat2.elemsize || mat1.elempack != mat2.elempack) {
-        return -2;
-    }
-
-    // 比较每个通道的像素差异
-    long sum_diff = 0;
-    int channels = std::min(mat1.c, mat2.c);
-    size_t plane_size = mat1.w * mat1.h * mat1.elemsize * mat1.elempack;
-    for (int c = 0; c < channels; c++) {
-
-        const unsigned char *data1 = mat1.channel(c);
-        const unsigned char *data2 = mat2.channel(c);
-
-//        fprintf(stderr, ", q=%d", sizeof(data1) / mat1.w / mat1.h);
-        for (size_t i = 0; i < plane_size; i++) {
-            int diff = abs(data1[i] - data2[i]);
-            sum_diff += diff;
-//            if(i<48)
-//                fprintf(stderr, "compare[%d]: %d, %d, diff=%d\n", i, data1[i],data2[i],diff);
-        }
-    }
-
-/*    for (int c = 0; c < channels; ++c) {
-        const unsigned char *data1 = mat1.channel(c);
-        const unsigned char *data2 = mat2.channel(c);
-        for (int i = 0; i < mat1.h; ++i) {
-            for (int j = 0; j < mat1.w; ++j) {
-                int diff = abs(data1[i * mat1.w + j] - data2[i * mat2.w + j]);
-                sum_diff += diff;
-            }
-        }
-    }*/
-
-    // 计算平均差异
-    float avg_diff = (float) sum_diff / (plane_size * channels);
-    return avg_diff;
-}
-
 void *save(void *args) {
     const SaveThreadParams *stp = (const SaveThreadParams *) args;
     const int verbose = stp->verbose;
@@ -485,9 +428,9 @@ void *save(void *args) {
 }
 
 #if _WIN32
-const std::wstring& optarg_in (L"A:\\Media\\realsr-ncnn-vulkan-20210210-windows\\input3.jpg");
-const std::wstring& optarg_out(L"A:\\Media\\realsr-ncnn-vulkan-20210210-windows\\output3.jpg");
-const std::wstring& optarg_mo (L"A:\\Media\\realsr-ncnn-vulkan-20210210-windows\\models-Real-ESRGAN-anime");
+const std::wstring& optarg_in (L"");
+const std::wstring& optarg_out(L"");
+const std::wstring& optarg_mo (L"");
 #else
 const char *optarg_in = "/sdcard/10086/input3.jpg";
 const char *optarg_out = "/sdcard/10086/output3.jpg";
@@ -507,7 +450,7 @@ int main(int argc, char **argv)
     path_t inputpath;
     path_t outputpath;
     int scale = 4;
-    int tilesize = 128;
+    int tilesize = 0;
 #if _DEMO_PATH
     path_t model = optarg_mo;
 #else
@@ -545,7 +488,10 @@ int main(int argc, char **argv)
             model = optarg;
             break;
         case L'g':
-//            gpuid = parse_optarg_int_array(optarg);
+            gpuid = parse_optarg_int_array(optarg);
+            if(gpuid.size()>0 && gpuid[0]==-1){
+                backend_type = MNN_FORWARD_CPU;
+            }
             break;
         case L'j':
             swscanf(optarg, L"%d:%*[^:]:%d", &jobs_load, &jobs_save);
@@ -563,7 +509,8 @@ int main(int argc, char **argv)
         case L'c':
             check_threshold = _wtoi(optarg);
             break;
-            case L'b':
+        case L'b':
+            if(backend_type != MNN_FORWARD_CPU)
                 backend_type = (MNNForwardType)_wtoi(optarg);
         case L'h':
         default:
@@ -591,7 +538,10 @@ int main(int argc, char **argv)
                 model = optarg;
                 break;
             case 'g':
-//                gpuid = parse_optarg_int_array(optarg);
+                gpuid = parse_optarg_int_array(optarg);
+                if (gpuid.size() > 0 && gpuid[0] == -1) {
+                    backend_type = MNN_FORWARD_CPU;
+                }
                 break;
             case 'j':
                 sscanf(optarg, "%d:%*[^:]:%d", &jobs_load, &jobs_save);
@@ -610,7 +560,8 @@ int main(int argc, char **argv)
                 check_threshold = atoi(optarg);
                 break;
             case 'b':
-                backend_type = (MNNForwardType) atoi(optarg);
+                if (backend_type != MNN_FORWARD_CPU)
+                    backend_type = (MNNForwardType) atoi(optarg);
             case 'h':
             default:
                 print_usage();
@@ -733,7 +684,7 @@ int main(int argc, char **argv)
 
     int prepadding = 0;
 
-    if (model.find(PATHSTR("models-")) != path_t::npos) {
+    if (model.find(PATHSTR("models-")) != path_t::npos ||(  model.size() >= 4 && model.compare(model.size() - 4, 4, ".mnn") == 0)) {
         prepadding = 5;
     } else {
         fprintf(stderr, "unknown model dir type\n");
@@ -747,14 +698,21 @@ int main(int argc, char **argv)
 
 #if _WIN32
     wchar_t modelpath[256];
-    swprintf(modelpath, 256, L"%s/x%d.mnn", model.c_str(), scale);
+    if(model.ends_with(".mnn")){
+        swprintf(modelpath, 256, L"%s", model.c_str());
+    }else{
+        swprintf(modelpath, 256, L"%s/x%d.mnn", model.c_str(), scale);
+    }
     fprintf(stderr, "search model: %ws\n", modelpath);
 
     path_t modelfullpath = sanitize_filepath(modelpath);
     FILE* mp = _wfopen(modelfullpath.c_str(), L"rb");
 #else
     char modelpath[256];
-    sprintf(modelpath, "%s/x%d.mnn", model.c_str(), scale);
+    if ((  model.size() >= 4 && model.compare(model.size() - 4, 4, ".mnn") == 0))
+        sprintf(modelpath, "%s", model.c_str());
+    else
+        sprintf(modelpath, "%s/x%d.mnn", model.c_str(), scale);
     fprintf(stderr, "search model: %s\n", modelpath);
 
     path_t modelfullpath = sanitize_filepath(modelpath);
@@ -793,6 +751,13 @@ int main(int argc, char **argv)
         return -1;
     }
 
+    // 移动到文件末尾
+    fseek(mp, 0, SEEK_END);
+    // 获取文件大小
+    long modelsize = ftell(mp)/1000000;
+    // 重置文件指针到开头
+    fseek(mp, 0, SEEK_SET);
+    fclose(mp);
 
 #if _WIN32
     CoInitializeEx(NULL, COINIT_MULTITHREADED);
@@ -805,25 +770,29 @@ int main(int argc, char **argv)
 
     const int use_gpu_count = 1;
     if (jobs_proc.empty()) {
-        jobs_proc.resize(use_gpu_count, 2);
+        jobs_proc.resize(use_gpu_count, 1);
     }
     int total_jobs_proc = 0;
     total_jobs_proc += jobs_proc[0];
 
     fprintf(stderr, "busy...\n");
     {
-        MNNSR mnnsr = MNNSR(0, false, 1);
+        MNNSR mnnsr = MNNSR();
 
-        if (tilesize == 0)
+        if (tilesize == 0){
             tilesize = 128;
+            tilesize =4000/ modelsize;
+            if(tilesize>300)
+                tilesize = 300;
+        }
         if (tilesize < 64)
             tilesize = 64;
         mnnsr.tilesize = tilesize;
         mnnsr.prepadding = prepadding;
         mnnsr.backend_type = backend_type;
 
         mnnsr.scale = scale;
-        mnnsr.load(modelfullpath);
+        mnnsr.load(modelfullpath, modelsize>10);
 
         // main routine
         {