Error when cross compiling plugin on Apple Silicon

[bedranfleck]

Hi @PatriceBlin I've been having some issues trying to build the plugin locally. I've been following your guide on how to do this with the android NDK.

I'm trying to do this for NDK r25 revision r450784d.

Environment is macOS 13.6 with an Apple Silicon chip.

LLVM_HOME is set to: ~/tmp/llvm-toolchain/out/stage2-install/
LLVM_DIR is set to: $LLVM_HOME/lib64/cmake
TOOLCHAIN is ~/tmp/android-ndk-r25/toolchains/llvm/prebuilt/darwin-x86_64
CXX_HEADERS is set to: $LLVM_HOME/include/c++/v1 but I've tried these as well:
android-ndk-r25/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include/c++
android-ndk-r25/toolchains/llvm/prebuilt/linux-x86_64/include/c++

When I try to build the plugin I get the following:

$cmake -S ~/obfuscator-llvm -B ~/tmp/llvm-obfuscation-ndk -D LLVM_DIR="$LLVM_DIR" -D CMAKE_CXX_COMPILER=$CXX -D CMAKE_CXX_FLAGS="$CXXFLAGS"
-- The C compiler identification is Clang 14.0.6
-- The CXX compiler identification is Clang 14.0.6
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - failed
-- Check for working C compiler: /Users/andre.bedran/tmp/android-ndk-r25/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang
-- Check for working C compiler: /Users/andre.bedran/tmp/android-ndk-r25/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang - broken
CMake Error at /opt/homebrew/Cellar/cmake/3.27.6/share/cmake/Modules/CMakeTestCCompiler.cmake:67 (message):
  The C compiler

    "/Users/andre.bedran/tmp/android-ndk-r25/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: '/Users/andre.bedran/tmp/llvm-obfuscation-ndk/CMakeFiles/CMakeScratch/TryCompile-wSc4q3'

    Run Build Command(s): /opt/homebrew/Cellar/cmake/3.27.6/bin/cmake -E env VERBOSE=1 /usr/bin/make -f Makefile cmTC_da402/fast
    /Applications/Xcode.14.3.1.14E300b.app/Contents/Developer/usr/bin/make  -f CMakeFiles/cmTC_da402.dir/build.make CMakeFiles/cmTC_da402.dir/build
    Building C object CMakeFiles/cmTC_da402.dir/testCCompiler.c.o
    /Users/andre.bedran/tmp/android-ndk-r25/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang   -isystem /Users/andre.bedran/tmp/android-ndk-r25/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/include/ -I/Users/andre.bedran/tmp/android-ndk-r25/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/include/c++/v1  -arch arm64 -isysroot /Applications/Xcode.14.3.1.14E300b.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk -MD -MT CMakeFiles/cmTC_da402.dir/testCCompiler.c.o -MF CMakeFiles/cmTC_da402.dir/testCCompiler.c.o.d -o CMakeFiles/cmTC_da402.dir/testCCompiler.c.o -c /Users/andre.bedran/tmp/llvm-obfuscation-ndk/CMakeFiles/CMakeScratch/TryCompile-wSc4q3/testCCompiler.c
    Linking C executable cmTC_da402
    /opt/homebrew/Cellar/cmake/3.27.6/bin/cmake -E cmake_link_script CMakeFiles/cmTC_da402.dir/link.txt --verbose=1
    /Users/andre.bedran/tmp/android-ndk-r25/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang -isystem /Users/andre.bedran/tmp/android-ndk-r25/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/include/ -I/Users/andre.bedran/tmp/android-ndk-r25/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/include/c++/v1  -arch arm64 -isysroot /Applications/Xcode.14.3.1.14E300b.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk -Wl,-search_paths_first -Wl,-headerpad_max_install_names -L  CMakeFiles/cmTC_da402.dir/testCCompiler.c.o -o cmTC_da402
    ld: error: unknown argument '-dynamic', did you mean '-Bdynamic'
    ld: error: unknown argument '-arch'
    ld: error: unknown argument '-syslibroot'
    ld: error: unknown argument '-search_paths_first'
    ld: error: unknown emulation: acosx_version_min
    ld: error: cannot open arm64: No such file or directory
    ld: error: cannot open 13.0.0: No such file or directory
    ld: error: cannot open /Applications/Xcode.14.3.1.14E300b.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk: Is a directory
    ld: error: unable to find library -lSystem
    clang-14: error: linker command failed with exit code 1 (use -v to see invocation)
    make[1]: *** [cmTC_da402] Error 1
    make: *** [cmTC_da402/fast] Error 2





  CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
  CMakeLists.txt:3 (project)


-- Configuring incomplete, errors occurred!```

Looks like either I have a completely broken clang in my NDK installation, or I'm missing something crucial for it to work.

I have applied the fixes to `CryptoUtils.h` and `CMakeLists.txt` from the macOS build fix branch already, and I've tried some things you mentioned in #6 without success.

Do you have any thoughts? Thanks in advance.

P.S.: If you have any guidance on how to use the plugin through gradle afterwards, that would be great too.

[PatriceBlin]

Hi @bedranfleck

/Users/andre.bedran/tmp/android-ndk-r25/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang

This is the x86_64 toolchain, for Intel/AMD CPUs, the new Apple Silicon CPUs are aarch64 (ARM 64).
Android NDKs only have "darwin" build, probably only x86.
I don't know how people proceed but it looks like some are using Rosetta (running x86 on ARM).

android/ndk#1299

Maybe try doing something like this

arch -x86_64 cmake ... <REST OF THE ARGS>

[bedranfleck]

The docs in here say the following about the x86_64 naming convention:

Despite the x86_64 tag in the Darwin name, those are fat binaries that include M1 support. The paths were not updated to reflect that support because doing so would have broken existing builds that encode those paths.

So in theory it should build without rosetta, but I'll try anyway.

[bedranfleck]

Ok, I was able to get past this issue by adding this to CMakeLists before the project() call:
set(CMAKE_TRY_COMPILE_TARGET_TYPE "STATIC_LIBRARY")

This allowed me to bypass the compiler check and execute cmake successfully. However, I'm now stuck with what appears to be a problem in the ordering of search paths when running cmake --build

$cmake --build ~/tmp/llvm-obfuscation-ndk
[ 12%] Building CXX object CMakeFiles/LLVMObfuscator.dir/Plugin.cpp.o
In file included from /Users/andre.bedran/obfuscator-llvm/Plugin.cpp:1:
In file included from /Users/andre.bedran/tmp/llvm-toolchain/out/stage2-install/include/llvm/ADT/SmallVector.h:17:
In file included from /Users/andre.bedran/tmp/llvm-toolchain/out/stage2-install/include/llvm/Support/type_traits.h:17:
In file included from /Applications/Xcode.15.0.0.15A240d.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk/usr/include/c++/v1/type_traits:538:
/Applications/Xcode.15.0.0.15A240d.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk/usr/include/c++/v1/cstdint:149:5: error: <cstdint> tried including <stdint.h> but didn't find libc++'s <stdint.h> header.           This usually means that your header search paths are not configured properly.           The header search paths should contain the C++ Standard Library headers before           any C Standard Library, and you are probably using compiler flags that make that           not be the case.
#   error <cstdint> tried including <stdint.h> but didn't find libc++'s <stdint.h> header. \
    ^
In file included from /Users/andre.bedran/obfuscator-llvm/Plugin.cpp:1:
In file included from /Users/andre.bedran/tmp/llvm-toolchain/out/stage2-install/include/llvm/ADT/SmallVector.h:17:
In file included from /Users/andre.bedran/tmp/llvm-toolchain/out/stage2-install/include/llvm/Support/type_traits.h:18:
In file included from /Applications/Xcode.15.0.0.15A240d.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk/usr/include/c++/v1/utility:265:
In file included from /Applications/Xcode.15.0.0.15A240d.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk/usr/include/c++/v1/compare:145:
In file included from /Applications/Xcode.15.0.0.15A240d.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk/usr/include/c++/v1/__compare/compare_partial_order_fallback.h:13:
In file included from /Applications/Xcode.15.0.0.15A240d.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk/usr/include/c++/v1/__compare/partial_order.h:14:
In file included from /Applications/Xcode.15.0.0.15A240d.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk/usr/include/c++/v1/__compare/weak_order.h:14:
In file included from /Applications/Xcode.15.0.0.15A240d.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk/usr/include/c++/v1/__compare/strong_order.h:20:
/Applications/Xcode.15.0.0.15A240d.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk/usr/include/c++/v1/cmath:320:5: error: <cmath> tried including <math.h> but didn't find libc++'s <math.h> header.           This usually means that your header search paths are not configured properly.           The header search paths should contain the C++ Standard Library headers before           any C Standard Library, and you are probably using compiler flags that make that           not be the case.
#   error <cmath> tried including <math.h> but didn't find libc++'s <math.h> header. \
    ^
/Applications/Xcode.15.0.0.15A240d.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk/usr/include/c++/v1/cmath:587:12: error: no member named '__builtin_isnan' in namespace 'std'; did you mean simply '__builtin_isnan'?
    return std::isnan(__lcpp_x);
           ^~~~~
/Applications/Xcode.15.0.0.15A240d.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk/usr/include/c++/v1/cmath:576:12: note: '__builtin_isnan' declared here
    return __builtin_isnan(__lcpp_x);
           ^
/Applications/Xcode.15.0.0.15A240d.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk/usr/include/c++/v1/cmath:607:12: error: no member named '__builtin_isinf' in namespace 'std'; did you mean simply '__builtin_isinf'?
    return std::isinf(__lcpp_x);
           ^~~~~
/Applications/Xcode.15.0.0.15A240d.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk/usr/include/c++/v1/cmath:596:12: note: '__builtin_isinf' declared here
    return __builtin_isinf(__lcpp_x);
           ^
/Applications/Xcode.15.0.0.15A240d.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk/usr/include/c++/v1/cmath:646:38: error: no member named '__promote' in namespace 'std'
                                std::__promote<_A1, _A2> >::type
                                ~~~~~^
/Applications/Xcode.15.0.0.15A240d.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk/usr/include/c++/v1/cmath:646:48: error: '_A1' does not refer to a value
                                std::__promote<_A1, _A2> >::type
                                               ^
/Applications/Xcode.15.0.0.15A240d.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk/usr/include/c++/v1/cmath:643:17: note: declared here
template <class _A1, class _A2>
                ^
/Applications/Xcode.15.0.0.15A240d.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk/usr/include/c++/v1/cmath:646:58: error: expected unqualified-id
                                std::__promote<_A1, _A2> >::type
                                                         ^
In file included from /Users/andre.bedran/obfuscator-llvm/Plugin.cpp:1:
In file included from /Users/andre.bedran/tmp/llvm-toolchain/out/stage2-install/include/llvm/ADT/SmallVector.h:17:
In file included from /Users/andre.bedran/tmp/llvm-toolchain/out/stage2-install/include/llvm/Support/type_traits.h:18:
In file included from /Applications/Xcode.15.0.0.15A240d.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk/usr/include/c++/v1/utility:277:
In file included from /Applications/Xcode.15.0.0.15A240d.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk/usr/include/c++/v1/cstdlib:87:
In file included from /Users/andre.bedran/tmp/android-ndk-r25c/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/include/stdlib.h:34:
In file included from /Users/andre.bedran/tmp/android-ndk-r25c/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/include/malloc.h:30:
In file included from /Users/andre.bedran/tmp/android-ndk-r25c/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/include/stdio.h:42:
In file included from /Users/andre.bedran/tmp/android-ndk-r25c/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/include/sys/types.h:36:
/Users/andre.bedran/tmp/android-ndk-r25c/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/include/linux/types.h:21:10: fatal error: 'asm/types.h' file not found
#include <asm/types.h>
         ^~~~~~~~~~~~~
8 errors generated.
make[2]: *** [CMakeFiles/LLVMObfuscator.dir/Plugin.cpp.o] Error 1
make[1]: *** [CMakeFiles/LLVMObfuscator.dir/all] Error 2
make: *** [all] Error 2

I've tried fiddling with how the search paths are being passed in CXXFLAGS and CFLAGS, but to no avail. Any thoughts?

[bedranfleck]

I've realized that the above won't work, since this will need to produce a shared library, not a static one. So I'm back to where I was before. Building with rosetta didn't work since the binaries at darwin-x86_64 path already include arm support.

[PatriceBlin]

What do you have for $CC, $CXX and $LD ?

Maybe check if the fat binaries compatible for M1 can do simple task like --version or compile an hello world.

Detecting C compiler ABI info - failed

Would be the first error I think