[SYCL RTC] Use in-memory libcxx/libc headers for SPIR-V targets #20502
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -39,6 +39,60 @@ set(SYCL_JIT_RESOURCE_DEPS ${SYCL_JIT_RESOURCE_INSTALL_COMPONENTS}) | |
| # OpenCL-Headers doesn't have a corresponding build target: | ||
| list(FILTER SYCL_JIT_RESOURCE_DEPS EXCLUDE REGEX "^OpenCL-Headers$") | ||
|
|
||
| # We also want to embed LLVM's libc/libcxx headers into resource. We don't want | ||
| # to use them through LLVM_ENABLE_RUNTIMES for a few reasons though: | ||
| # * We configure them in a way that might be incompatible with their normal | ||
| # usage | ||
| # * We don't want to include them in all/install targets | ||
| # As such, configure libc/libcxx via explicit `llvm_ExternalProject_Add` in a | ||
| # separate location. | ||
| set(SYCL_JIT_RUNTIME_PROJECTS "libc;libcxx") | ||
| if (NOT WIN32) | ||
| list(APPEND SYCL_JIT_RUNTIME_PROJECTS libcxxabi libunwind) | ||
| endif() | ||
|
|
||
| # Couldn't pass -DLLVM_ENABLE_RUNTIMES=<libc;libcxx;...> through CMAKE_ARGS | ||
| # below because semicolon is used as a separate for CMAKE_ARGS itself. | ||
|
Contributor
There was a problem hiding this comment. Not sure I understand the limitation we're hiting
Contributor
Author
There was a problem hiding this comment. llvm/llvm/cmake/modules/LLVMExternalProjectUtils.cmake Lines 68 to 73 in ccac4e0 would see |
||
| # Workaround by passing it through PASSTHROUGH_PREFIXES by saving/restoring that | ||
| # variable's original value. | ||
| set(SYCL_JIT_LLVM_ENABLE_RUNTIMES_COPY ${LLVM_ENABLE_RUNTIMES}) | ||
| set(LLVM_ENABLE_RUNTIMES ${SYCL_JIT_RUNTIME_PROJECTS}) | ||
| llvm_ExternalProject_Add(sycl-jit-extra-headers | ||
| ${CMAKE_CURRENT_SOURCE_DIR}/../../runtimes | ||
| CMAKE_ARGS -DCOMPILER_RT_BUILD_BUILTINS=Off | ||
| -DLLVM_INCLUDE_TESTS=Off | ||
| -DLLVM_DEFAULT_TARGET_TRIPLE=${LLVM_TARGET_TRIPLE} | ||
| -DLLVM_ENABLE_PROJECTS_USED=${LLVM_ENABLE_PROJECTS_USED} | ||
| -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=${LLVM_ENABLE_PER_TARGET_RUNTIME_DIR} | ||
| -DLLVM_BUILD_TOOLS=${LLVM_BUILD_TOOLS} | ||
| -DCMAKE_C_COMPILER_WORKS=ON | ||
| -DCMAKE_CXX_COMPILER_WORKS=ON | ||
| -DCMAKE_Fortran_COMPILER_WORKS=ON | ||
| -DCMAKE_ASM_COMPILER_WORKS=ON | ||
| # libc config options: | ||
| -DLLVM_LIBC_FULL_BUILD=ON | ||
| -DLLVM_LIBC_ALL_HEADERS=1 | ||
| -DLIBC_CONFIG_PATH=${CMAKE_CURRENT_SOURCE_DIR}/lib/libc-config | ||
| # libcxx config options: | ||
| -DLIBCXX_HAS_EXTERNAL_THREAD_API=ON | ||
| TARGET_TRIPLE ${LLVM_TARGET_TRIPLE} | ||
| USE_TOOLCHAIN | ||
| PASSTHROUGH_PREFIXES LLVM_ENABLE_RUNTIMES | ||
| EXCLUDE_FROM_ALL | ||
| NO_INSTALL | ||
| ) | ||
| set(LLVM_ENABLE_RUNTIMES ${SYCL_JIT_LLVM_ENABLE_RUNTIMES_COPY}) | ||
| list(APPEND SYCL_JIT_RESOURCE_DEPS sycl-jit-extra-headers-configure) | ||
| list(APPEND SYCL_JIT_PREPARE_RESOURCE_COMMANDS | ||
| # libc | ||
| COMMAND ${CMAKE_COMMAND} --build ${CMAKE_BINARY_DIR}/tools/sycl-jit/jit-compiler/sycl-jit-extra-headers-bins --target generate-libc-headers | ||
| COMMAND ${CMAKE_COMMAND} -E copy_directory ${CMAKE_BINARY_DIR}/tools/sycl-jit/jit-compiler/sycl-jit-extra-headers-bins/libc/include ${SYCL_JIT_RESOURCE_INSTALL_DIR}/include/libc | ||
|
|
||
| # libcxx | ||
| COMMAND ${CMAKE_COMMAND} --build ${CMAKE_BINARY_DIR}/tools/sycl-jit/jit-compiler/sycl-jit-extra-headers-bins --target generate-cxx-headers | ||
| COMMAND ${CMAKE_COMMAND} --install ${CMAKE_BINARY_DIR}/tools/sycl-jit/jit-compiler/sycl-jit-extra-headers-bins --prefix ${SYCL_JIT_RESOURCE_INSTALL_DIR} --component cxx-headers | ||
| ) | ||
|
|
||
| # This is very hacky and I don't quite know what I'm doing, but it's necessary | ||
| # to have `resource.cpp` re-generated/re-built when some SYCL header changes. | ||
| # | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| if(EXISTS "${LIBC_SOURCE_DIR}/config/${LIBC_TARGET_OS}/${LIBC_TARGET_ARCHITECTURE}/entrypoints.txt") | ||
| include("${LIBC_SOURCE_DIR}/config/${LIBC_TARGET_OS}/${LIBC_TARGET_ARCHITECTURE}/entrypoints.txt") | ||
| else() | ||
| include("${LIBC_SOURCE_DIR}/config/${LIBC_TARGET_OS}/entrypoints.txt") | ||
| endif() |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| include("${LIBC_SOURCE_DIR}/config/linux/x86_64/headers.txt") |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,78 @@ | ||
| _LIBCPP_BEGIN_NAMESPACE_STD | ||
|
|
||
| using __libcpp_timespec_t = int; | ||
|
|
||
| // | ||
| // Mutex | ||
| // | ||
| using __libcpp_mutex_t = int; | ||
| #define _LIBCPP_MUTEX_INITIALIZER 0 | ||
|
|
||
| using __libcpp_recursive_mutex_t = int; | ||
|
|
||
|
Contributor
There was a problem hiding this comment. I have a feeling I asked this before on another PR but are these files basically to support stuff not supported by LLVM's libc/cxx?
Contributor
Author
There was a problem hiding this comment. The short answer something didn't work when I used libcxx in (default) mode that uses pthreads to implement |
||
| int __libcpp_recursive_mutex_init(__libcpp_recursive_mutex_t*); | ||
| _LIBCPP_NO_THREAD_SAFETY_ANALYSIS int __libcpp_recursive_mutex_lock(__libcpp_recursive_mutex_t*); | ||
| _LIBCPP_NO_THREAD_SAFETY_ANALYSIS bool __libcpp_recursive_mutex_trylock(__libcpp_recursive_mutex_t*); | ||
| _LIBCPP_NO_THREAD_SAFETY_ANALYSIS int __libcpp_recursive_mutex_unlock(__libcpp_recursive_mutex_t*); | ||
| int __libcpp_recursive_mutex_destroy(__libcpp_recursive_mutex_t*); | ||
|
|
||
| _LIBCPP_NO_THREAD_SAFETY_ANALYSIS int __libcpp_mutex_lock(__libcpp_mutex_t*); | ||
| _LIBCPP_NO_THREAD_SAFETY_ANALYSIS bool __libcpp_mutex_trylock(__libcpp_mutex_t*); | ||
| _LIBCPP_NO_THREAD_SAFETY_ANALYSIS int __libcpp_mutex_unlock(__libcpp_mutex_t*); | ||
| int __libcpp_mutex_destroy(__libcpp_mutex_t*); | ||
|
|
||
| // | ||
| // Condition Variable | ||
| // | ||
| using __libcpp_condvar_t = int; | ||
| #define _LIBCPP_CONDVAR_INITIALIZER 0 | ||
|
|
||
| int __libcpp_condvar_signal(__libcpp_condvar_t*); | ||
| int __libcpp_condvar_broadcast(__libcpp_condvar_t*); | ||
| _LIBCPP_NO_THREAD_SAFETY_ANALYSIS int __libcpp_condvar_wait(__libcpp_condvar_t*, __libcpp_mutex_t*); | ||
| _LIBCPP_NO_THREAD_SAFETY_ANALYSIS | ||
| int __libcpp_condvar_timedwait(__libcpp_condvar_t*, __libcpp_mutex_t*, __libcpp_timespec_t*); | ||
| int __libcpp_condvar_destroy(__libcpp_condvar_t*); | ||
|
|
||
| // | ||
| // Execute once | ||
| // | ||
| using __libcpp_exec_once_flag = int; | ||
| #define _LIBCPP_EXEC_ONCE_INITIALIZER 0 | ||
|
|
||
| int __libcpp_execute_once(__libcpp_exec_once_flag*, void (*__init_routine)()); | ||
|
|
||
| // | ||
| // Thread id | ||
| // | ||
| using __libcpp_thread_id = int; | ||
|
|
||
| bool __libcpp_thread_id_equal(__libcpp_thread_id, __libcpp_thread_id); | ||
| bool __libcpp_thread_id_less(__libcpp_thread_id, __libcpp_thread_id); | ||
|
|
||
| // | ||
| // Thread | ||
| // | ||
| #define _LIBCPP_NULL_THREAD 0 | ||
| using __libcpp_thread_t = int; | ||
|
|
||
| bool __libcpp_thread_isnull(const __libcpp_thread_t*); | ||
| int __libcpp_thread_create(__libcpp_thread_t*, void* (*__func)(void*), void* __arg); | ||
| __libcpp_thread_id __libcpp_thread_get_current_id(); | ||
| __libcpp_thread_id __libcpp_thread_get_id(const __libcpp_thread_t*); | ||
| int __libcpp_thread_join(__libcpp_thread_t*); | ||
| int __libcpp_thread_detach(__libcpp_thread_t*); | ||
| void __libcpp_thread_yield(); | ||
| void __libcpp_thread_sleep_for(const chrono::nanoseconds&); | ||
|
|
||
| // | ||
| // Thread local storage | ||
| // | ||
| #define _LIBCPP_TLS_DESTRUCTOR_CC 0 | ||
| using __libcpp_tls_key = int; | ||
|
|
||
| int __libcpp_tls_create(__libcpp_tls_key*, void (*__at_exit)(void*)); | ||
| void* __libcpp_tls_get(__libcpp_tls_key); | ||
| int __libcpp_tls_set(__libcpp_tls_key, void*); | ||
|
|
||
| _LIBCPP_END_NAMESPACE_STD | ||
Empty file.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -262,6 +262,103 @@ class SYCLToolchain { | |
| DAL.AddJoinedArg(nullptr, OptTable.getOption(OPT_offload_arch_EQ), CPU); | ||
| } | ||
|
|
||
| // Reasons why the following is done here and not in the clang driver: | ||
| // | ||
| // 1) Unlike libcxx, upstream libc is installed directly into | ||
| // `<toolchain>/include` or `<toolchain>/<target>/include` together with | ||
| // other compiler headers meaning we can't magically turn it on or off | ||
| // (unless we introduce a dedicated VFS overlay just for libc). | ||
| // 2) Having multiple C libraries in include search paths is unsupported, | ||
| // so in order to use LLVM libc we have to remove default system | ||
| // includes. That in turn excludes (at the very least) CUDA/HIP SDKs, so | ||
| // we want that behavior to be optional. That, in turn, means that | ||
| // because of (1) we have to have non-standard libc install location (we | ||
| // chose `<toolchain>/include/libc`) and that has no support in the | ||
| // clang driver, so we have to add libc headers to system include | ||
| // directories manually. | ||
| // 3) However, libcxx headers search path must come *before* libc includes, | ||
| // but `-isystem` and similar options prepend the list of search paths. | ||
| // As such, we can't just have the driver do part of the job and then | ||
| // adjust the behavior via extra options, so we need to maintain | ||
| // everything on our own. | ||
| // 4) We could do everything via custom code in the clang driver, but the | ||
| // location of `include/libc` is controlled in this `sycl-jit` project | ||
| // and it was slightly more convenient to implement it here, at least | ||
| // for the downstream implementation. | ||
| // 5) Once we upstream SYCL support there will be a use-case to move libc | ||
| // headers installation to a separate directory (similar to libcxx), at | ||
| // that time we might have support for this in the clang driver | ||
| // directly and would be able to avoid doing that here. | ||
|
|
||
| // Prefer using in-memory as that's friendlier for the end users of SYCL | ||
| // applications as that mode doesn't require any C/C++ toolchain to be | ||
| // installed on the system. | ||
| bool UseInMemoryCxxCHeaders = true; | ||
|
|
||
| // Unless explicitly told not to: | ||
| if (UserArgList.hasArg(OPT_sycl_rtc_use_system_includes)) | ||
| UseInMemoryCxxCHeaders = false; | ||
|
|
||
| // CUDA/HIP need SDK headers that we can't distribute ourselves, so we have | ||
| // to use system includes as well: | ||
| if (Format == BinaryFormat::PTX || Format == BinaryFormat::AMDGCN) | ||
| UseInMemoryCxxCHeaders = false; | ||
|
|
||
| if (UseInMemoryCxxCHeaders) { | ||
| DAL.AddFlagArg(nullptr, OptTable.getOption(OPT_nostdlibinc)); | ||
| auto AddInc = [&](auto RelPath) { | ||
| DAL.AddJoinedArg(nullptr, OptTable.getOption(OPT_isystem), | ||
| (getPrefix() + RelPath).str()); | ||
| }; | ||
| // Must come before C/C++ headers as we're intercepting them in those | ||
| // wrappers: | ||
| AddInc("include/sycl/stl_wrappers"); | ||
| // Extra headers we provide as part of jit-compiler, e.g. | ||
| // `__external_threading` and `linux/errno.h` that are needed to make | ||
| // LLVM's libc/libcxx work. As far as I know, can be anywhere in the | ||
| // includes search path as those files aren't provide anywhere else. | ||
| AddInc("include/sycl-rtc-standalone/"); | ||
| #if !defined(_WIN32) | ||
| // On Windows `LIBCXX_GENERATED_INCLUDE_TARGET_DIR` is off and thus we | ||
|
Contributor
There was a problem hiding this comment. is this option always off for windows and always on for linux? or can the user set it or something?
Contributor
Author
There was a problem hiding this comment. Don't know. Probably off by default but can be set: Lines 1011 to 1017 in ccac4e0 However, we configure libcxx ourselves and we don't pass many passthrough CMake variables, so I'd think it will be off reliably. Also, if somehow it changes, |
||
| // don't need this. | ||
| AddInc("include/x86_64-unknown-linux-gnu/c++/v1"); | ||
| #endif | ||
| // libcxx headers, must come before libc headers: | ||
| AddInc("include/c++/v1"); | ||
| // libc headers, our (SYCL RTC) custom non-standard location: | ||
| AddInc("include/libc"); | ||
| // SYCL/SYCL-related headers actually, because `<sycl/sycl.hpp>` and not | ||
| // just `<sycl.hpp>`. Can be argued that actual installation layout should | ||
| // actually be `include/sycl/ur_api.h` and `include/sycl/sycl/sycl.hpp` | ||
| // but that's outside the SYCL RTC scope. I think any relative order in | ||
| // relation to libcxx/libc is allowed. | ||
| AddInc("include/"); | ||
| // NOTE: `include/lib/clang/<version>/include/` is added automatically (we | ||
| // use `--nostdlibinc` and not `--nostdinc`). | ||
|
|
||
| DAL.AddJoinedArg(nullptr, OptTable.getOption(OPT_D), | ||
| "_LIBCPP_REMOVE_TRANSITIVE_INCLUDES"); | ||
| #if defined(_WIN32) | ||
| // LLVM's libc implements very limited number of entrypoints on WIN, | ||
| // almost to be unusable, so nobody actually cares about using libcxx over | ||
| // LLVM libc on that platform. We only use declaration and not definition | ||
| // so we force libc to generate more header/entrypoints but it's not | ||
| // working well by default. Options below were find by trial-and-error. | ||
| DAL.AddJoinedArg(nullptr, OptTable.getOption(OPT_D), | ||
aelovikov-intel marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| "_LIBCPP_WCHAR_H_HAS_CONST_OVERLOADS"); | ||
| DAL.AddJoinedArg(nullptr, OptTable.getOption(OPT_D), | ||
| "_LIBCPP_NO_VCRUNTIME"); | ||
| DAL.AddJoinedArg(nullptr, OptTable.getOption(OPT_U), "__ELF__"); | ||
|
|
||
| #endif | ||
| // Similarly to Windows case above, libcxx over libc isn't fully | ||
| // supported upstream, even on Linux. Faced some errors (mostly around | ||
| // `_LIBCPP_USING_IF_EXISTS`) if the files below aren't included early: | ||
| DAL.AddJoinedArg(nullptr, OptTable.getOption(OPT_include), "stdio.h"); | ||
| DAL.AddJoinedArg(nullptr, OptTable.getOption(OPT_include), "wchar.h"); | ||
aelovikov-intel marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| DAL.AddJoinedArg(nullptr, OptTable.getOption(OPT_include), "time.h"); | ||
| } | ||
|
|
||
| ArgStringList ASL; | ||
| for (Arg *A : DAL) | ||
| A->render(DAL, ASL); | ||
|
|
@@ -543,9 +640,15 @@ class SYCLToolchain { | |
| std::vector<std::string> CommandLine = | ||
| createCommandLine(UserArgList, Format, SourceFilePath); | ||
|
|
||
| llvm::IntrusiveRefCntPtr<llvm::vfs::OverlayFileSystem> FS; | ||
| if (UserArgList.hasArg(OPT_sycl_rtc_in_memory_fs_only)) { | ||
| FS = llvm::makeIntrusiveRefCnt<llvm::vfs::OverlayFileSystem>( | ||
| getToolchainFS()); | ||
| } else { | ||
| FS = llvm::makeIntrusiveRefCnt<llvm::vfs::OverlayFileSystem>( | ||
| llvm::vfs::getRealFileSystem()); | ||
| FS->pushOverlay(getToolchainFS()); | ||
| } | ||
| if (FSOverlay) | ||
| FS->pushOverlay(std::move(FSOverlay)); | ||
|
|
||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's commit libc and libcxx changes to https://github.com/llvm/llvm-project/ as well. It would be ideal if we merge them to https://github.com/llvm/llvm-project/ before merging this PR to gather the feedback from the LLVM's libc and libcxx communities.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't have an in-tree use-case to make upstream PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you please check with libc/libcxx maintainers if such use case is required?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't feel comfortable asking to contribute something that I myself believe shouldn't go upstream. For bugfixes I intended to do so, only to find later that those were fixed in the trunk, so it's not that I refuse to work with them.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@bader , ping
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@aelovikov-intel, do you have a question for me?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, I have an answer to your previous question. I don't think those changes can/should be upstreamed. If it's your stance that somebody has to go to community and get either an approval or rejection there before moving this PR forward, then we need to find another owner for this task.