Runtest issues when running on Arm HPC cluster

[green-br]

A few findings when running tests on HPC cluster (this case checking for any ARM issues)

1. Test seems to be incorrectly failing - looks like PB is defined as 2**44 when it should be 2**50.

Test Failed at julia-1.11.7/share/julia/test/file.jl:1821
  Expression: dstat.total < 32PB
   Evaluated: 2623670942244864 < 562949953421312

2. Some comparison issues

Test Failed at julia-1.11.7/share/julia/stdlib/v1.11/Statistics/test/runtests.jl:477
  Expression: cov(A) ≈ cov(A, A) ≈ (cov(reshape(A, :, 1)))[1] ≈ (cov(reshape(A, :, 1)))[1] ≈ var(A)
   Evaluated: 400.0748f0 ≈ 399.82156f0 ≈ 400.07483f0 ≈ 400.07483f0 ≈ 400.0748f0
...
Test Failed at julia-1.11.7/share/julia/stdlib/v1.11/Profile/test/runtests.jl:72
  Expression: data_without[1] == data_with[1]
   Evaluated: 0x0000000000000000 == 0x0000000000000001
...

3. Memory mapping issue - original reason I ran tests due to known issue with LLVM and memory mapping with user hitting it. e.g. https://docs.julialang.org/en/v1/devdocs/build/arm/#Known-issues - reran with increased limit which helped but see 4.

Error During Test at julia-1.11.7/share/julia/test/testdefs.jl:24
  Got exception outside of a @test
  LoadError: SystemError: memory mapping failed: Invalid argument
  Stacktrace:
   [1] systemerror(p::String, errno::Int32; extrainfo::Nothing)
     @ Base ./error.jl:176
   [2] systemerror
     @ ./error.jl:175 [inlined]
   [3] mmap(io::IOStream, ::Type{Matrix{Int64}}, dims::Tuple{Int64, Int64}, offset::Int64; grow::Bool, shared::Bool)
     @ Mmap julia_mem_bug/julia-1.11.7/share/julia/stdlib/v1.11/Mmap/src/Mmap.jl:240
   [4] mmap(io::IOStream, ::Type{Matrix{Int64}}, dims::Tuple{Int64, Int64}, offset::Int64)
     @ Mmap julia_mem_bug/julia-1.11.7/share/julia/stdlib/v1.11/Mmap/src/Mmap.jl:188
   [5] top-level scope
     @ julia-1.11.7/share/julia/stdlib/v1.11/Mmap/test/runtests.jl:269
  in expression starting at julia_mem_bug/julia-1.11.7/share/julia/stdlib/v1.11/Mmap/test/runtests.jl:269

4. Had to interrupt since it seem to have got stuck even with increasing sysctl -w vm.max_map_count=262144

Error in testset Pkg:
Interrupted
Error in testset ccall:
Interrupted
Error in testset precompile:
Interrupted
Error in testset SharedArrays:
Interrupted
Error in testset threads:
Interrupted
Error in testset Distributed:
Interrupted
Error in testset gc:
Interrupted
Error in testset stress:
Interrupted

System information:

Julia Version 1.11.7
Commit f2b3dbda30a (2025-09-08 12:10 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (aarch64-linux-gnu)
  CPU: 144 × unknown
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, neoverse-v2)
Threads: 1 default, 0 interactive, 1 GC (on 144 virtual cores)

[giordano]

1. Test seems to be incorrectly failing - looks like PB is defined as 2**44 when it should be 2**50.

Oh, that's fun, luckily it's easy to fix.

2. Some comparison issues

There are known issue with libunwind on aarch64 (#54839), occasional failures in the Profile testset are expected. About the Statistics failure, I may be able to reproduce the issue, what CPU is this? I'll need the global RNG seed which is printed in the summary of all the failures.

4. Had to interrupt since it seem to have got stuck even with increasing sysctl -w vm.max_map_count=262144

Can't be sure it's exactly the same issue, but these weeks we had an issue with hanging tests in our CI, due to the memory mappings stuff (#59546 and #59658). @Keno what value did you use? I'll note that many Linux distro are moving to using 1048576 as default value, see for example Archlinux and Fedora

[green-br]

The RNG number was 0x15239d02f8ccb5a7fab7231a61f0925

The CPU is NVIDIA Grace CPU Superchip

I can try rerunning with a higher max_map_count and see if it still freezes. I may avoid some of the time consuming tests such as:

    LinearAlgebra/matmul                            |     1674                    3      1677  4m46.2s
    LinearAlgebra/lu                                |     1447                           1447  4m59.0s
    LinearAlgebra/symmetric                         |     3157                           3157  5m38.6s
    LinearAlgebra/diagonal                          |     3058                           3058  6m03.5s
    LinearAlgebra/dense                             |     8592                           8592  6m22.1s
    LinearAlgebra/bidiag                            |     7648                           7648  10m41.2s
    SparseArrays/sparsevector                       |    11959                    4     11963  10m44.2s
    LinearAlgebra/triangular                        |    38696                   16     38712  24m22.5s

[giordano]

BTW, do you still see JIT session error: Cannot allocate memory printed while running the tests, after increasing the memory mappings?

[giordano]

The RNG number was 0x15239d02f8ccb5a7fab7231a61f0925

The CPU is NVIDIA Grace CPU Superchip

Thanks (is that Isambard by any chance?). I can reproduce it on another Grace CPU and also with a different RNG seed, for example 0x57895fed49e65e1f3d0e86c30f372f74, so the numerical accuracy issue seems to be independent of the RNG seed). I'll open an issue in the Statistics.jl repository.

[Keno]

@Keno what value did you use? I'll note that many Linux distro are moving to using 1048576 as default value, see for example Archlinux and Fedora

I used 1048576

[giordano]

For the record, it turned out the test failure in Statistics is an upstream bug in OpenBLAS: OpenMathLib/OpenBLAS#5466. Only the neoversev1 type of cblas_sdot kernel gets the inaccurate results when using a single thread, armv8 and neoversen1 are good with any number of threads.

[green-br]

Isambard 3 - left job running overnight with the even higher setting. Thanks for tracing it to OpenBLAS - likely the SVE version in dot_kernel_sve.c - my assembly knowledge would need further work to debug that. There is change in flow where single thread uses the dot_compute directly compared to multi thread using blas_level1 interface to run a threaded version of the code.

[green-br]

Job still getting stuck - looking at ps output on node its in pkg.jl specifically "test atomicity of write_env_usage with $(Sys.CPU_THREADS) parallel processes" where I would assume it should be using mktempdir instead of tempdir to create an empty directory and not reuse same directory. Also I would question the use of tempname which I do not think is thread safe and likely the same file is being created by some of the processes.