Binary construction produces incorrect result

[mat-hek]

Trying to downstream FissionVM I found out that Base.decode16 (the Elixir's stdlib implementation) returns an incorrect binary. The smallest reproduction I was able to create is the following:

defmodule M do
  def process(binary) do
    process(binary, <<>>)
  end

  defp process(<<c1::8, c2::8, rest::binary>>, acc) do
    process(rest, <<acc::binary, c1::4, c2::4>>)
  end

  defp process(<<>>, acc) do
    acc
  end
end

Calling M.process("ABCDEF") returns <<21, 115, 81>>, while the correct value is <<18, 52, 86>>.

It only occurs when you enable ASAN set -fno-omit-frame-pointer, needed for ASAN:

# src/libAtomVM/CMakeLists.txt

target_compile_options(libAtomVM PUBLIC -fno-omit-frame-pointer)

and it seems it was introduced in #1906

[pguyot]

I confirm some kind of reproduction, without the -fno-omit-frame-pointer for emscripten node build. The problem goes away if I enable trace in opcodesswitch.h. I have <<41, 52, 86>> instead of <<18, 52, 86>>. I cannot reproduce it yet with a pure Erlang snippet.