iOS build crashing on remote player join

[towneh]

Describe the bug?

iOS build running with xcode on macbook, able to successfully join basis server and load the asset bundle for the world. When a remote player is present or connects after this causes the iOS program to crash. Stacktrace below.

Possibly may be related to threading?

Steps to Reproduce

Build Basis for iOS and connect to a server where a remote user is connected

The expected behavior

Not crashing

Screenshots

BasisVR Release Version

iOS

What VR headsets are being used during this issue?

None

Log Files

message.txt

Additional Context

No response

[towneh]

Debug errors point towards OpusSharp

[katruud]

I have been able to get the client to work in iOS by disabling audio encoding/decoding. Debugging with Claude has yielded the following suggestions but has not fixed the crash.

This is the current crash:

Summary

The crash is caused by multiple issues in the OpusSharp library when used on iOS with IL2CPP static linking.

Root Causes Identified

1. Missing Native Wrapper Functions

OpusSharp declares P/Invoke functions (opus_encode24, opus_decode24, etc.) that don't exist in the standard Opus library. These are custom 24-bit audio wrappers that OpusSharp expects but never provides.

Fix: Created opus_wrapper.c with implementations that convert between int32 and float, then call the standard Opus API.

2. Incorrect DllImport Name

StaticNativeOpus.cs uses DllImport("Internal") (with trailing underscores) but iOS IL2CPP requires DllImport("__Internal").

Fix: Changed Internal → __Internal

3. Missing use_static: true

The encoder/decoder were created without use_static: true, causing them to try dynamic linking (DllImport("opus")) which doesn't exist on iOS.

Fix: Added use_static: true for iOS in BasisAudioReceiver.cs and BasisAudioTransmission.cs

4. Stack Overflow with VAR_ARRAYS

Opus built with VAR_ARRAYS uses stack-allocated Variable Length Arrays. iOS thread stacks are limited (~512KB), causing overflow during encoding.

Fix: Rebuilt libopus.a with OPUS_NONTHREADSAFE_PSEUDOSTACK=ON which uses heap allocation instead.

5. PSEUDOSTACK Race Condition (Current Blocker)

The PSEUDOSTACK implementation uses a single shared global buffer for ALL Opus operations. When the local encoder and remote player's decoder run simultaneously, they corrupt each other's temporary data → crash.

Files Modified

• StaticNativeOpus.cs - Fixed DllImport name
• OpusEncoder.cs - Added lock for PSEUDOSTACK serialization
• OpusDecoder.cs - Added lock for PSEUDOSTACK serialization
• BasisAudioReceiver.cs - Added use_static: true for iOS
• BasisAudioTransmission.cs - Added use_static: true for iOS
• libopus.a - Rebuilt with PSEUDOSTACK + custom wrapper functions

Files Added

• opus_wrapper.c - Native wrapper functions for 24-bit audio
• build_opus_ios_with_wrappers.sh - Build script for iOS libopus

Upstream Issues

These bugs exist in the upstream OpusSharp library:

1. Declares non-existent opus_*24 functions
2. Uses incorrect Internal DllImport name
3. iOS listed as "Not yet available" in README

ios.log

[towneh]

@SineVector241 anything here useful that you could suggest?

[katruud]

iOS Opus Crash Investigation Summary

Problem

The app crashes on iOS when another player joins the server. The crash occurs in the Opus audio codec's encode/decode functions.

Environment

• Platform: iOS (iPhone, ARM64)
• Unity Version: IL2CPP build
• Opus: Statically linked (__Internal DllImport)
• Opus Source: Built from https://github.com/xiph/opus main branch (commit 59f13a3)

Crash Details

• Crash Location: opus_encode / opus_encode_float / opus_decode / opus_decode_float
• Trigger: When a remote player joins and audio encoding/decoding begins
• Thread: Can occur on main thread (encoder) or ReceiveThread (decoder)

What Works

• opus_encoder_create - Successfully creates encoder handle
• opus_decoder_create - Successfully creates decoder handle
• opus_encoder_destroy / opus_decoder_destroy - Work fine
• The library loads and links correctly
• Skipping actual encode/decode calls (returning silence) - NO CRASH

What Crashes

• opus_encode (int16 API) - CRASHES
• opus_encode_float (float API) - CRASHES
• opus_decode (int16 API) - CRASHES
• opus_decode_float (float API) - CRASHES

Library Build Configurations Tested (ALL CRASH)

1. Default (Release, -O2)
2. No intrinsics (-DOPUS_DISABLE_INTRINSICS=ON)
3. Fixed point (-DOPUS_FIXED_POINT=ON)
4. Fixed point + no intrinsics
5. No optimizations (-O0)
6. No optimizations + no intrinsics (-O0 -DOPUS_DISABLE_INTRINSICS=ON)
7. O1 optimization
8. O1 + no intrinsics

C# Interop Approaches Tested (ALL CRASH)

1. SafeHandle with DangerousGetHandle() - Crash
2. Raw IntPtr (bypass SafeHandle) - Crash
3. Pinned arrays with fixed statement - Crash
4. Marshal.AllocHGlobal (native heap memory) - Crash
5. Float API (opus_encode_float/decode_float) - Crash
6. Int16 API (opus_encode/decode) with C# conversion - Crash
7. Global lock for thread safety - Crash

Fixes attempted (none fixed crash)

1. Static Linking Setup

• Changed DllImport("opus") to DllImport("__Internal") for iOS static linking
• Created StaticNativeOpus.cs with all P/Invoke declarations for __Internal
• Added use_static parameter to OpusEncoder/OpusDecoder constructors
• iOS code path uses static versions: #if UNITY_IOS && !UNITY_EDITOR

2. SafeHandle Variants

• Created StaticOpusEncoderSafeHandle and StaticOpusDecoderSafeHandle classes
• These call StaticNativeOpus.opus_encoder_destroy / opus_decoder_destroy in ReleaseHandle()

3. Raw Pointer Bypass (Avoid SafeHandle Entirely)

• Added _rawEncoderPtr / _rawDecoderPtr fields to store IntPtr directly
• Constructor stores raw pointer instead of wrapping in SafeHandle
• Encode/Decode methods use raw pointer directly instead of _handler.DangerousGetHandle()
• Dispose calls destroy function directly on raw pointer

4. Thread Safety

• Added GlobalOpusLock static object in StaticNativeOpus.cs
• All encode/decode/create/destroy calls wrapped in lock (StaticNativeOpus.GlobalOpusLock)

5. Memory Allocation Strategies

• Pinned arrays: fixed (float* ptr = array) - standard C# pinning
• Native heap: Marshal.AllocHGlobal() / Marshal.FreeHGlobal() - native memory allocation
• Both approaches crash identically

6. API Switching (Float vs Int16)

• Tried opus_encode_float / opus_decode_float (float API)
• Tried opus_encode / opus_decode (int16 API) with C# float↔int16 conversion
• Both APIs crash

7. Frame Size Validation

• Added GetValidOpusFrameSize() helper to ensure valid Opus frame sizes
• Valid sizes at 48kHz: 120, 240, 480, 960, 1920, 2880
• Confirmed frame_size=960 is being passed correctly

8. Parameter Validation

• Added null checks for encoder/decoder pointers before calls
• Added bounds checking for input/output buffers
• Added validation that frame_size > 0 and buffers are large enough

9. Encoder/Decoder Initialization

// iOS-specific initialization with use_static=true
encoder = new OpusEncoder(
    sampleRate,    // 48000
    channels,      // 1
    application,   // VOIP
    use_static: true  // Use __Internal DllImport
);

P/Invoke Signatures

[DllImport("__Internal", CallingConvention = CallingConvention.Cdecl)]
internal static extern unsafe int opus_encode(
    IntPtr st,
    short* pcm,
    int frame_size,
    byte* data,
    int max_data_bytes);

[DllImport("__Internal", CallingConvention = CallingConvention.Cdecl)]
internal static extern unsafe int opus_encode_float(
    IntPtr st,
    float* pcm,
    int frame_size,
    byte* data,
    int max_data_bytes);

Valid Parameters Confirmed

• frame_size: 960 (valid for 48kHz, 20ms)
• max_data_bytes: Properly sized output buffer
• encoder/decoder pointer: Non-null, created successfully
• input/output buffers: Properly allocated and sized

Build Script

cmake "$OPUS_SRC" \
    -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_SYSTEM_NAME=iOS \
    -DCMAKE_OSX_ARCHITECTURES="arm64" \
    -DCMAKE_OSX_DEPLOYMENT_TARGET="13.0" \
    -DCMAKE_OSX_SYSROOT="${SDK_PATH}" \
    -DCMAKE_POSITION_INDEPENDENT_CODE=ON \
    -DBUILD_SHARED_LIBS=OFF \
    -DOPUS_BUILD_PROGRAMS=OFF \
    -DOPUS_BUILD_TESTING=OFF \
    -DCMAKE_C_FLAGS="-O0" \
    -DOPUS_DISABLE_INTRINSICS=ON

Key Observations

1. The crash only happens on iOS - works fine on other platforms (Windows, macOS, Android)
2. Encoder/decoder creation works - only the actual encode/decode functions crash
3. The crash happens regardless of optimization level or intrinsics settings
4. The crash happens with both float and int16 APIs
5. Memory allocation strategy (pinned vs native heap) doesn't affect the crash

Possible Next Steps

1. Identify issue with Opus build
2. Try building with Xcode directly instead of CMake
3. Investigate if this is an IL2CPP bug
4. Try older Opus versions (1.3.x, 1.4.x)

Reproduction

1. Build Basis for iOS with IL2CPP
2. Deploy to iOS device
3. Connect to a server
4. Have another player join the same server
5. App crashes when audio encoding/decoding starts

Related Links

• OpusSharp: https://github.com/AvionBlock/OpusSharp
• Opus official: https://github.com/xiph/opus