Audio

[MrBrixican]

Just wanted to know whether audio support is a planned addition to this library. Audio is arguably a necessary component of most games, and I think it's something that's generally expected of most game frameworks. If you are interested, I have a feature writeup:

---

When it comes to cross platform game audio libraries/wrappers, the big names are essentially OpenAL(Soft), SDL_Mixer, SoLoud, FMOD, Wwise, and BASS. The first two are pretty dated, and the latter three are commercially licensed, which I don't think fits the theme of this repository. SoLoud is great, but I had a few concerns (see note 1).

Thus, I suggest a relatively thin wrapper around cute_sound. I like it for the following reasons:

• hooks into sdl for non standard platforms
• single header (technically two with ogg vorbis), very lightweight
• has instanced audio (which allows control after an instance has been played)
• has the ability to get/set current sample position (very important for rhythm games, haha)
• simple music fade support

The following implementation details would be pretty important

• separate namespace (~Foster.Framework.Audio) and optionally enabled (via App.Run), to avoid conflicts if a user decides to use another audio library (see note 2)

I know you used cute_sound with blah, so your opinion would obviously mean more than these superficial observations.

If this is something that is interesting to you, I'd be more than happy to take an initial crack at it.

note 1: SoLoud is an extremely honorable mention. It's pretty weird to build, but I have managed it. It's also a very fully featured library with effects, buses, and 3d spatialization, which may or may not be overkill. However, the effect on binary size would probably be considerable (it's actually pretty light < 1MB, forgot I published more than just the binary as an artifact) and it would most definitely constitute a dependency.

note 2: A one size fits all approach will simply not meet the complex needs of all games. Similar to animation, sprite, and ECS implementations, the correct audio implementation largely depends on the use case. If we cover 90-99% of simple 2D use cases with a lightweight abstraction and don't get in a user's way if they decide to use something different, that would be good enough.

[MrBrixican]

miniaudio is more mature than I remember it being. It's high level (engine) api seems like it could be a good fit. Seems like the only component that's a bit iffy is having to init a sound per instance and then deiniting once it's no longer in use (what defines that would be up to us). Also not sure about ogg support.

[NoelFB]

Yeah I definitely want some form of Audio support & agree with everything you've written up here. The main reason I don't have it yet is I 1) know very little about how the internals work and 2) work with audio teams that usually want to use existing proprietary tools like FMOD.

But that said I do want something simple and open source that can be used for small games. I am definitely on the same page regarding note 2 - Trying to implement a large scale generic system is not something I want to try to maintain, especially when people will likely just not use it anyway if they need more advanced features.

So that said, I do feel like a wrapper around cute_sound or miniaudio makes sense, similar to how the platform API currently wraps stb_image/stb_truetype. I also agree it should be in its own namespace, but probably still exist in the Framework folder (so Foster.Framework.Audio, like you suggested).

If you're interesting in taking a swing at this I'd be more than happy to give feedback and merge that in once ready. I'm not super sure whether cute_sound or miniaudio makes more sense, although I do agree I like both of these over soloud just due to their single-file nature.

[NoelFB]

Oh also it might make sense to let App.Run take a flags parameter, for various initialization flags such as disabling audio. I'm not sure what else it would have yet, though.

[MrBrixican]

Sounds good, I'll work on this later this week. I'll probably do an initial implementation with cute_sound just to get something quickly working, but I have a feeling that miniaudio will be better featured if I can get instancing implemented correctly.

[amerkoleci]

Sounds good, I'll work on this later this week. I'll probably do an initial implementation with cute_sound just to get something quickly working, but I have a feeling that miniaudio will be better featured if I can get instancing implemented correctly.

I would go with miniaudio, as it offers more stuff than cute_audio, or FAudio as it integrates greatly with SDL

[MrBrixican]

I agree with you on miniaudio, it would probably be the ideal long term solution. However, FAudio retains a lot of bloat in order to maintain full XNA compatability (not saying that's bad but not the same requirements we have).

[NoelFB]

Yeah I am in favor of miniaudio I think. I also feel like FAudio is a great project but it falls into the "larger audio" systems for me, and I feel like I want to try to keep the core of this framework as light as possible. If someone wants to use that instead they can just disable our default audio implementation.

[amerkoleci]

Sounds good, at this point I would suggest to turn off unused subsystems from SDL (like audio, renderer and probably others) this way resulting library is smaller

[NoelFB]

That's a good idea! I can probably control those with CMake compile flags somehow.

[MrBrixican]

I have been messing around with miniaudio and must say, it's a fantastic library. It should be pretty trivial to create some sound management Platform methods that are very thin layers over what is already built in.

I do have a couple of thoughts I'd like to cover before I begin my implementation:

---

Sound Instancing

miniaudio has a very minimal implementation of sound instancing. For the most part, it is very similar to OpenAL in that you must manually create and manage sounds (equivalent to OpenAL sources). In order to play the same audio multiple times concurrently, you would need to instantiate multiple sounds. Fire and forget audio would obviously require a layer that sits above miniaudio. Additionally, by default the resource manager will deallocate resources based on reference counts, so if you uninit all sounds pointing to an audio file, it will unload the data and further dispatches of the same audio would incur a file load.

Manual sound management is an unfortunate burden to place on the user, who should probably not be required to have knowledge of these inner workings. Thus, I will create a set of classes/structs that will provide a further abstraction for audio, similar to how Batch abstracts rendering.

Mainly, it will allow for loading an audio file one time, and then being able to create sound instances that can optionally be manipulated. An instance will automatically be disposed of behind the scenes when it stops playing.

---

Scope

I will aim to support basic sound features:

• play
• pause
• stop
• volume
• pitch
• pan
• current position / play time
• length
• looping

I may add spatialization, fading, and scheduled stop/start seeing as they're already implemented in miniaudio.

I will not be adding:

• Effects
  • The node based effect system in miniaudio is pretty complex and this would add a significant amount of complexity to an already complex first pass
• Microphone support
  • Most games don't use mic input, so this is not a priority
• Directly writing pcm data to output device
  • Again, most games don't require this functionality

---

Misc

The docs and issues made note of potential issues with ogg vorbis files (specifically get length methods), however I didn't notice any issues at all (possible my test files are exceptions).

There are some intricacies around the config settings you can use for the built in resource manager and the effect that has on runtime memory/cpu usage. However, those are easily configured so I'll leave that for the final pass.

---

If you have any thoughts you'd like to share, please chime in.

[NoelFB]

Hey this all sounds super good to me, thanks for working on this! I'm on the same page regarding implementation and features.

Various thoughts:

• I was thinking it might be worth making it possible to exclude the audio stuff from the native lib under an optional flag in case you want to run this on a weird platform that miniaudio doesn't support, but it's probably not necessary. I assume miniaudio just won't do anything on platforms it can't handle.
• I am also definitely in favor of handling the memory/instancing internally. I think the use case of this API should definitely favor simplicity and easy of use. If people want more complicated audio they can use a big external library.
• One consideration, depending on how this is set up, is trying to avoid GC allocations when playing new sound instances. If possible I could see the API working more like you get a struct handle back when playing a new instance, instead of allocating a class object. I'm not sure how realistic this is, but it would be nice to avoid allocating C# objects every time the user plays a sound.

[MrBrixican]

Regarding unsupported platforms, miniaudio defaults to a null backend which simply does nothing, as you say. Optionally, if it's a really weird platform miniaudio can be configured to back onto sdl. That should really only be consoles though, since miniaudio supports emscripten, desktop, and mobile platforms out of the box.

I 100% agree on the GC allocations, struct handles were what I had in mind, as it was something that bothered me with the XNA implementation. There will be plenty of unavoidable c allocations, but those will not touch the .NET GC.

[NoelFB]

Yeah, that makes sense to me! Sounds great.

[Flip120]

Hi, I created a very basic audio c# wrapper for miniaudio to be included in my foster game, some things still to be implemented/improved like instance management and adding more features but I guess this is more or less the approach you were discussing about using handles isn't it?
https://github.com/Flip120/miniaudio_csharp/tree/main

Here the platform part:
https://github.com/Flip120/miniaudio_csharp/blob/main/platform/audio_api.h
https://github.com/Flip120/miniaudio_csharp/blob/main/platform/audio_api.c

The c# wrapper:
https://github.com/Flip120/miniaudio_csharp/blob/main/C_Lib_Test/AudioPlatform.cs

And a little test program:
https://github.com/Flip120/miniaudio_csharp/blob/main/C_Lib_Test/Program.cs

[MrBrixican]

Apologies, I haven't updated this thread with the latest.

I made a miniaudio wrapper to be used with Foster with the following:

• Automated builds for Linux/Mac/Windows
• Supported features
  • Loading WAV, MP3, FLAC, QOA, OGG, and raw PCM data
  • Essential operations/settings: Play, Pause, Stop, Seek, Volume, Pitch, Pan, Looping, Spatialization
  • Sound groups to manage multiple sound instances (useful for sound category volume management)
  • Garbage free managed sound instances

There's still a bit more I want to do with effects and writing audio to file (likely breaking changes), but it's pretty well rounded for now. Also included is a small music visualizer example:

https://github.com/MrBrixican/Foster.Audio