VideoPlayback: feature requests

[Krakean]

Hello, dear author!
Can you please add support for following features to VideoPlayback example, so it can be considered full-featured:
a) Add support to display subtitles.
b) Add support to load subtitle from external file (srt is enough, but support for .ass would be really nice)
c) Add support to load audio from external file.
d) Add support to set/switch audio and subtitle streams before video is shown to user.

Benefits: modern games allows user to set audio/subtitle language from Game Settings menu.
Thus, implementing features above will allow a developer to let user choice what language they want to hear/read when watching in-game movies, while having just one video source (lets say, gameintro.mkv), and multiple audio/subtitles sources (gameintro_en.ogg, gameintro_en.srt, gameintro_cn.ogg, gameintro_cn.srt), thus it also will make life much easier for potential translators to different languages of the game - they will not be forced to re-render video file everytime when they for example make any correction to subtitle file for specific language.
P.S. .ass subtitle support can be harder to implement, so, for beginning, much more easier alternative can be used instead - implement just .srt parsing support, which do not require any special library to render like in case with .ass (libass), because .srt format is pretty simple - timecode and text with tags (b, i, u) to highlight specific phrases.

[hozuki]

Thank you for the feature requests. I'll explain my thoughts on the requested features.

a) Add support to display subtitles.

Hmmm this can be done.

b) Add support to load subtitle from external file (srt is enough, but support for .ass would be really nice)

As I'm thinking, SRT support does not make a difference with ASS. The format is very simple, But for rendering, it can be even harder to implement.
The most difficult step (of SRT) is rasterizing the text. This requires reading font file, correctly doing layout, computing geometries, and rasterizing geometries to bitmap (or any image buffer). Even if I can complete the last three tasks (they are not easy though) on my own, I still need to obtain the font information, which does not have cross-platform support in .NET. SharpFont is a choice but there are also pitfalls... I had experience in the other three tasks (in this repo and other repos) but my implementations are problematic. Sadly, there is no library providing only the last three tasks.
So now, since 1) libass provides a de facto standard of rendering subtitles, 2) it is robust and matured, and 3) ASS is a superset of textual subtitles (SRT, SUB, etc.), I would love to incorporate libass for subtitle rendering. For textual subtitles, I will convert them into ASS format and then render them by libass.
There is drawback using libass though. libass uses font discovery mechanism on different OSs: DirectWrite on Windows, CoreText on macOS, and fontconfig on Linux. There is a way to configure fontconfig to be used on all OSs, but setting up fontconfig will be a nightmare for Windows users. So the best solution for now is compiling libass for each of the OSs, and then use a <dllmap> to redirect library loading.

c) Add support to load audio from external file.

MonoGame seems to limit audio file input to wave banks generated by the content pipeline. But you can also load audio from file (or any input stream) using DynamicSoundEffectInstance. The only annoying thing is you have to decode the audio to wave audio by yourself. In fact, this is how the sound support in VideoPlayback is implemented.
Decoding audio leads to many choices. Does the code need to be fully managed? Which implementation should be used? That is something up to you. For example, for Ogg/Vorbis, there is NVorbis.
After creating a DynamicSoundEffectInstance to play external file, you can mute the VideoPlayer. Now the player only hears the sound from the external audio file.
Note: there should be a bug in current code that, the VideoPlayer only accepts video files with both video and audio stream(s), otherwise it crashes. I'll fix that later.

d) Add support to set/switch audio and subtitle streams before video is shown to user.

Setting tracks before playing can be done. However, switching on-the-fly requires (potentially?) more work on dynamically handling stream contexts and designing new synchronization method. I think for a small MonoGame extension, which does not provide full functionalities compared to professional video players, this goes too far. Besides, I don't have that depth of understanding of FFmpeg for now.

But contributions are always welcome!

[Krakean]

As I'm thinking, SRT support does not make a difference with ASS.

It makes big difference. SRT is just a timecode, text and possible tags within text, and those tags just three - b (bold), i (italic), u (underline), to highlight some phrase within a text (though, there is also font color tag supported in srt which allow you to specify a color for specific phrase, but I haven't seen such subtitles). Thats it. Nothing more can be set from srt, you can't even specify a font in case of srt, because assumed that srt is rendered with some kind of either default/already loaded/application-custom font (i.e., any font).
In case of ass, you can specify everything - font for whole text or specific phrase, positions, animations, even images. So, ass is much more complex than srt. Magnitude complex.
So, to answer this question:

I still need to obtain the font information, which does not have cross-platform support in .NET. SharpFont is a choice but there are also pitfalls

It can be solved easily. You can take a look to this project - https://github.com/rds1983/StbSharp (specifically - StbTrueType), it has demo - https://github.com/rds1983/StbSharp/tree/master/samples/StbSharp.MonoGame.Test - which is MonoGame + OGG loading example + Font loading example and test phrases. Just compile and you will see that mostly everything you need to display a phrase from SRT is actually already done :) All you need to do is figure out how to feed a phrase that needed to be rendered to a consumer, and its consumer job (in case of SRT) to already have font loaded. At least this is how it seems to me.

So now, since 1) libass provides a de facto standard of rendering subtitles, 2) it is robust and matured, and 3) ASS is a superset of textual subtitles (SRT, SUB, etc.), I would love to incorporate libass for subtitle rendering.

I'm absolutely ok with this approach, but just suggesting that SRT will be easier for beginning. With SRT you will get working result much quicker, than with ASS. ASS can be added later, for example. But its up to you, obviously. Both options are good. SRT is good, .ASS is much more better. But .ASS will take much more time from you to implement/integrate.

Does the code need to be fully managed? Which implementation should be used?
NVorbis is just fine. OGG is more than enough for speech/sound in case of video playback.

However, switching on-the-fly requires (potentially?) more work on dynamically handling stream contexts and designing new synchronization method.
No need to switch on-the-fly at all. Have an ability to set it before playing is more than enough. Or, there is no need in it all and it can be done automatically when for example after we set what video we want to play and then we set (external) audio track and (external) subtitle track.
Though, its not really clear for me at the moment how (stream buffer like MemoryReader, for example?) potential user can be able to set/load external audio/subtitle if they are within game archive for example. Because, by "external" I mean that they are not bundled into video file itself, but all of them (video, audios, subtitles) can be within one game archive.

[hozuki]

StbSharp seems pretty good for simple SRTs, but with limited support to CJK characters (because it writes to sprite font). I will consider making an interface for different subtitle providers.

[hozuki]

I implemented an external subtitle renderer using libass. The API is at draft level. See the example usage here. Please have a try; feedback is very much appreciated.

Note that the native libraries (ass.dll) requires VC++ 2019 redistributable. To use an earlier VC++, you can also compile libass yourself, see my recent repo.

[hozuki]

Oh and now a SRT renderer (see example) and experimental stream selection (not tested) are added