Reason to move from OpenCV to ffmpeg

[Wovchena]

I see that diffusers.utils.export_to_video() encourages ffmpeg usage instead of OpenCV. Can you share the reason? I'm looking for a way to add video decoding to my project so I'm collecting arguments.

[xipingyan]

I don't see recommend in https://huggingface.co/docs/diffusers/en/api/utilities#diffusers.utils.export_to_video,
As I known, OpenCV also call ffmepg to decode.
For our sample, calling ffmpeg directly is OK, I just suggest to use OpenCV, because OpenCV's interface is friendly and easy to call.

[Wovchena]

Here it is

diffusers/src/diffusers/utils/export_utils.py

Line 172 in 55d49d4

"It is recommended to use `export_to_video` with `imageio` and `imageio-ffmpeg` as a backend. \n"

[asomoza]

actually diffusers uses imageio for loading/exporting videos and in turn it uses ffmpeg which is also the most used and universal library for encoding videos. You can learn the reason for this in this PR

[Wovchena]

Thank you!

allows videos to play on a larger number of platforms

appears less prone to codec issues

Can you remember specific examples? Especially given @xipingyan's statement

As I known, OpenCV also call ffmepg to decode.

[asomoza]

that PR wasn't me, so maybe @DN6 could provide more insights.

if you ask me, I would rather not use any python wrapper for this, probably just export the images (lossless) and use ffmpeg directly, any python library will make it slower and will prevent any real multi threading, but opencv is faster than imageio though.

Currently video models generate/use low framerate and shorts videos so it really doesn't matter which one we use for this.

[Wovchena]

@DN6, kind reminder

[vladmandic]

fyi, actually, i find pyav https://pypi.org/project/av/ to be the best solution, it also uses ffmpeg but it does so quite efficiently without the need for temporary files for each frame - it can stream frames directly into ffmpeg

[DN6]

export_to_video is meant to be a convenient utility function to save videos and is not designed to be an interface for full featured video encoding. OpenCV sometimes requires FFmpeg libraries (libavcodec-dev etc) to be installed on the system in order to work. imageio-ffmpeg ships with the executable, so it just works OOTB more reliably + the API is simple

[Wovchena]

Do you remember what cases require libavcodec-dev for OpenCV?