In the last few months, we have been actively developing a lot of new AI solutions for videos. Automated video processing is a growing field of AI, with many interesting applications. It however brought quite a few new challenges (huge amount of data, processing time, precision, and so on) that didn’t need to be taken into consideration when building classic image-processing systems, including converting the standard video into streaming format. This article might prove useful to those who encountered similar challenges.
Contents:
- The Automated Video Processing
- What is the Streaming Format, and What is it Good For?
- HLS or DASH Format – Comparison
- How to convert .mp4, .mkv or .mov videos to HLS?
- How to convert videos to DASH?
- How can I stream videos on my website?
- Is there an API for conversion to DASH or HLS?
The Automated Video Processing
According to this research by Deloitte, it is typical for younger generations to build a dynamic portfolio of media and entertainment options. Consumers across generations have been spending more time watching online TV (Nielsen) and browsing the internet using social media and video-on-demand services on a daily basis.
There is no doubt that automated video processing is going to become as normal as image processing by AI, revolutionizing not only platforms such as YouTube, TikTok, Instagram, or Twitch – but the way we work with and perceive video content.
One of the projects that we are co-developing called Get Moments required a lot of working with FFmpeg, Python, OpenCV and Machine Learning (mostly TensorFlow). One of the challenges we encountered was converting a standard movie format into a streaming format, so there are quite a few tips I can now share with you.
What is the Streaming Format, and What is it Good For?
The need for a streaming format came with the rise and popularity of YouTube. Different users around the world have different internet connection speeds, and they can watch different parts of a video with different quality. That is possible because the video is delivered to them in a streaming format, without the need to load it fully.
Converting a video into a streaming format means you create multiple copies of this video with different qualities, all of which are chunked into short segments.
Instead of downloading and playing a video file in classic MP4 container format with H.264 (video codec), only the parts of the video that are currently watched, are loaded and streamed in the quality corresponding with the user’s internet connection quality. That is possible because when converting a standard video file into a streaming format, you create multiple copies of this video with different qualities, all of which are chunked into short segments.
HLS or DASH Streaming Format – Comparison
The full power of streaming video formats comes with CDNs (content delivery networks) that are able to deliver content over the internet very fast. There are several video streaming formats, but the currently most used are HLS and DASH. Both protocols run over HTTP, use TCP, are supported via HTML5 video player, and both chunk videos into segments with intervals of 2–10 seconds.
HLS (HTTP Live Streaming) is a live-streaming protocol with adaptive bitrate. Because it was developed by Apple, there is support for all Apple devices. HLS is using H.264 for video compression, with AAC and MP3 for an audio stream.
DASH (MPEG-DASH) is more open and standardized. It is widely used, for example on YouTube in HTM5 player. Unlike HLS, the DASH video can be encoded with different codes (codec-agnostic) for both video and audio streams.
I personally prefer the HTTP Live Streaming format for several reasons. The m3u8 index/header file looks much nicer, there is better support on Apple devices, and the conversion to HLS is much easier than to DASH. Nevertheless, not every video player is supporting the HLS or DASH format, so be careful what you have on your website or mobile app.
How much space do the HLS and DASH formats take up?
Let’s convert a sample video file with:
- Length 60 seconds
- Resolution 1080p
- Size 25 MB
- Encoded with H.264 codec
- No audio track
I converted this video to both HLS and DASH formats in 360p, 720p and 1080p resolutions. You can select your own resolution via encoding with FFmpeg.
When I converted the video to DASH with only two resolutions (360p and 1080p), the size was 32 MB. And when I added the third resolution (720p), I got to a similar size as with HLS. In both cases, the total size of the three files with different qualities together was around 55 MB, so a bit over double the size of the original file. Of course, the size can also change depending on the used codecs.
What is the data structure of HLS and DASH?
The folder with HLS format contains video encoded to 360p, 720p and 1080p. You can see the .ts files representing the chunks of 10-second intervals. Because we have a 60-second video, it contains 6 chunks – 6 .ts files.
In the case of DASH format, each video chunk has 5 seconds, so the video with DASH folder contains 12 chunks with a .m4s suffix.
You can also see index.m3u8, which is our index file. It is linked to the video player on the website where we are streaming. It is a simple text file containing information on which resolution and bandwidth these videos have. The content looks like this:
#EXTM3U #EXT-X-STREAM-INF:BANDWIDTH=375000,RESOLUTION=640x360 360_video.m3u8 #EXT-X-STREAM-INF:BANDWIDTH=2000000,RESOLUTION=1280x720 720_video.m3u8 #EXT-X-STREAM-INF:BANDWIDTH=3500000,RESOLUTION=1920x1080 1080_video.m3u8
The file 360_video.m3u8 defines the length of the chunk .ts files, and it looks like this:
#EXTM3U #EXT-X-VERSION:3 #EXT-X-TARGETDURATION:11 #EXT-X-MEDIA-SEQUENCE:0 #EXTINF:10.135122, 360_video0.ts #EXTINF:10.760756, 360_video1.ts #EXTINF:10.135122, 360_video2.ts #EXTINF:9.634622, 360_video3.ts #EXTINF:9.884878, 360_video4.ts #EXTINF:9.468422, 360_video5.ts #EXT-X-ENDLIST
The video converted to DASH format also has a manifest/index file with XML Structure.
How to convert .mp4, .mkv or .mov videos to HLS?
For converting the video to HLS streaming video format with three qualities (1920p, 720p and 360p) you can call the FFmpeg directly:
mkdir hls ffmpeg -i minute.mp4 -profile:v baseline -level 3.0 -s 640x360 -start_number 0 -hls_time 10 -hls_list_size 0 -f hls hls/360_video.m3u8 ffmpeg -i minute.mp4 -profile:v baseline -level 3.0 -s 1280x720 -start_number 0 -hls_time 10 -hls_list_size 0 -f hls hls/720_video.m3u8 ffmpeg -i minute.mp4 -profile:v baseline -level 3.0 -s 1920x1080 -start_number 0 -hls_time 10 -hls_list_size 0 -f hls hls/1080_video.m3u8
You can select the preferred resolutions via -s arg. For example, you can additionally create a video in 480p resolution if needed. With -hls_time, you are able to specify the length of chunks. After the conversion is done, we can manually or programmatically create an index.m3u8 file, which is used as a link in your web player.
You can also call conversion of MP4 to HLS via Python with a subprocess module:
def call_ffmpeg(cmd):
with subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) as process:
process.communicate()
return True
Read the FFMPEG documentation for more information about optimal setting your HLS video. There is a lot of parameters to tune for example you can set maxrate, bufsize, average bit rate and much more. Here are few links to start:
How to convert videos to DASH?
Similar to HLS, we can convert the video to DASH in two resolutions (360p and 1080p) with this command:
ffmpeg -re -i minute.mp4 -map 0 -map 0 -c:a libfdk_aac -c:v libx264 \ -b:v:0 300k -b:v:1 3000k -s:v:0 640x360 -s:v:1 1920x1080 -profile:v:1 baseline \ -profile:v:0 main -bf 1 -keyint_min 120 -g 120 -sc_threshold 0 \ -b_strategy 0 -ar:a:1 22050 -use_timeline 1 -use_template 1 \ -adaptation_sets "id=0,streams=v id=1,streams=a" \ -f dash dash/out.mpd
The video conversion is using the H.264 codec via -c:v libx264 argument. The resolution is set via -s:v argument. Whenever you are playing a DASH video, your entry point is this .mpd file, which will be generated during the conversion.
How can I stream videos on my website?
You can for example upload your converted videos to any storage like Amazon S3, Wasabi or DigitalOcean, and put the CDN (Cloudflare, CDN77 or Bunny) in front of your storage. For a web player, you could use for example the Bradmax player.
Did you find this guide useful? Check our other guides on custom visual search, image recognition & object detection systems, or various applications of visual AI.
Is there an API for conversion to streaming formats?
We’ve been working with video conversion and processing by artificial intelligence for a while and gained a lot of experience with FFmpeg on videos. If you would like to try our API for video conversion into streaming formats, cutting videos, concatenating, and video trimming, contact us at . In case you have any other questions or ideas, contact us through the contact form. We’re here to help!
