Add HLS sink with multi-variant playlist support

hlsmultivariantsink adds support for the following as per RFC 8216

• Multivariant/master playlist
• Alternate Renditions
• Variant Streams

Closes #489.

Motivation

HTTP Live Streaming specification in RFC 8216 defines alternate renditions and variant streams.

Existing HLS sink elements viz. hlssink2, hlssink3 and s3hlsssink do not have support for alternate renditions and variant streams as RFC 8216. hlsmultivariantsink is written with the intent of supporting alternate rendition and variant streams.

If alternate renditions or variant streams are not required, use of hlssink3 or hlssink2 is recommended instead.

Reducing the problem space

To simplify the implementation, hlsmultivariantsink purposely opts to limit itself with respect to the following two points.

• Handling of media streams

hlsmultivariantsink does not do any handling of the media streams. The user is expected to provide hlsmultivariantsink with the required media streams as input.

For example, consider the multiple video variant case, where the same audio is expected to be muxed with all the three video variants. The onus of tee’ing the audio and providing the same audio stream to be muxed with each of the three video variants stream is on the user.

• Muxing of audio and video

Except for the multi video variant when using MPEG-TS case, where the same audio is muxed with all the video variants, audio and video are non-muxed. Muxed audio and video is not supported with alternate renditions. For example, as in section 8.7 of RFC 8216, where all video renditions are required to contain the audio is not supported.

Design Considerations

Alternate rendition and variant stream parameters to be taken as input from the user.

Two approaches for this were considered.

1. Property settings on the element

Alternate rendition and variant stream is provided via the property setting on the element. The element will set up independent hlssink3 for each rendition or variant stream requested and emit pad-added signal for pads added, to be used by the application to provide the streams as input for the respective renditions and variant streams. With the hlssink3 connected downstream for each rendition or variant stream requested, each stream gets its own playlist.

2. Pad property settings

Alternate rendition and variant stream are property settings on the pad. For each audio or video pad requested, the user sets either the alternate rendition or variant stream property on the pad. Each requested pad is connected to a hlssink3 downstream with each rendition or variant stream, thus getting its own playlist.

Implementation

hlsmultivariantsink opts for the second approach above, where an audio or video pad has alternate rendition or variant stream as pad settings. Alternate rendition or variant stream inputs provided via these pad settings are used in the construction of multivariant/master playlist.

Whether CMAF or MPEG-TS is used, is selected by a muxer setting on the element. This in turn controls whether hlscmafsink or hlssink3 is used for generating the media playlist and segments.

For generating media playlist, a hlssink3 or hlscmafsink is connected downstream for each of the audio or video stream requested as part of an alternate rendition group or a variant stream.

CODECS string is required for variant stream and thus for building multivariant playlist. For the CMAF case, the implementation takes care of generating the CODECS string. If the CODECS field is not set when providing a variant stream input, the video codec string will not contain the profile level information, which can result in media playback failures in the case of browsers. For the MPEG-TS case, H264/H265 will use byte-stream as the stream-format which does not have codec_data. In the absence of codec_data, for MPEG-TS with H264/5, we resort to parsing the byte-stream for getting the relevant profile level information.

Testing

See the test code in net/hlsmultivariantsink/tests/hlsmultivariantsink.rs for examples.

The multivariant playlist generated can be tested as follows. In the directory where the master playlist is written, run a simple HTTP server using Python.

python3 -m http.server

GStreamer

gst-play-1.0 http://localhost:8000/multivariant.m3u8

ffmpeg

ffplay http://localhost:8000/multivariant.m3u8

video.js

To test with video.js, add the below HTML in an index.html file in the same location as the multivariant playlist. Open localhost:8000 in the browser to test.

<head>
  <link href="https://vjs.zencdn.net/8.10.0/video-js.css" rel="stylesheet" />

  <!-- If you'd like to support IE8 (for Video.js versions prior to v7) -->
  <!-- <script src="https://vjs.zencdn.net/ie8/1.1.2/videojs-ie8.min.js"></script> -->
</head>

<body>
  <video
    id="my-video"
    class="video-js"
    controls
    preload="auto"
    width="640"
    height="480"
    data-setup="{}"
  >
    <source src="http://localhost:8000/multivariant.m3u8" type="application/x-mpegURL" />
    <p class="vjs-no-js">
      To view this video please enable JavaScript, and consider upgrading to a
      web browser that
      <a href="https://videojs.com/html5-video-support/" target="_blank"
        >supports HTML5 video</a
      >
    </p>
  </video>

  <script src="https://vjs.zencdn.net/8.10.0/video.min.js"></script>
</body>

This example has been taken from here.

TODO:

• Support for closed captions
• Support for WebVTT subtitles
• Support for S3?

The work for the above TODOs will not be part of the work in this MR.