webrtcsink: imx8mp h264 encoder support
This is a fix to support NXP vpu h264 encoder, is tested on an NXP imx8mp board.
imx8mp was running the signaling server and the following pipeline using webrtcsink built from this very MR:
GST_DEBUG_FILE=~/congestion_test.log GST_DEBUG=4 gst-launch-1.0 webrtcsink do-retransmission=false name=ws congestion-control=2 videotestsrc ! videoconvert ! "video/x-raw,format=YUY2,width=400,height=300,framerate=(fraction)30/1" ! ws.
This is the used vpu:
====== VPUENC: 4.8.0 build on May 10 2023 01:50:04. ====== wrapper: 3.0.0 (VPUWRAPPER_ARM64_LINUX Build on Aug 31 2022 01:28:14) vpulib: 1.1.1 firmware: 1.1.1.65535
Here some important log captured from the pipeline running on imx8mp:
0:00:37.433769723 [35m21232[00m 0xffff68006b60 [36mINFO [00m [00;01;31m GST_STATES gstelement.c:2816:gst_element_continue_state:<vpuenc_h264-1>[00m completed state change to PLAYING 0:01:04.578035999 [35m21232[00m 0xffff6000f6a0 [36mINFO [00m [00m vpuenc gstvpuenc.c:913:gst_vpu_enc_set_format:<vpuenc_h264-1>[00m setting bitrate to 101 kbps and GOP size to 30
Computer was receiving the stream with same version of webrtcrs plugins (main commit 0bd98e2c):
imx8mp was also running a test application that was reading the stats property from the webrtcsink plugin while streaming, here some of them:
application/x-webrtcsink-stats, 4f8919b7-40d1-401c-a6eb-1fb4f9d0fed7=(structure)[application/x-webrtc-stats, peer-connection-stats=(structure)[peer-connection, data-channels-opened=(uint)0, data-channels-closed=(uint)0, data-channels-requested=(uint)0, data-channels-accepted=(uint)0, type=(GstWebRTCStatsType)peer-connection, timestamp=(double)595549707.40499997, id=(string)peer-connection-stats;], codec-stats-sink_0=(structure)[codec, type=(GstWebRTCStatsType)codec, timestamp=(double)595549707.40499997, id=(string)codec-stats-sink_0, payload-type=(uint)96, clock-rate=(uint)90000, ssrc=(uint)2311868531, mime-type=(string)video/H264, codec-type=(string)encode, sdp-fmtp-line=(string)"96\ packetization-mode\=1\;sprop-parameter-sets\=J0LgH4mNUHgLdCAAAAMAIAAAAwPB4sS3\,KM4CXIA\=\;profile-level-id\=42e01f\;level-asymmetry-allowed\=1\;level-asymmetry-allowed\=1";], ice-candidate-local_1_10.2.27.61_48746=(structure)[local-candidate, type=(GstWebRTCStatsType)local-candidate, timestamp=(double)595549707.40499997, id=(string)ice-candidate-local_1_10.2.27.61_48746, transport-id=(string)transport-stats_webrtcdtlstransport0, address=(string)10.2.27.61, port=(uint)48746, candidate-type=(string)host, priority=(uint)2015363327, protocol=(string)udp, url=(string)"";], ice-candidate-remote_1_10.13.44.67_50684=(structure)[remote-candidate, type=(GstWebRTCStatsType)remote-candidate, timestamp=(double)595549707.40499997, id=(string)ice-candidate-remote_1_10.13.44.67_50684, transport-id=(string)transport-stats_webrtcdtlstransport0, address=(string)10.13.44.67, port=(uint)50684, candidate-type=(string)host, priority=(uint)2015363839, protocol=(string)udp;], ice-candidate-pair_webrtcnicetransport0=(structure)[candidate-pair, type=(GstWebRTCStatsType)candidate-pair, timestamp=(double)595549707.40499997, id=(string)ice-candidate-pair_webrtcnicetransport0, local-candidate-id=(string)ice-candidate-local_1_10.2.27.61_48746, remote-candidate-id=(string)ice-candidate-remote_1_10.13.44.67_50684, gst-twcc-stats=(structure)[RTPTWCCStats, bitrate-sent=(uint)1307024, bitrate-recv=(uint)1307384, packets-sent=(uint)10, packets-recv=(uint)10, packet-loss-pct=(double)0, avg-delta-of-delta=(gint64)1975;];], transport-stats_webrtcdtlstransport0=(structure)[transport, type=(GstWebRTCStatsType)transport, timestamp=(double)595549707.40499997, id=(string)transport-stats_webrtcdtlstransport0, selected-candidate-pair-id=(string)ice-candidate-pair_webrtcnicetransport0;], rtp-outbound-stream-stats_2311868531=(structure)[outbound-rtp, type=(GstWebRTCStatsType)outbound-rtp, timestamp=(double)595549707.40499997, id=(string)rtp-outbound-stream-stats_2311868531, ssrc=(uint)2311868531, codec-id=(string)codec-stats-sink_0, transport-id=(string)transport-stats_webrtcdtlstransport0, kind=(string)video, bytes-sent=(guint64)8138563, packets-sent=(guint64)7219, fir-count=(uint)0, pli-count=(uint)0, nack-count=(uint)58, gst-rtpsource-stats=(structure)[application/x-rtp-source-stats, ssrc=(uint)2311868531, internal=(boolean)true, validated=(boolean)true, received-bye=(boolean)false, is-csrc=(boolean)false, is-sender=(boolean)true, seqnum-base=(int)-1, clock-rate=(int)90000, octets-sent=(guint64)8138563, packets-sent=(guint64)7219, octets-received=(guint64)8138563, packets-received=(guint64)7219, bytes-received=(guint64)8485075, bitrate=(guint64)813163, packets-lost=(int)0, jitter=(uint)0, sent-pli-count=(uint)0, recv-pli-count=(uint)0, sent-fir-count=(uint)0, recv-fir-count=(uint)0, sent-nack-count=(uint)0, recv-nack-count=(uint)58, recv-packet-rate=(uint)158, have-sr=(boolean)true, sr-ntptime=(guint64)16872275041947140864, sr-rtptime=(uint)1087676117, sr-octet-count=(uint)7969274, sr-packet-count=(uint)7068;];], rtp-remote-inbound-stream-stats_2311868531=(structure)[remote-inbound-rtp, type=(GstWebRTCStatsType)remote-inbound-rtp, timestamp=(double)595549707.40499997, id=(string)rtp-remote-inbound-stream-stats_2311868531, local-id=(string)rtp-outbound-stream-stats_2311868531, ssrc=(uint)2311868531, codec-id=(string)codec-stats-sink_0, transport-id=(string)transport-stats_webrtcdtlstransport0, kind=(string)video, packets-lost=(gint64)-1, jitter=(double)0.00037777777777777777, fraction-lost=(double)0, round-trip-time=(double)0.050506591796875, gst-rtpsource-stats=(structure)[application/x-rtp-source-stats, ssrc=(uint)1545267163, internal=(boolean)false, validated=(boolean)true, received-bye=(boolean)false, is-csrc=(boolean)false, is-sender=(boolean)false, seqnum-base=(int)-1, clock-rate=(int)-1, octets-sent=(guint64)0, packets-sent=(guint64)0, octets-received=(guint64)0, packets-received=(guint64)0, bytes-received=(guint64)0, bitrate=(guint64)0, packets-lost=(int)0, jitter=(uint)0, sent-pli-count=(uint)0, recv-pli-count=(uint)0, sent-fir-count=(uint)0, recv-fir-count=(uint)0, sent-nack-count=(uint)0, recv-nack-count=(uint)0, recv-packet-rate=(uint)0, have-sr=(boolean)false, sr-ntptime=(guint64)0, sr-rtptime=(uint)0, sr-octet-count=(uint)0, sr-packet-count=(uint)0, sent-rb=(boolean)false, sent-rb-fractionlost=(uint)0, sent-rb-packetslost=(int)0, sent-rb-exthighestseq=(uint)0, sent-rb-jitter=(uint)0, sent-rb-lsr=(uint)0, sent-rb-dlsr=(uint)0, have-rb=(boolean)true, rb-ssrc=(uint)2311868531, rb-fractionlost=(uint)0, rb-packetslost=(int)-1, rb-exthighestseq=(uint)17850, rb-jitter=(uint)34, rb-lsr=(uint)1555940004, rb-dlsr=(uint)170622, rb-round-trip=(uint)3310;];], rtp-inbound-stream-stats_1545267163=(structure)[inbound-rtp, type=(GstWebRTCStatsType)inbound-rtp, timestamp=(double)595549707.40499997, id=(string)rtp-inbound-stream-stats_1545267163, ssrc=(uint)1545267163, codec-id=(string)codec-stats-sink_0, transport-id=(string)transport-stats_webrtcdtlstransport0, kind=(string)video, packets-received=(guint64)0, jitter=(double)-0, remote-id=(string)rtp-remote-outbound-stream-stats_1545267163, bytes-received=(guint64)0, fir-count=(uint)0, pli-count=(uint)0, nack-count=(uint)0, gst-rtpsource-stats=(structure)[application/x-rtp-source-stats, ssrc=(uint)1545267163, internal=(boolean)false, validated=(boolean)true, received-bye=(boolean)false, is-csrc=(boolean)false, is-sender=(boolean)false, seqnum-base=(int)-1, clock-rate=(int)-1, octets-sent=(guint64)0, packets-sent=(guint64)0, octets-received=(guint64)0, packets-received=(guint64)0, bytes-received=(guint64)0, bitrate=(guint64)0, packets-lost=(int)0, jitter=(uint)0, sent-pli-count=(uint)0, recv-pli-count=(uint)0, sent-fir-count=(uint)0, recv-fir-count=(uint)0, sent-nack-count=(uint)0, recv-nack-count=(uint)0, recv-packet-rate=(uint)0, have-sr=(boolean)false, sr-ntptime=(guint64)0, sr-rtptime=(uint)0, sr-octet-count=(uint)0, sr-packet-count=(uint)0, sent-rb=(boolean)false, sent-rb-fractionlost=(uint)0, sent-rb-packetslost=(int)0, sent-rb-exthighestseq=(uint)0, sent-rb-jitter=(uint)0, sent-rb-lsr=(uint)0, sent-rb-dlsr=(uint)0, have-rb=(boolean)true, rb-ssrc=(uint)2311868531, rb-fractionlost=(uint)0, rb-packetslost=(int)-1, rb-exthighestseq=(uint)17850, rb-jitter=(uint)34, rb-lsr=(uint)1555940004, rb-dlsr=(uint)170622, rb-round-trip=(uint)3310;];], rtp-remote-outbound-stream-stats_1545267163=(structure)[remote-outbound-rtp, type=(GstWebRTCStatsType)remote-outbound-rtp, timestamp=(double)595549707.40499997, id=(string)rtp-remote-outbound-stream-stats_1545267163, ssrc=(uint)1545267163, codec-id=(string)codec-stats-sink_0, transport-id=(string)transport-stats_webrtcdtlstransport0, remote-timestamp=(double)0, local-id=(string)rtp-inbound-stream-stats_1545267163;], consumer-stats=(structure)[application/x-webrtcsink-consumer-stats, video-encoders=(structure)< [application/x-webrtcsink-video-encoder-stats, bitrate=(int)1252000, mitigation-mode=(GstWebRTCSinkMitigationMode)Lowered resolution, codec-name=(string)video/x-h264, fec-percentage=(uint)0;] >;];];