va: need to move common logic to create a va lib.
As the va plugins become mature, we want to use it more. Is that possible to extract the common logic(display/memory/surface) from plugins to a lib? Just as the gst-plugins-bad/gst-libs/gst/d3d11 does.
The current common way of using vaapi is:
vaapixxxdec ! vaapixxxenc
or
msdkxxxdec ! msdkxxxenc
the linkages between vaapi and MSDK plugins are few because the DMA sharing support is not very perfect(For example, the different video formats and tilings).
When it comes to vaxxxdec plugins, we must connect vaxxxdec with msdkxxxenc, because the new va plugins does not support encoders and it seems that the encoder part will not be ready within a short period. So the common usage will be
vaxxxdec ! msdkxxxenc
in the future.
And on the new coming Intel's GPU(Gen12+), the "modifier" is a headache. The DMA sharing between va plugins and MSDK need a explicit "modifier" negotiation. And for Intel standalone GPU cards, it has multi GPU groups, and we need to make sure the vaxxxdec and msdkxxxenc run on the same device and GPU group, it is also easy to make mistake. Sharing the VADisplay and VASurface can avoid this.
Another important point is that when we want to extend our support of gstreamer on to windows, we do not find a good way to connect the d3d11xxxdec with the msdk plugins. There is no such a counterpart of the DMA thing on Windows. But fortunately, we find there is already a d3d11 libs existing.
So, our ideal target is: Using vaxxxdec ! msdkxxxenc on Linux, linking them with VAMemory caps and share VADisplay/VASurface.
And using d3d11xxxdec ! msdkxxxenc on Windows, linking them with divx surface caps(not very accurate, need to ask @seungha.yang).
Then, it is symmetric. And so we may want this va lib. This lib can also be used by other modules and customers such as deep learning projects, which is still in development and just want to use our VA surface quickly.