ci: Bake libdrm into the docker image

Instead of building it in each build job.

This allows dropping its build dependencies from the docker image as well.

Plus miscellaneous other CI improvements & cleanups based on experience from Mesa and other projects.