ci: Bake libdrm into the docker image
Instead of building it in each build job.
This allows dropping its build dependencies from the docker image as well.
Plus miscellaneous other CI improvements & cleanups based on experience from Mesa and other projects.