Podman: bump the lock limit
Podman has a limit of 2048 containers, pods, and volumes by default. This may be hit relatively-quickly because the gitlab-runner tends to generate a lot of volumes as a caching mechanism.
Freedesktop recently bumped the limit, and I think we should consider doing the same: https://gitlab.freedesktop.org/freedesktop/helm-gitlab-config/-/commit/b11cddfd9180ed0ae8ec025bb9b3ffbb95bf247d
Here is relevant information I found while discussing this issue with bentiss:
num_locks=2048
Number of locks available for containers and pods. Each created container or pod consumes one lock. The default number available is 2048. If this is changed, a lock renumbering must be performed, using the podman system renumber command.
And we might want to add podman system renumber
early in the boot process, as otherwise it may not apply. See https://docs.podman.io/en/latest/markdown/podman-system-renumber.1.html for more details.