alloc: Allow disabling memfd usage at runtime with XSHMFENCE_NO_MEMFD=1

For thin VMs used for compatibility reasons, it is desirable to directly proxy the X11 protocol between an application running in a VM and an X server running on the host, including DRI3 support (with cross-domain buffer sharing using virtgpu). We can proxy futex wakeups with some watcher code, but we need the futex memory page itself to be shared so the atomics work as intended (and to avoid overhead in the happy path). This works when /dev/shm is mounted with virtiofs with DAX, but cannot work for memfds.

Add a XSHMFENCE_NO_MEMFD environment variable that, when set, disables the memfd codepath and falls back to /dev/shm.

We have a proof-of-concept of this proxying here (guest proxy side) with the host side in libkrunfw (note: I just fixed a bunch of bugs last night, those branches aren't up to date with the currently working state of code with the new /dev/shm fd passing approach that is needed for this PR to work as intended, but should be updated soon). We've explored a number of alternatives:

• Using ptrace on the client app to yank out the memfd mapping and replace it with a /dev/shm mapping when the fd is sent to the X server. This works (and is implemented as a fallback in x112virtgpu), but it is quite fragile and it breaks using things like debuggers and strace on X clients.
• Using an LD_PRELOAD to hijack libxshmfence. This works, but since we're also doing all this across three architectures on a single machine (aarch64, emulated x86, emulated x86-64), carrying around three copies of the hijack library is quite painful and can also have poor interactions with the emulation and containerization layers involved.
• Using seccomp-bpf to disable memfd_create(). This works well, but since seccomp-bpf cannot dereference the name argument to memfd_create() and test it, we have to do it for all users of this syscall, not just libxshmfence, which is a pretty big hammer and might break other things.
• Not proxying X, instead run XWayland in the guest and proxy Wayland. This is actually what we're doing right now, but it's quite broken due to many issues with sommelier (the Wayland proxying compositor we're using). Even if all of those issues were fixed, we'd still lose on the native compositor's integration with its own hosted XWayland instance, which is often significantly more complete (e.g. tray icons, DPI scaling, and things like that). Further, it prevents direct interactions between guest X apps and host X apps (e.g. XEmbed), which is required for use cases such as running DAW VST plugins emulated in the guest and hosted in a DAW in the host. Proxying X directly eliminates all these problems.

Backstory on why we're doing all this crazy stuff: https://alx.sh/gaming

Signed-off-by: Asahi Lina lina@asahilina.net