Skip to content

xserver: fix RGB mask handling

Leandro Lupori requested to merge luporl/xserver:fix-rgb-mask-handling into master

On FreeBSD 13.0-CURRENT for PowerPC64 big-endian (BE), X was crashing in some cases. For instance, when twm was started and the background was clicked to open its menu, X crashed with a segmentation fault, trying to dereference a null pointer at CreatePicture().

There were 2 issues with xorg-server handling of RGB masks that caused the pointer above to be null and thus the crash:

  • wrong use of ffs() to get the RGB offsets from the masks
  • overflow when shifting a 16-bit integer

This change fixes both issues. They happen when the system is BE but has a video adapter using a little-endian (LE) ARGB32 framebuffer. In order to display the correct colors, this setup requires a BE RGBA32 color format to be used by X, by setting the RGB masks appropriately, that didn't work properly because of the issues above.

Merge request reports