Skip to content

Draft: audioconvert: improve precision of S32<->F32 conversions

Roman Lebedev requested to merge LebedevRI/pipewire:fp32 into master

The largest integer that 32-bit floating point can exactly represent is actually (2^24)-1, not(2^23)-1 like the code assumes. This means, whenever we use s24 as an intermediate step to go between f32 and s32, we lose 1 LSB bit of precision.

For S32->F32, we can do even better for certain cases. For F32->S32, improving precision further is complicated (2147483647 is actually 2147483648.0f).

Refs. #4057 (closed)

Merge request reports