This is basically not-portable (I doubt unaligned loads in this context are faster than performing scalar operations), and possibly inefficient on x86 too.

To make it account for potentially random alignment will be awkward, but it might be possible to do efficiently.