Measure SO(3) error in matrix space

The original implementation measured the error of SO(3)-valued
functions in quaternion space in a few places.  This patch
converts another occurrence of this to matrix space.