Measure error for SO(3) in matrix space

Previously, the error norms where computed in the respective global
embedding space of the TargetSpace.  For the case of Rotation<3>
this was the space of quaternions, because this is how the
Rotation class is implemented.  However, this does not lead to
correct results.  Therefore, this patch adds special handling
for Rotation<3>, and computes the errors norms in the space of
3x3 matrices.