Constrained spherical deconvolution (CSD) is the most widely used algorithm to estimate fiber orientations for tractography in diffusion-weighted magnetic resonance imaging. CSD models the diffusion-weighted signal as the convolution of a fiber orientation distribution function and a “single fiber response function”, representing the signal profile of a population of aligned fibers. The performance of CSD relies crucially on the robust and accurate estimation of this response function, which is typically done by aligning and averaging a set of noisy, rotated single fiber signals. We show that errors in the alignment step of this procedure lead to an observable bias, and introduce an alternative algorithm based on rotational invariants that entirely avoids the problematic alignment step. The corresponding estimator is proven to be unbiased and consistent, which is verified experimentally.