Successful face verification and recognition require matching corresponding points in a pair of images, and it is commonly acknowledged that alignment is a critical step prior to matching. Once aligned, a portion of the image can be compared or features can be extracted. This portion of the image, which we will call the face box, is often just the output of a face detector. While a good deal of effort has been devoted to alignment, the choice of face box has been largely neglected. This paper presents the first systematic study of the shape and size of the face box on face verification accuracy. We use representative algorithms on a dataset that allows for experimentation with differing 3-D pose, blur, noise, misalignment, and background clutter. The experiments lead to clear conclusions and recommendations that can improve the accuracy of other face recognition methods and guide future research.