We demonstrate how to design and apply domain-specific modifications to convolutional neural networks (CNNs) to improve model performance on hyperspectral images of grain kernels. We use hyperspectral images of grain kernels captured in the near-infrared wavelength range of 900 to 1700 nm as a case for supporting our argumentation. This part of the electromagnetic spectrum contains convoluted signals with chemical and physical information relevant to grain quality. For standard chemometric models, domain knowledge is used to select from a plethora of combinations of preprocessing techniques helpful in extracting relevant chemical and physical features for a given task. By incorporating domain-specific design modifications in the preexisting architectures of ResNet-18 and a simple CNN, we show that model performance can be increased significantly and that applying domain knowledge to CNNs is much more important than complexity is to their performance.