In this work, we propose a new algorithm for boosting Deep Convolutional Neural Networks (BoostCNN) to combine the merits of boosting and these networks. To learn this new model, we propose a novel algorithm to incorporate boosting weights into the deep learning architecture based on least square objective function. We also show that it is possible to use networks of different structures within the proposed boosting framework and BoostCNN is able to select the best network structure in each iteration. This not only results in superior performance but also reduces the required manual effort for finding the right network structure. Experiments show that the proposed method is able to achieve state-of-the-art performance on several fine-grained classification tasks such as bird, car, and aircraft classification.