Semi-supervised learning (SSL) for medical image classification has achieved exceptional success on efficiently exploiting knowledge from unlabeled data with limited labeled data. Nevertheless, recent SSL methods suffer from misleading hard-form pseudo labeling, exacerbating the confirmation bias issue due to rough training process. Moreover, the training schemes excessively depend on the quality of generated pseudo labels, which is vulnerable against the inferior ones. In this paper, we propose TEmporal knowledge-Aware Regularization (TEAR) for semi-supervised medical image classification. Instead of using hard pseudo labels to train models roughly, we design Adaptive Pseudo Labeling (AdaPL), a mild learning strategy that relaxes hard pseudo labels to soft-form ones and provides a cautious training. AdaPL is built on a novel theoretically derived loss estimator, which approximates the loss of unlabeled samples according to the temporal information across training iterations, to adaptively relax pseudo labels. To release the excessive dependency of biased pseudo labels, we take advantage of the temporal knowledge and propose Iterative Prototype Harmonizing (IPH) to encourage the model to learn discriminative representations in an unsupervised manner. The core principle of IPH is to maintain the harmonization of clustered prototypes across different iterations. Both AdaPL and IPH can be easily incorporated into prior pseudo labeling-based models to extract features from unlabeled medical data for accurate classification. Extensive experiments on three semi-supervised medical image datasets demonstrate that our method outperforms state-of-the-art approaches. The code is available at https://github.com/CityU-AIM-Group/TEAR.