Super-Resolution (SR) techniques can compensate for the missing information of low-resolution images and further promote experts and algorithms to make accurate diagnosis decisions. Although the existing pixel-loss based SR works produce high-resolution images with impressive objective metrics, the over-smoothed contents that lose high-frequency information would disturb the visual experience and the subsequent diagnosis. To address this issue, we propose a joint Spatial-Wavelet super-resolution Network (SWD-Net) with collaborative Dual-stream. In the spatial stage, a Refined Context Fusion (RCF) is proposed to iteratively rectify the features by a counterpart stream with compensative receptive fields. After that, the wavelet stage enhances the reconstructed images, especially the structural boundaries. Specifically, we design the tailor-made Wavelet Features Adaptation (WFA) to adjust the wavelet coefficients for better compatibility with networks and Wavelet-Aware Convolutional blocks (WAC) to exploit features in the wavelet domain efficiently. We further introduce the wavelet coefficients supervision together with the traditional spatial loss to jointly optimize the network and obtain the high-frequency enhanced SR images. To evaluate the SR for medical images, we build a benchmark dataset with histopathology images and evaluate the proposed SWD-Net under different settings. The comprehensive experiments demonstrate our SWD-Net outperforms state-of-the-art methods. Furthermore, SWD-Net is proven to promote medical image diagnosis with a large margin. The source code and dataset are available at https://github.com/franciszchen/SWD-Net.