RGB-D camera can provide effective support with additional depth cue for many RGB-D perception tasks beyond traditional RGB information. However, current feature representations based on RGB-D camera utilize depth information only to extract local features, without considering it for the improvement of robustness and discriminability of the feature representation by merging depth cues into feature pooling. Spatial pyramid model (SPM) has become the standard protocol to split 2D image plane into sub-regions for feature pooling in RGB-D object classification. We argue that SPM may not be the optimal pooling scheme for RGB-D images, as it only pools features spatially and completely discards the depth topological information. Instead, we propose a novel joint spatial-depth pooling scheme (JSDP) which further partitions SPM using the depth cue and pools features simultaneous in 2D image plane and the depth direction. Embedding the JSDP with the standard feature extraction and feature encoding modules, we achieve superior performance to the state-of-the-art methods on benchmarks for RGB-D object classification and detection.