Pan-sharpening is a significant task in remote sensing image processing, which merges a high-resolution panchromatic (PAN) image and a low-resolution multispectral (MS) image to create a high-resolution MS image. In this paper, we propose a novel deep learning-based MS image pan-sharpening method which combines a Shallow-Deep Convolutional Network (SDCN) and a Spectral Discrimination-based Detail Injection (SDDI) model. SDCN consists of a shallow network and a deep network, which can capture mid-level and high-level spatial features from PAN images. SDDI, inspired by the “Amélioration de la Résolution Spatial par Injection de Structures” (ARSIS) concept, is developed to merge the spatial details extracted by SDCN into MS images with minimal spectral distortion. SDCN and SDDI are collaboratively learned for achieving high spatial resolution MS image and preserving more spectral information. Both the visual assessment and the quantitative assessment results on IKONOS and QuickBird datasets confirmed that the proposed method outperforms several state-of-the-art pan-sharpening methods.