Most existing deep learning-based methods use feature maps extracted from convolutional neural networks (CNNs) for classification and detection of high-resolution remote sensing images (HRSI). However, directly applying these features to classification and object detection in HRSI is problematic because of rotational variations. In this work, we design networks using the polycentric circle pooling (PCP) strategy to alleviate the above problem. The PCP network (PCP-net) structure can generate a fixed-length representation for different input image sizes and encode rotation-invariant information. With these advantages, PCP-net should in general improve the CNN-based HRSI classification methods. Specifically, on the basis of the concentric circle pooling network structure, we improve the structure using multiple concentric circle centers to generate more robust rotation-invariant information. Using two challenging HRSI scene datasets, we prove that PCP-net improves the accuracy of CNN architectures for scene classification tasks. PCP-net can be conveniently applied to object detection because the output size is fixed regardless of image size. Experiments applying the Faster region-CNN to a publicly available 10-class object detection dataset demonstrate that our proposed polycentric circle pooling can achieve accuracy higher than that of region of interest pooling in the HRSI object detection task.