Hyperspectral image cross-scene classification (HSICC) tasks are confronted with tremendous challenges due to spectral shift phenomena across scenes and the tough work of obtaining labels. Unsupervised domain adaptation has proven its effectiveness in tackling this issue, but it has a fatal limitation of intending to narrow the disparity between source and target domains by utilizing fully labeled source data and unlabeled target data. However, it is costly even to attain labels from source domains in many cases, rendering sufficient labeling as used in prior work impractical. In this work, we investigate an extreme and realistic scenario where unsupervised domain adaptation methods encounter sparsely labeled source data when handling HSICC tasks, namely, few-shot unsupervised domain adaptation. We propose an end-to-end refined bi-directional prototypical contrastive learning (RBPCL) framework for overcoming the HSICC problem with only a few labeled samples in the source domain. RBPCL captures category-level semantic features of hyperspectral data and performs feature alignment through in-domain refined prototypical self-supervised learning and bi-directional cross-domain prototypical contrastive learning, respectively. Furthermore, our framework introduces the class-balanced multicentric dynamic prototype strategy to generate more robust and representative prototypes. To facilitate prototype contrastive learning, we employ a Siamese-style distance metric loss function to aggregate intra-class features while increasing the discrepancy of inter-class features. Finally, extensive experiments and ablation analysis implemented on two public cross-scene data pairs and three pairs of self-collected ultralow-altitude hyperspectral datasets under different illumination conditions verify the effectiveness of our method, which will further enhance the practicality of hyperspectral intelligent sensing technology.