Automated giant panda (Ailuropoda melanoleuca) behavior recognition (GPBR) systems are highly beneficial for efficiently monitoring giant pandas in wildlife conservation missions. While video-based behavior recognition attracts a lot of attention, few studies have focused on audio-based methods. In this paper, we propose the exploitation of the audio data recorded by collar-mounted devices on giant pandas for the purpose of GPBR. We construct a new benchmark audio dataset of giant pandas named abPanda-5 for GPBR, which consists of 18,930 samples from five giant panda individuals with five main behaviors. To fully explore the bioacoustic features, we propose an audio-based method for automatic GPBR using competitive fusion learning. The method improves behavior recognition accuracy and robustness, without additional computational overhead in the inference stage. Experiments performed on the abPanda-5 dataset demonstrate the feasibility and effectiveness of our proposed method.