Oracle bone inscriptions are China’s earliest writing system, dating back 3000 years. Usually carved on bones and tortoise shells, they serve as invaluable records of early Chinese civilisation. Due to erosion, a large number of oracle bones have been fragmented into small pieces. This fragmentation often results in incomplete sentences and the loss of contextual information, which poses significant challenges for accurate interpretation and digital reconstruction. Previous rejoining methods mainly rely on edge patterns, while determining bone-level associations through contextual information remains largely unsolved. To address this issue, we present the first public benchmark for predicting bone-level associations of oracle bone inscription sentences. We also propose a novel multi-modal deep learning method to predict associations between sentence pairs that achieves competitive performance. The proposed dataset and method aim to assist researchers in identifying likely associations among fragments, thereby facilitating the reconstruction and understanding of damaged oracle bone inscription texts.