28 March 2026 in International Journal on Artificial Intelligence Tools
21 April 2026 in SSRN Electronic Journal
Test collections are essential for evaluating retrieval and re-ranking models. However, constructing such collections is challenging due to the high cost of manual annotation, particularly in specialised domains like Algerian legal texts, where high-quality corpora and relevance judgements are not merely scarce but entirely unavailable. To address this limitation, we propose STCALIR, a framework for generating semi-synthetic test collections directly from raw legal documents. The pipeline follows the Cranfield paradigm, maintaining its core components of corpus, topics and relevance judgements, while significantly reducing manual effort through automated multi-stage retrieval and reranking, achieving a 99% reduction in annotation workload. We validate STCALIR using the Mr TyDi benchmark, demonstrating that the resulting semi-synthetic relevance judgements yield retrieval effectiveness comparable to human judgement (Hit@10 ≈0.785). Furthermore, at the system level, the semi-synthetic test judgements demonstrate a strong correlation with human judgements, as measured by Kendall’s 𝜏 (0.89) and Spearman’s 𝜌 (0.92). Overall, STCALIR offers a reproducible and cost-efficient solution for constructing reliable test collections in low-resource domains.