XAnnoRank is a web-based framework designed to support explain-ability studies in rankings via user-centered evaluation. XAnno-Rank provides an easily adaptable user interface (UI) that enables systematic assessment of explainability methods by presenting ranked lists with and without explanations, while collecting both implicit and explicit user feedback. Its flexibility facilitates diverse user design, including different explanation types, datasets, ranking models, and researcher-defined configurations. The tool has been successfully demonstrated and validated in a high-stakes domain through a study on explainability in recruitment conducted as part of the EU-funded FINDHR1 project.2 By streamlining the UI setup, XAnnoRank reduces implementation efforts and enables reproducibility across studies. The code and documentation3 are publicly available, together with a demonstration video4.