Several established tools are available to assess study quality and reporting of randomized controlled trials; however, these tools were designed with clinical intervention trials in mind. In exercise training intervention trials some of the traditional study quality criteria, such as participant or researcher blinding, are extremely difficult to implement. We developed the Tool for the assEssment of Study qualiTy and reporting in EXercise (TESTEX) – a study quality and reporting assessment tool, designed specifically for use in exercise training studies. Our tool is a 15-point scale (5 points for study quality and 10 points for reporting) and addresses previously unmentioned quality assessment criteria specific to exercise training studies. There were no systematic differences between the summated TESTEX scores of each observer [H(2) = 0.392, P = 0.822]. There was a significant association between the summated TESTEX scores of the three observers, with almost perfect agreement between observers 1 and 2 [intra-class correlation coefficient (ICC) = 0.93, 95% confidence interval (CI) 0.82–0.97, P < 0.001], observers 1 and 3 (ICC = 0.96, 95% CI 0.89–0.98, P < 0.001) and observers 2 and 3 (ICC = 0.91, 95% CI 0.75–0.96, P < 0.001). The TESTEX scale is a new, reliable tool, specific to exercise scientists, that facilitates a comprehensive review of exercise training trials.