Background and Purpose: The methodological quality of randomized controlled trials (RCTs) is commonly evaluated in order to assess the risk of biased estimates of treatment effects. The purpose of this systematic review was to identify scales used to evaluate the methodological quality of RCTs in health care research and summarize the content, construction, development, and psychometric properties of these scales.

Background: Manual therapy (MT) and exercise have been extensively used to treat people with musculoskeletal conditions such as temporomandibular disorders (TMD). The evidence regarding their effectiveness provided by early systematic reviews is outdated.Purpose: The aim of this study was to summarize evidence from and evaluate the methodological quality of randomized controlled trials that examined the effectiveness of MT and therapeutic exercise interventions compared with other active interventions or standard care for treatment of TMD.Data Sources: Electronic data searches of 6 databases were performed, in addition to a manual search.Study Selection: Randomized controlled trials involving adults with TMD that compared any type of MT intervention (eg, mobilization, manipulation) or exercise therapy with a placebo intervention, controlled comparison intervention, or standard care were included. The main outcomes of this systematic review were pain, range of motion, and oral function. Forty-eight studies met the inclusion criteria and were analyzed.Data Extraction: Data were extracted in duplicate on specific study characteristics.Data Synthesis: The overall evidence for this systematic review was considered low. The trials included in this review had unclear or high risk of bias. Thus, the evidence was generally downgraded based on assessments of risk of bias. Most of the effect sizes were low to moderate, with no clear indication of superiority of exercises versus other conservative treatments for TMD. However, MT alone or in combination with exercises at the jaw or cervical level showed promising effects.Limitations: Quality of the evidence and heterogeneity of the studies were limitations of the study.Conclusions: No high-quality evidence was found, indicating that there is great uncertainty about the effectiveness of exercise and MT for treatment of TMD.

There is debate on how the methodological quality of clinical trials should be assessed. We compared trials of physical therapy (PT) judged to be of adequate quality based on summary scores from the Physiotherapy Evidence Database (PEDro) scale with trials judged to be of adequate quality by Cochrane Risk of Bias criteria. Meta-epidemiological study within Cochrane Database of Systematic Reviews. Meta-analyses of PT trials were identified in the Cochrane Database of Systematic Reviews. For each trial PeDro and Cochrane assessments were extracted from the PeDro and Cochrane databases. Adequate quality was defined as adequate generation of random sequence, concealment of allocation, and blinding of outcome assessors (Cochrane criteria) or as trials with a PEDro summary score ≥5 or ≥6 points. We combined trials of adequate quality using random-effects meta-analysis. Forty-one Cochrane reviews and 353 PT trials were included. All meta-analyses included trials with PEDro scores ≥5, 37 (90.2%) included trials with PEDro scores ≥6 and only 22 (53.7%) meta-analyses included trials of adequate quality according to the Cochrane criteria. Agreement between PeDro and Cochrane was poor for PeDro scores of ≥5 points (kappa = 0.12; 95% CI 0.07 to 0.16) and slight for ≥6 points (kappa 0.24; 95% CI 0.16-0.32). When combining effect sizes of trials deemed to be of adequate quality according to PEDro or Cochrane criteria, we found that a substantial difference in the combined effect size (≥0.15) was evident in 9 (22%) out of the 41 meta-analyses for PEDro cutoff ≥5 and 10 (24%) for cutoff ≥6. The PeDro and Cochrane approaches lead to different sets of trials of adequate quality, and different combined treatment estimates from meta-analyses of these trials. A consistent approach to assessing RoB in trials of physical therapy should be adopted.

To test the inter-rater reliability of the RoB tool applied to Physical Therapy (PT) trials by comparing ratings from Cochrane review authors with those of blinded external reviewers. Randomized controlled trials (RCTs) in PT were identified by searching the Cochrane Database of Systematic Reviews for meta-analysis of PT interventions. RoB assessments were conducted independently by 2 reviewers blinded to the RoB ratings reported in the Cochrane reviews. Data on RoB assessments from Cochrane reviews and other characteristics of reviews and trials were extracted. Consensus assessments between the two reviewers were then compared with the RoB ratings from the Cochrane reviews. Agreement between Cochrane and blinded external reviewers was assessed using weighted kappa (κ). In total, 109 trials included in 17 Cochrane reviews were assessed. Inter-rater reliability on the overall RoB assessment between Cochrane review authors and blinded external reviewers was poor (κ = 0.02, 95%CI: −0.06, 0.06]). Inter-rater reliability on individual domains of the RoB tool was poor (median κ = 0.19), ranging from κ = −0.04 (“Other bias”) to κ = 0.62 (“Sequence generation”). There was also no agreement (κ = −0.29, 95%CI: −0.81, 0.35]) in the overall RoB assessment at the meta-analysis level. Risk of bias assessments of RCTs using the RoB tool are not consistent across different research groups. Poor agreement was not only demonstrated at the trial level but also at the meta-analysis level. Results have implications for decision making since different recommendations can be reached depending on the group analyzing the evidence. Improved guidelines to consistently apply the RoB tool and revisions to the tool for different health areas are needed.