1 January 2016 in Journal of Clinical Epidemiology
To develop ROBIS, a new tool for assessing the risk of bias in systematic reviews (rather than in primary studies). We used four-stage approach to develop ROBIS: define the scope, review the evidence base, hold a face-to-face meeting, and refine the tool through piloting. ROBIS is currently aimed at four broad categories of reviews mainly within health care settings: interventions, diagnosis, prognosis, and etiology. The target audience of ROBIS is primarily guideline developers, authors of overviews of systematic reviews ("reviews of reviews"), and review authors who might want to assess or avoid risk of bias in their reviews. The tool is completed in three phases: (1) assess relevance (optional), (2) identify concerns with the review process, and (3) judge risk of bias. Phase 2 covers four domains through which bias may be introduced into a systematic review: study eligibility criteria; identification and selection of studies; data collection and study appraisal; and synthesis and findings. Phase 3 assesses the overall risk of bias in the interpretation of review findings and whether this considered limitations identified in any of the phase 2 domains. Signaling questions are included to help judge concerns with the review process (phase 2) and the overall risk of bias in the review (phase 3); these questions flag aspects of review design related to the potential for bias and aim to help assessors judge risk of bias in the review process, results, and conclusions. ROBIS is the first rigorously developed tool designed specifically to assess the risk of bias in systematic reviews.
9 February 2010 in Implementation Science
Background: There is growing interest in the use of cognitive, behavioural, and organisational theories in implementation research. However, the extent of use of theory in implementation research is uncertain.Methods: We conducted a systematic review of use of theory in 235 rigorous evaluations of guideline dissemination and implementation studies published between 1966 and 1998. Use of theory was classified according to type of use (explicitly theory based, some conceptual basis, and theoretical construct used) and stage of use (choice/design of intervention, process/mediators/moderators, and post hoc/explanation).Results: Fifty-three of 235 studies (22.5%) were judged to have employed theories, including 14 studies that explicitly used theory. The majority of studies (n = 42) used only one theory; the maximum number of theories employed by any study was three. Twenty-five different theories were used. A small number of theories accounted for the majority of theory use including PRECEDE (Predisposing, Reinforcing, and Enabling Constructs in Educational Diagnosis and Evaluation), diffusion of innovations, information overload and social marketing (academic detailing).Conclusions: There was poor justification of choice of intervention and use of theory in implementation research in the identified studies until at least 1998. Future research should explicitly identify the justification for the interventions. Greater use of explicit theory to understand barriers, design interventions, and explore mediating pathways and moderators is needed to advance the science of implementation research.
25 October 2021 in Systematic Reviews
Background: AMSTAR-2 is a 16-item assessment tool to check the quality of a systematic review and establish whether the most important elements are reported. ROBIS is another assessment tool which was designed to evaluate the level of bias present within a systematic review. Our objective was to compare, contrast and establish both inter-rater reliability and usability of both tools as part of two overviews of systematic reviews. Strictly speaking, one tool assesses methodological quality (AMSTAR-2) and the other assesses risk of bias (ROBIS), but there is considerable overlap between the tools in terms of the signalling questions. Methods: Three reviewers independently assessed 31 systematic reviews using both tools. The inter-rater reliability of all sub-sections using each instrument (AMSTAR-2 and ROBIS) was calculated using Gwet’s agreement coefficient (AC1 for unweighted analysis and AC2 for weighted analysis). Results: Thirty-one systematic reviews were included. For AMSTAR-2, the median agreement for all questions was 0.61. Eight of the 16 AMSTAR-2 questions had substantial agreement or higher (> 0.61). For ROBIS, the median agreement for all questions was also 0.61. Eleven of the 24 ROBIS questions had substantial agreement or higher. Conclusion: ROBIS is an effective tool for assessing risk of bias in systematic reviews and AMSTAR-2 is an effective tool at assessing quality. The median agreement between raters for both tools was identical (0.61). Reviews that included a meta-analysis were easier to rate with ROBIS; however, further developmental work could improve its use in reviews without a formal synthesis. AMSTAR-2 was more straightforward to use; however, more response options would be beneficial.
18 October 2013 in Cochrane Database of Systematic Reviews
6 May 2021 in Humanities and Social Sciences Communications
Researchers are increasingly encouraged to co-produce research, involving members of the public, service users, policy makers and practitioners in more equal relationships throughout a research project. The sharing of power is often highlighted as a key principle when co-producing research. However, health and social care research, as with many other academic disciplines, is carried out within embedded hierarchies and structural inequalities in universities, public service institutions, and research funding systems—as well as in society more broadly. This poses significant challenges to ambitions for co-production. This article explores the difficulties that are faced when trying to put ideal co-production principles into practice. A reflective account is provided of an interdisciplinary project that aimed to better understand how to reduce power differentials within co-produced research. The project facilitated five workshops, involving researchers from different disciplines, health, social care and community development staff and public contributors, who all had experience in co-production within research. In the workshops, people discussed how they had attempted to enable more equal relationships and shared ideas that supported more effective and equitable co-produced research. Shared interdisciplinary learning helped the project team to iteratively develop a training course, a map of resources and reflective questions to support co-produced research. The gap between co-production principles and practice is challenging. The article examines the constraints that exist when trying to share power, informed by multidisciplinary theories of power. To bring co-production principles into practice, changes are needed within research practices, cultures and structures; in understandings of what knowledge is and how different forms of knowledge are valued. The article outlines challenges and tensions when co-producing research and describes potential ideas and resources that may help to put co-production principles into practice. We highlight that trying to maintain all principles of co-production within the real-world of structural inequalities and uneven distribution of resources is a constant challenge, often remaining for now in the realm of aspiration.
11 April 2014 in British Journal of Anaesthesia
Undergoing general anaesthesia is distressing for children, with up to two-thirds demonstrating abnormal behaviours after their procedure, such as emergence delirium (ED) and post-hospitalization behaviour change. The aim of this systematic review was to determine the effect of intraoperative i.v. α2-adrenergic agonists on postoperative behaviour in children. We included published full-text reports of randomized controlled trials involving children who received i.v. clonidine or dexmedetomidine after induction of general anaesthesia, who were assessed for postoperative behavioural disturbance. After screening of references identified by the search strategy, a data collection form was developed and piloted. Data extraction was performed by one reviewer and checked by a second. Twelve randomized trials met the inclusion criteria. Ten studies (n=669) reported dichotomous data that were included in the pooled analysis of α2-adrenergic agonists vs placebo. There was strong evidence that i.v. α2-adrenergic agonists reduced postoperative ED (overall summary odds ratio 0.28, 95% confidence interval 0.19–0.40, P<0.001). No studies examined post-hospitalization negative behaviour changes. There was evidence that α2-adrenergic agonists prolonged time in the recovery room. No adverse haemodynamic events were reported in any arm of any study. This meta-analysis provides evidence that intraoperative i.v. α2-adrenergic agonists reduce the incidence of emergency delirium in children. The prolongation of time in recovery is unlikely to be clinically relevant. The absence of data regarding the effect on post-hospitalization behavioural changes provides an opportunity for future research.