Systematic reviews, which assess the risk of bias in included studies, are increasingly used to develop environmental hazard assessments and public health guidelines. These research areas typically rely on evidence from human observational studies of exposures, yet there are currently no universally accepted standards for assessing risk of bias in such studies. The risk of bias in non-randomised studies of exposures (ROBINS-E) tool has been developed by building upon tools for risk of bias assessment of randomised trials, diagnostic test accuracy studies and observational studies of interventions. This paper reports our experience with the application of the ROBINS-E tool. We applied ROBINS-E to 74 exposure studies (60 cohort studies, 14 case-control studies) in 3 areas: environmental risk, dietary exposure and drug harm. All investigators provided written feedback, and we documented verbal discussion of the tool. We inductively and iteratively classified the feedback into 7 themes based on commonalities and differences until all the feedback was accounted for in the themes. We present a description of each theme. We identified practical concerns with the premise that ROBINS-E is a structured comparison of the observational study being rated to the ‘ideal’ randomised controlled trial. ROBINS-E assesses 7 domains of bias, but relevant questions related to some critical sources of bias, such as exposure and funding source, are not assessed. ROBINS-E fails to discriminate between studies with a single risk of bias or multiple risks of bias. ROBINS-E is severely limited at determining whether confounders will bias study outcomes. The construct of co-exposures was difficult to distinguish from confounders. Applying ROBINS-E was time-consuming and confusing. Our experience suggests that the ROBINS-E tool does not meet the need for an international standard for evaluating human observational studies for questions of harm relevant to public and environmental health. We propose that a simpler tool, based on empirical evidence of bias, would provide accurate measures of risk of bias and is more likely to meet the needs of the environmental and public health community.
Ultrasound imaging is widely used to measure architectural features of human skeletal muscles in vivo. We systematically reviewed studies of the reliability and validity of two-dimensional ultrasound measurement of muscle fascicle lengths or pennation angles in human skeletal muscles. A comprehensive search was conducted in June 2011. Thirty-six reliability studies and six validity studies met the inclusion criteria. Data from these studies indicate that ultrasound measurements of muscle fascicle lengths are reliable across a broad range of experimental conditions [intraclass correlation coefficient (ICC) and r values were always > 0.6, and coefficient of variation values were always < 10%]. The reliability of measurements of pennation angles is broadly similar (ICC and r values were always > 0.5 and coefficient of variation values were always < 14%). Data on validity are less extensive and probably less robust, but suggest that measurement of fascicle lengths and pennation angles are accurate (ICC > 0.7) under certain conditions, such as when large limb muscles are imaged in a relaxed state and the limb or joint remains stationary. Future studies on validity should consider ways to test for the validity of two-dimensional ultrasound imaging in contracted or moving muscles and the best method of probe alignment.
What is the incidence of recovery of ambulation and upper limb function six months after stroke? Can measures such as age and the National Institutes of Health Stroke Scale (NIHSS) be used to develop models to predict the recovery of ambulation and upper limb function? Prospective cohort study. Consecutive sample of 200 people with stroke admitted to a Sydney Hospital. Ambulation was measured with item 5 of the Motor Assessment Scale (MAS); patients scoring ≥3 could ambulate independently. Upper limb function was measured with items 7 and 8 of the MAS; patients scoring ≥5 could move a cup across the table and feed themselves with a spoonful of liquid with the hemiplegic arm. Of the 114 stroke survivors who were unable to ambulate initially, 80 (70%) achieved independent ambulation at six months. Of the 51 stroke survivors who could not move a cup across the table initially, 21 (41%) achieved the upper limb task at six months. Of the 56 stroke survivors who were unable to feed themselves initially, 25 (45%) could feed themselves at six months. Models containing age and severity of stroke (measured with NIHSS) predicted recovery of ambulation and ability to move a cup across the table, whilst a model containing severity of stroke predicted ability to feed oneself. All prediction models showed good discrimination (AUC 0.73 to 0.84). More than two-thirds of people after stroke recovered independent ambulation and less than half recovered upper limb function at six months. Models using age and NIHSS can predict independent ambulation and upper limb function but these prediction models now require external validation before use in clinical practice.
The Journal of Physiology and British Journal of Pharmacology jointly published an editorial series in 2011 to improve standards in statistical reporting and data analysis. It is not known whether reporting practices changed in response to the editorial advice. We conducted a cross-sectional analysis of reporting practices in a random sample of research papers published in these journals before (n = 202) and after (n = 199) publication of the editorial advice. Descriptive data are presented. There was no evidence that reporting practices improved following publication of the editorial advice. Overall, 76-84% of papers with written measures that summarized data variability used standard errors of the mean, and 90-96% of papers did not report exact p-values for primary analyses and post-hoc tests. 76-84% of papers that plotted measures to summarize data variability used standard errors of the mean, and only 2-4% of papers plotted raw data used to calculate variability. Of papers that reported p-values between 0.05 and 0.1, 56-63% interpreted these as trends or statistically significant. Implied or gross spin was noted incidentally in papers before (n = 10) and after (n = 9) the editorial advice was published. Overall, poor statistical reporting, inadequate data presentation and spin were present before and after the editorial advice was published. While the scientific community continues to implement strategies for improving reporting practices, our results indicate stronger incentives or enforcements are needed.