6 January 2021 in Wellcome Open Research
Background: Behaviour change interventions (BCI), their contexts and evaluation methods are heterogeneous, making it difficult to synthesise evidence and make recommendations for real-world policy and practice. Ontologies provide a means for addressing this. They represent knowledge formally as entities and relationships using a common language able to cross disciplinary boundaries and topic domains. This paper reports the development of the upper level of the Behaviour Change Intervention Ontology (BCIO), which provides a systematic way to characterise BCIs, their contexts and their evaluations. Methods: Development took place in four steps. (1) Entities and relationships were identified by behavioural and social science experts, based on their knowledge of evidence and theory, and their practical experience of behaviour change interventions and evaluations. (2) The outputs of the first step were critically examined by a wider group of experts, including the study ontology expert and those experienced in annotating relevant literature using the initial ontology entities. The outputs of the second step were tested by (3) feedback from three external international experts in ontologies and (4) application of the prototype upper-level BCIO to annotating published reports; this informed the final development of the upper-level BCIO. Results: The final upper-level BCIO specifies 42 entities, including the BCI scenario, elaborated across 21 entities and 7 relationship types, and the BCI evaluation study comprising 10 entities and 9 relationship types. BCI scenario entities include the behaviour change intervention (content and delivery), outcome behaviour, mechanism of action, and its context, which includes population and setting. These entities have corresponding entities relating to the planning and reporting of interventions and their evaluations. Conclusions: The upper level of the BCIO provides a comprehensive and systematic framework for representing BCIs, their contexts and their evaluations.
26 February 2021 in Wellcome Open Research
Background: Investigating and improving the effects of behaviour change interventions requires detailed and consistent specification of all aspects of interventions. An important feature of interventions is the way in which these are delivered, i.e. their mode of delivery. This paper describes an ontology for specifying the mode of delivery of interventions, which forms part of the Behaviour Change Intervention Ontology, currently being developed in the Wellcome Trust funded Human Behaviour-Change Project. Methods: The Mode of Delivery Ontology was developed in an iterative process of annotating behaviour change interventions evaluation reports, and consulting with expert stakeholders. It consisted of seven steps: 1) annotation of 110 intervention reports to develop a preliminary classification of modes of delivery; 2) open review from international experts (n=25); 3) second round of annotations with 55 reports to test inter-rater reliability and identify limitations; 4) second round of expert review feedback (n=16); 5) final round of testing of the refined ontology by two annotators familiar and two annotators unfamiliar with the ontology; 6) specification of ontological relationships between entities; and 7) transformation into a machine-readable format using the Web Ontology Language (OWL) and publishing online. Results: The resulting ontology is a four-level hierarchical structure comprising 65 unique modes of delivery, organised by 15 upper-level classes: Informational, Environmental change, Somatic, Somatic alteration, Individual-based/ Pair-based /Group-based, Uni-directional/Interactional, Synchronous/ Asynchronous, Push/ Pull, Gamification, Arts feature. Relationships between entities consist of is_a. Inter-rater reliability of the Mode of Delivery Ontology for annotating intervention evaluation reports was a=0.80 (very good) for those familiar with the ontology and a= 0.58 (acceptable) for those unfamiliar with it. Conclusion: The ontology can be used for both annotating and writing behaviour change intervention evaluation reports in a consistent and coherent manner, thereby improving evidence comparison, synthesis, replication, and implementation of effective interventions.
18 December 2020 in Wellcome Open Research
Background: Behaviour and behaviour change are integral to many aspects of wellbeing and sustainability. However, reporting behaviour change interventions accurately and synthesising evidence about effective interventions is hindered by lacking a shared, scientific terminology to describe intervention characteristics. Ontologies are standardised frameworks that provide controlled vocabularies to help unify and connect scientific fields. To date, there is no published guidance on the specific methods required to develop ontologies relevant to behaviour change. We report the creation and refinement of a method for developing ontologies that make up the Behaviour Change Intervention Ontology (BCIO). Aims: (1) To describe the development method of the BCIO and explain its rationale; (2) To provide guidance on implementing the activities within the development method. Method and results: The method for developing ontologies relevant to behaviour change interventions was constructed by considering principles of good practice in ontology development and identifying key activities required to follow those principles. The method’s details were refined through application to developing two ontologies. The resulting ontology development method involved: (1) defining the ontology’s scope; (2) identifying key entities; (3) refining the ontology through an iterative process of literature annotation, discussion and revision; (4) expert stakeholder review; (5) testing inter-rater reliability; (6) specifying relationships between entities, and; (7) disseminating and maintaining the ontology. Guidance is provided for conducting relevant activities for each step. Conclusions: We have developed a detailed method for creating ontologies relevant to behaviour change interventions, together with practical guidance for each step, reflecting principles of good practice in ontology development. The most novel aspects of the method are the use of formal mechanisms for literature annotation and expert stakeholder review to develop and improve the ontology content. We suggest the mnemonic SELAR3, representing the method’s first six steps as Scope, Entities, Literature Annotation, Review, Reliability, Relationships.
9 December 2024 in Wellcome Open Research
Background Advancing and applying behavioural science requires clear, consistent and coherent descriptions of behaviour change interventions. Ontologies provide a means to do this and enable information to be computer-readable. This study aimed to develop an ontology describing the ‘schedule of delivery’ of behaviour change interventions within the Behaviour Change Intervention Ontology (BCIO). Methods The Behaviour Change Intervention Schedule of Delivery Ontology (BCISDO) was developed following the method developed for the BCIO: 1) defining the ontology’s scope through literature review, 2) specifying the entities, their labels, definitions and relationships, 3) refining the ontology by piloting its use in annotations, 4) stakeholder review of the ontology’s comprehensiveness and clarity; 5) inter-rater reliability testing through annotating intervention reports using the ontology; 6) specifying the relationships between classes; and 7) publishing a machine-readable version of the ontology. Results Schedule of Delivery was defined as ‘A behaviour change intervention attribute that is its temporal organisation’. The initial version comprised 9 classes (e.g., intervention temporal part, intervention duration). Following annotations, stakeholder review and team discussions, four classes were added. Inter-rater reliability was α= 0.54 for researchers familiar with the ontology and, following updates to the annotation guidance, α=0.60 (acceptable) for researchers unfamiliar with it. Differences were readily resolved through discussion. The results from inter-rater reliability and further team discussions led to additional changes in the ontology. The final ontology contains 49 classes; 14 refer to the duration and frequency of an intervention, its temporal parts and contact events, and 45 refer to their associated aggregation data items (e.g. mean, maximum). Conclusions The BCISDO provides a reliable means of describing the temporal organisation of behaviour change interventions. It is linked with other BCIO ontologies describing characteristics of delivery (e.g. mode of delivery), content and dose (i.e. the amount), contributing to the accumulation of evidence in behavioural science.
5 March 2025 in Wellcome Open Research
Background The uptake, effectiveness and generalisability of interventions are influenced by the features of the populations targeted. However, populations exposed to interventions are not consistently specified in published reports. Purpose To create an Intervention Population Ontology providing a clear, usable and reliable classification system to specify characteristics of populations exposed to interventions. Methods The Intervention Population Ontology was developed in seven main stages 1) Defining the ontology’s scope, (2) identifying key entities by reviewing existing classification systems (top-down) and 100 intervention reports (bottom-up), 3) Refining the preliminary ontology by annotating ~150 intervention reports, 4) Stakeholder review by 29 behavioural science and public health experts, 5) Assessing inter-rater reliability of using the ontology by two coders familiar with the ontology and two coders unfamiliar with it, 6) Specifying ontological relationships between entities in the ontology and 7) making the Intervention Population Ontology machine-readable using Web Ontology Language (OWL) and publishing online. Results The Intervention Population Ontology features 218 entities representing attributes of human individuals across 12 key groupings: personal attributes, geographic location, person, quality, mental capability, role, expertise, objects possessed, behaviour, personal vulnerability and personal history. It has a further 666 classes relating to how individual-level attributes are aggregated to describe groups of people. Inter-rater reliability was α=0.79 for coders familiar with the ontology and 0.85 for coders unfamiliar with the ontology. Conclusions The Intervention Population Ontology can be applied to specify precisely information from diverse sources, annotate population characteristics in existing intervention evaluation reports and guide future reporting.
2 December 2021 in medRxiv
Objectives: Concerns on the lack of reproducibility and transparency in science have led to a range of research practice reforms, broadly referred to as ‘Open Science’. The extent that physical activity interventions are embedding Open Science practices is currently unknown. In this study, we randomly sampled 100 reports of recent physical activity behaviour change interventions to estimate the prevalence of Open Science practices. Methods: One hundred reports of randomised controlled trial physical activity behaviour change interventions published between 2018-2021 were identified. Open Science practices were coded in identified reports, including: study pre-registration, protocol sharing, data-, materials- and analysis scripts-sharing, replication of a previous study, open access publication, funding sources and conflict of interest statements. Coding was performed by two independent researchers, with inter-rater reliability calculated using Krippendorff’s alpha. Results: 78% of the 100 reports provided details of study pre-registration and 41% provided evidence of a published protocol. 4% provided accessible open data, 8% provided open materials and 1% provided open analysis scripts. 73% of reports were published as open access and no studies were described as replication attempts. 93% of reports declared their sources of funding and 88% provided conflicts of interest statements. A Krippendorff’s alpha of 0.73 was obtained across all coding. Conclusion: Open data, materials, analysis and replication attempts are currently rare in physical activity behaviour change intervention reports, whereas funding source and conflict of interest declarations are common. Future physical activity research should increase the reproducibility of their methods and results by incorporating more Open Science practices.