The use and reporting of primary and secondary outcome measures in trials evaluating childhood obesity treatment interventions
© Bryant et al.; licensee BioMed Central Ltd. 2014
Received: 7 April 2014
Accepted: 7 November 2014
Published: 2 December 2014
Existing systematic reviews aimed at comparing effectiveness of childhood obesity treatment interventions are limited by a lack of quality in the conduct and reporting of trials in this area. This study aimed to identify the number and types of primary and secondary outcomes used within trials of childhood obesity treatments and to determine the degree to which these trials correctly report their use of outcome measures. A systematic literature review was conducted to identify phase II (i.e. pilot and feasibility studies) and phase III (e.g. determining effectiveness) trials of childhood obesity treatments across 11 databases. Data were extracted from eligible manuscripts pertaining to the number and type of outcome measures used, in addition to details of citations provided for these measures.
145 different outcome measures were reported to be used within 200 identified eligible trial manuscripts. Citations were provided to indicate the provenance for 417 measures, but only 13% of these were correctly linked to papers describing the development and/or evaluation of measures.
This study identified inconsistencies and inaccuracies in the use and reporting of outcome measures used by eligible trials. Researchers in this area are urged to consider guidelines such as CONSORT and the National Obesity Observatory Standard Evaluation Framework in the design and reporting of future trials.
KeywordsReporting Obesity Outcome Measures Consort
Many interventions to treat obesity are aimed at children, but there remains a lack of high quality evidence on effective childhood obesity interventions in the literature . Existing systematic reviews aimed at comparing effectiveness of intervention programmes (particularly those conducting meta-analysis) are hampered by a lack of quality in the conduct and reporting of trials in this area. There has been some attenuation in the rising rates of childhood obesity in recent years, and it is therefore probable that many attempts to treat obesity in children have been of some success . The problem, therefore, may lie in the methods used to evaluate and report interventions.
The degree to which weight management leads to improvements in a child’s health is reflected by measuring change in outcomes in clinical trials. Outcomes either directly measure a trial primary outcome, (i.e. primary outcome of weight loss); or assess proximal/secondary outcomes (e.g. change in diet) that impact on the primary outcome. In the design phase of a trial, choosing the appropriate outcomes is essential. Use of inappropriate outcomes will result in data that are inaccurate or biased and that do not indicate the effectiveness of an intervention. Moreover, collection of data using poorly chosen outcomes is a waste of resources, both for the researchers and participants involved in the trial . Inappropriate selection of outcomes in childhood obesity research is likely due to the uncertainty about which outcome domains are most relevant to children and their families . Furthermore, there has been no guidance on which outcome measures are valid and reliable for use.
Guidance tools are available to facilitate the design of high quality research, including the Medical Research Council guidance for the evaluation of complex interventions and, more specifically, the National Obesity Observatory Standard Evaluation Framework (NOO SEF) for childhood obesity evaluation (http://www.noo.org.uk/core/SEF) . The latter (commissioned by the UK Department of Health) was produced with guidance from prominent obesity researchers to aid clinicians in their evaluation of childhood obesity programmes. It now stands as a grounded tool to enable consistency with research design and includes items for guidance with data collection and dissemination (e.g. essential information to describe the intervention, participant demographics, essential and desirable outcome measures (e.g. adiposity), process measurement; and analysis and interpretation) . The primary audience for the NOO SEF was those evaluating public health obesity programmes. However, much of the advice is of relevance to researchers conducting trial evaluations. Recommended outcomes with the NOO SEF are listed and described as ‘essential’ or ‘desirable’. This resembles the output of a core outcome set, though the inclusion of each outcome has not been based on formal consensus methodologies, such as those described by ‘COMET’ (Core Outcome Measures in Effectiveness Trials). Further, many of the recommendations are not ‘outcomes’ but include guidance on the collection of demographic information, process evaluation, and analysis and dissemination. Core outcome sets are a minimum set of outcomes that should be measured and reported within trials or other forms of research for a specific condition (http://www.comet-initiative.org/). The use of core outcome sets permits comparisons between trials that are agreed upon by experts within each disease area. As present, there is not a core outcome set for obesity research; partly because of the complexity and variability in intervention targets (requiring potentially different outcomes). The NOO SEF therefore stands as a guide, rather than a minimum set of outcomes. Importantly, the NOO SEF does not provide advice or details of outcome measures that should be used within each outcome domain that it recommends.
This paper aimed to identify primary and secondary outcomes used within trials of childhood obesity treatments and to determine the degree to which these trials correctly report their use of outcome measures. To our knowledge, there have been no previous publications describing this methodology to highlight issues in the use and reporting of outcome measures within childhood obesity trials in order to make recommendations for optimising the design and dissemination of future trials in this area.
A systematic literature review was conducted as part of a study aiming to develop a framework of recommended outcome measures for use in trials of obesity treatments (HTA 07/127/09 ,). Included manuscripts were those describing randomised controlled trials, pilot and feasibility studies of childhood obesity treatment evaluation studies (with the intent of identifying outcome measures (and corresponding citations) already used in trials). The search was conducted from August 2011 to October 2011 in 11 databases including: MEDLINE, MEDLINE in process, EMBASE, PsycINFO, HMIC, AMED, Global Health, Maternity and Infant Care (all Ovid); Cinahl (EbscoHost); Science Citation Index (WoS); and the Cochrane Library (Wiley) from the date of inception, with no language restrictions.
Search concepts included obesity terms AND child terms AND evaluative studies terms. The evaluative studies search consisted of focused ‘textword’ and subject heading searches (MeSH: exp clinical trial/, or evaluation studies/ or meta-analysis/ or validation studies/, Randomised Controlled Trials as Topic/) (See Additional file 1). Child obesity terms identified in the Cochrane review  were also incorporated where appropriate. Unpublished literature was also obtained by searching a range of relevant databases including Inside Conferences, Systems for Information in Grey Literature (SIGLE), Web of Science Conference Proceedings Citation Index- Science (Thomson) and ClinicalTrials.gov. The same eligibility criteria were applied for each of these additional sources.
Only manuscripts describing primary research of obesity treatment intervention evaluation studies were eligible, including: RCTs, pilot studies and feasibility studies (with the intention of carrying out RCT). Within trials of obesity treatment interventions, phase II trials are usually either pilot or feasibility studies which evaluate the feasibility of the intervention process and/or the trial procedures. Feasibility studies test out individual components of a future trial and do not have to be randomised (though often are). Pilot studies are a small version of a phase III trial and can be internal (i.e. first part of a phase III, in which case the data may be used for the phase III) or external (i.e. conducted separate to the phase III). Phase III trials are studies that tell us whether or not interventions are efficacious (e.g. blinded drug interventions) or effective (e.g. lifestyle interventions) . Inclusion of both treatment and prevention interventions for this study was not considered to be feasible following a scoping exercise of the number of potential papers. Prevention interventions also have a wider scope and would include more environmental and political outcomes, which we believe warranted a separate framework. Although there is scope to use the guidance when choosing outcome measures for prevention trials, consideration for their recommendation (inclusion to the framework) has been based on their use in treatment trials (and has considered their historical use in previous treatment intervention evaluations).
Quality assessment was not an inclusion criterion, as the aim was not to ascertain evidence on intervention effectiveness. The included samples were any childhood study population (equal to or less than 18 years at baseline). Studies with special populations (i.e. those with a cause of obesity such as Prader Willi) were included. In terms of intervention, we included any type of intervention to treat obesity, including drug and surgery interventions. These are defined according to categories of strategies set a Cochrane Review of childhood obesity treatment trials  as ‘lifestyle’ (dietary, physical activity and/or behavioural therapy interventions), Drug (e.g. orlistat, metformin, sibutramine, rimonabant) and surgical interventions.
All studies had to have obesity reduction as a primary outcome and a list of eligible measures was generated to comply with this criteria. Examples included: Body mass index (BMI), Waist circumference, Skin fold thickness, Dual energy X-ray absorptiometry (DXA), Bioelectrical Impedance (BIA) etc. Included secondary outcomes were: diet, eating behaviour, physical activity, sedentary behaviour/time, fitness, psychological well-being, health related quality of life, environment and physiology. Similar to primary outcome measures, another list of all potential types of outcome measures was generated for each outcome domain.
Excluded manuscripts were those: describing studies without a primary outcome of obesity reduction such as weight loss, BMI or adiposity reduction; with a secondary aim of obesity reduction (e.g. those with a primary aim to control diabetes); providing details of outcome measures for adults (or childhood outcomes are not reported separately); obesity prevention studies (or designs other than those listed in the inclusion criteria, including letters, editorials, commentaries, dissertation, books, erratum, notes, introductory, conference proceedings, meeting abstracts and case reports); general reviews or guidelines (unless specifically about the evaluation of childhood obesity treatment interventions (e.g. Luttikhuis et al.  in which citations were searched); papers without sufficient information to determine eligibility (where author cannot provide missing information); those not specifically focusing on all obese subjects for intervention; maintenance studies that were retrospective to studies previously carried out; and phase 1 testing for drug trials (i.e. safety, tolerance, effect).
A description of papers fulfilling the eligibility criteria was entered onto a trial specific pre-prepared standardised data-extraction form. Three versions of paper based forms were initially piloted until a final form was created and incorporated into the ‘Bristol Online Survey’ (BOS http://www.survey.bris.ac.uk). This enabled relocation of all data into an excel database. In addition to information on the trial (sample characteristics, type of intervention, study design), detailed data were gathered on the use of outcome measures, including the outcome domain (e.g. diet), the type of outcome measure (e.g. recall questionnaire) and the provenance of each measure (i.e. linked citations). All citations linked to outcome measures were then individually sourced.
Across all 200 trial manuscripts, there were 417 citations linked to 145 different primary and secondary outcome measures with the aim to report the origins (including development and/or evaluation) of these measures. However, only 56 (13%) of these citations referred to manuscripts that actually described the development and/or evaluation of the outcome measures, with correct citations reported in 46 out of 200 trial manuscripts. This discrepancy in numbers was generated by trials papers that correctly provided for some citations, but not necessary for all of the included measures. Incorrect citations were linked to the majority of outcome measures; most commonly linking to a previous study that had used the same measure.
This study identified inconsistencies and inaccuracies in the use and reporting of outcome measures used by eligible trials. It is likely that inconsistencies in the use of outcome measures relates to the specific requirements of each study (e.g. use of measures that are specific to age and ethnicity of the sample). However, even taking this into account, there were a great number of outcome measures used which had similar characteristics. In terms of reporting, only 13% of citations for outcome measures used were correctly linked to a manuscript describing the development and/or evaluation of that measure. A majority of inaccurate citations were linked to previous trials also using the measure. The variation in the use of appropriate outcome measures for the reliable and valid assessment of childhood obesity interventions means that comparisons between interventions are consequently difficult, partly because of a shortage of validated outcome measures available, but also because the selected outcome measures differ between studies. Consequently, it is a challenge to identify which interventions are the most effective. Such a lack of consistency and inadequacy impedes the progress of childhood obesity research.
When designing trials, there is always a danger of including outcomes that are not relevant or necessary to the research, particularly in large teams with varied interests. This does not appear to have happened in the 200 included trials here, with an average of 2 secondary outcomes reported per trial. However, this is likely to be an underestimation of the actual number of outcomes measured, as we only recorded outcomes within 9 outcome domains. Additionally, some trials included multiple outcome measures for the same outcome. Lastly, there is potential for some reporting bias here, given the other observed inaccuracies in reporting, with some outcomes not reported (especially those generating poor or null findings).
However, discrepancies also arise owing to the large availability of differing measures, and combined witha lack of agreement of which measures are most valid, reliable and feasible for use. This study was conducted as part of a study aiming to develop a framework of recommended outcome measures (the Childhood obesity Outcomes Review (CoOR Framework) for use in trials of obesity treatments aimed at reducing such inconsistencies (HTA 09/127/07). This framework includes 52 outcome measures across 10 outcome domains (2 primary and 50 secondary), which are recommended for use in childhood obesity treatment evaluations ,. Manuscripts identified here were supplemented by additional manuscripts describing the development and/or evaluation of outcome measures from a specific search of outcome measures and resulted in the combined identification of 379 eligible manuscripts; describing 180 outcome measures. These outcome measures were appraised for quality using international guidelines ,, followed by expert agreement to enable the production of a framework of recommended 2 primary and 50 validated secondary outcome measures within 10 outcome domains for use in childhood obesity treatment evaluations (HTA 09/127/07 ,).
When choosing measures, researchers should first consider which (if any) secondary outcome domains are most closely aligned to the targets of the intervention under investigation; including, those that are expected to change, those that are expected to mediate this change (if appropriate); and any that may indicate an adverse event (if appropriate). Any selected measure needs to be aligned to the intervention targets, be developed for use in a similar population and be feasible to implement. In deciding the similarity between populations, validation of a measure is only really relevant to the population that it was evaluated in and researchers should make informed decisions whether the characteristics of their populations are sufficiently close to the population in which the tool was developed.
Inaccuracies in reporting impact substantially on the assessment of the quality, and therefore, the efficacy/effectiveness of trials through use of a tool in inappropropriate populations or the use of tools with inadequate evidence of validation. To reduce such errors, the CONSORT (Consolidated Standards of Reporting Trials) statement was published in 1996 with the aim of alleviating problems of inadequate reporting . This statement compromises a 25-item checklist of which, Item 6a states that ‘All outcome measures, whether primary or secondary, should be identified and completely defined’. Additionally, this states that ‘Authors should indicate the provenance and properties of scales’. Thirty eight of the 200 (19%) eligible trial manuscripts were published before the release of CONSORT. Of these, only 1 provided at least 1 correct citation for included outcome measures. Of the 154 trials providing incorrect citations for their outcomes, 37 (25%) were published before 1996. This may demonstrate an important impact of the CONSORT statement, though this study was not set up to assess the impact of CONSORT and differences in reporting may relate to other factors. Importantly, 117 of the trial manuscripts which had incorrect citations were published after 1996, highlighting continued issues with reporting of trials evaluating childhood obesity treatment interventions.
It is imperative that researchers choose appropriate and valid outcomes and measures in their evaluation of childhood obesity programmes. This will support the robust development of evidence-base in this area and will enable future assessment of the most effective approaches through meta-analysis. When selecting outcomes, researchers should utilise tools such as the Standard Evaluation Framework  and the CoOR Framework , and consideration should carefully balance the scientific need of measurement against practicalities of assessment. Importantly, the selection of outcomes should not be dictated by what is deemed to be interesting, but rather by an a-prior plan of investigation. Once conducted, it is then essential that optimal impact is gained through correct reporting standards using guidance from the CONSORT statement . This will facilitate improved appraisal of trials, enable clear comparisons between them and will reduce presentation of biased estimates of the benefits of interventions under investigation . In addition to aiding appraisal of trials, the correct use and reporting of data from outcome measures will reduce waste related to the collection of poor data, both for the researchers and participants involved in the trials. Over-use of outcome measures without a pre-determined plan of evaluation is unethical and wasteful. Use of incorrect outcomes is similarly wasteful and importantly, does not answer the research question. Presentation of results from inappropriate outcomes is therefore misleading and biases the evidence base. Rigour in outcome selection and reporting is essential in effectively moving forward obesity research which evaluates interventions in order to produce a truly evidence-base of what is and is not effective.
Childhood obesity Outcomes Review
Consolidated Standards of Reporting Trials
Randomised Controlled Trial
This work was funded by an NIHR HTA grant (09/127/07). This paper presents independent research commissioned by the National Institute for Health Research (NIHR). The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health. No funding bodies had any role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
- Oude Luttikhuis H, Baur L, Jansen H, Shrewsbury VA, O'Malley C, Stolk RP, Summerbell CD: Interventions for treating obesity in children.Cochrane Database Syst Rev 2009, Issue 1: Art No:CD001872. doi:10.1002/14651858.CD001872.pub2.,
- Health Survey for England 2010 TNIC, Lifestyles Statistics: Statistics on obesity, physical activity and diet. The NHS Information Centre for Health and Social Care; 2012.
- Holloway RG, Dick AW: Clinical trial end points: on the road to nowhere?. Neurology. 2002, 58: 679-686. 10.1212/WNL.58.5.679.View ArticlePubMedGoogle Scholar
- Sinha I, Jones L, Smyth RL, Williamson PR: A systematic review of studies that Aim to determine which outcomes to measure in clinical trials in children. PLoS Med. 2008, 5: e96-10.1371/journal.pmed.0050096.View ArticlePubMedPubMed CentralGoogle Scholar
- Roberts K, Cavill N, Rutter H: Standard evaluation framework for weight management interventions.National Obesity Observatory 2009, ., [http://www.noo.org.uk/core/frameworks]
- Bryant M, Ashton LJ, Nixon J, Wright J, Jebb SA, Katharine R, Brown J: Framework of outcome measures recommended for use in the evaluation of childhood obesity treatment interventions: The CoOR framework.Pediatric Obesity 2014. 10.1111/j.2047-6310.2014.220.x.,
- Bryant M, Ashton LJ, Brown J, Wright J, Jebb SA, Roberts K, Nixon J: Systematic review to identify and appraise outcome measures used to evaluate childhood obesity treatment interventions: evidence of purpose, application, validity, reliability and sensitivity. HTA monograph Series. 2014, 18 (51): 2014-doi:10.3310/hta18510Google Scholar
- Oude Luttikhuis H, Baur L, Jansen H, Shrewsbury V, O’Malley C, Stolk R, Summerbell C: Interventions for treating obesity in children (Cochrane Review). 2009, Wiley and Sons Ltd, Chichester, UKGoogle Scholar
- Arain M, Campbell M, Cooper C, Lancaster G: What is a pilot or feasibility study? A review of current practice and editorial policy. BMC Med Res Meth. 2010, 10: 67-10.1186/1471-2288-10-67.View ArticleGoogle Scholar
- Gropper SS, Acosta PB: The therapeutic effect of fiber in treating obesity. J Am Coll Nutr. 1987, 6: 533-535. 10.1080/07315724.1987.10720213.View ArticlePubMedGoogle Scholar
- McCallum Z, Wake M, Gerner B, Baur LA, Gibbons K, Gold L, Gunn J, Harris C, Naughton G, Riess C, Sanci L, Sheehan J, Ukoumunne OC, Waters E: Outcome data from the LEAP (Live, Eat and Play) trial: a randomized controlled trial of a primary care intervention for childhood overweight/mild obesity. Int J Obes (Lond). 2007, 31: 630-636.Google Scholar
- US Department of Health and Human Services FDA: Guidance for industry: patient-reported outcome measures: use in medical product development to support labeling claims.Health Qual Life Outcomes 2006, 4:. doi:10.1186/1477-7525-1184-1179.,
- Assessing health status and quality of life instruments: attributes and review criteria. Qual Life Res. 2002, 11: 193-205. 10.1023/A:1015291021312.
- Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne D, Gotzsche PC, Lang T, Consort G: The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Intern Med. 2001, 134: 663-694. 10.7326/0003-4819-134-8-200104170-00012.View ArticlePubMedGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.