Treatment fidelity in the Camden Weight Loss (CAMWEL) intervention assessed from recordings of advisor-participant consultations

Background Variations in the delivery of content and process can alter the effectiveness of complex interventions. This study examined the fidelity of a weight loss intervention (Camden Weight Loss) from recorded consultations by assessing advisors’ delivery of content, use of motivational interviewing approach and therapeutic alliance. Methods A process evaluation was conducted of advisor-participant consultations in a 12-month randomised controlled trial of an intervention for adult volunteers with a body mass index categorised as overweight or obese. A convenience sample of 22 consultations (12% of 191 participants) recorded at the intervention mid-point were available for analysis. Consultations were independently rated by two observers independent of intervention or study delivery, using: a fidelity scale, the Motivational Interviewing Treatment Integrity Scale and the Primary Care Therapy Process Rating Scale. Raters were blind to participants’ responses to the intervention and weight outcomes. Half the participants (N = 11) achieved significant weight loss (≥ 5% of baseline weight). Results A mean of 41% of prescribed content was delivered, with a range covered per session of 8–98%, falling below the 100% content expected per session. Tasks included most frequently were: taking weight and waist measurements (98%), scheduling next appointment (86%), review of general progress (85%) and reviewing weight change (84%). Individual items most frequently addressed were ‘giving encouragement’ and ‘showing appreciation of participant’s efforts’ (95 and 88% respectively). Consultation length (mean 19 min, range 9–30) was shorter than the 30-min allocation. Quantity of content correlated with consultation length (p < 0.01). Advisors’ use of motivational interviewing was rated at ‘beginner proficiency’ for Global Clinician Rating, Reflection to Question Ratio and Percent Open Questions. Therapeutic alliance scores were moderate. Affective aspects were rated highly (e.g. supportive encouragement, involvement and warmth). Conclusions Intervention fidelity varied in both content and process, emphasising the importance of ongoing fidelity checks in a complex intervention. Advisors focused on certain practical aspects of the intervention and providing an encouraging interpersonal climate. This concurs with other research findings, which have revealed the value participants in a weight loss intervention place on an empathic advisor-participant relationship. Clinical trials registration Registered with Clinicaltrials.gov, number NCT00891943, on 1 May 2009.


Background
The increasing impact of obesity on health has caused international alarm [1,2]. It is estimated that 2.8 million people die annually due to overweight or obesity [2]. In the UK, 65% of men, 58% of women and 33% of children aged 10-11 are overweight or obese [3]. Health problems related to obesity are estimated to cost the UK National Health Service £5 billion per year [4].
Guidance recommends multi-component weight management interventions focusing on dietary intake, physical activity and behaviour change, and that behaviour modification addresses: 'problem solving; goal setting; how to carry out a particular task or activity; planning to provide social support or make changes to the social environment; self-monitoring of weight and behaviours that can affect weight; and feedback on performance' [5]. Affective features of weight management interventions are also highlighted, emphasising empathy, support and encouragement, and a respectful and non-judgemental approach.
Guidance reflects the complexity of evidence about weight management and the theoretical basis for behaviour change [6][7][8][9][10][11][12]. Multi-component interventions are superior to single-component interventions and result in greater longer term weight loss than control conditions [7,8]. However, intervention success varies, with variation not accounted for by participant characteristics or programme components (such as length, intensity or face-to-face contact). Long term weight loss remains a challenge [9,12,13].
Behaviour change interventions aim to encourage people to self-manage their weight in the long term. Motivational interviewing aims to support this by identifying and enhancing an individual's own motivation and self-efficacy. The health professional employs an empathic, supportive and collaborative approach, emphasising the individual's autonomy and encouraging the person to explore their own reasons for, and ambivalence about, changing the target behaviour [14].
Whilst motivational interviewing is effective in promoting behaviour change, many health professionals are 'generalists' , using a variety of approaches rather than a single, 'pure' approach. 'Motivational interviewing-style' approaches, which employ some of the elements (such as empathy) without using the full range of techniques, have been investigated [15]. Weight management programmes including either pure or adapted forms of motivational interviewing improve outcomes relative to traditional behaviour change interventions or control conditions [16,17]. However, in primary care consultations with patients who were overweight or obese, low levels of techniques consistent with a motivational interviewing approach were observed, specifically empathy and motivational interviewing 'spirit' [18].
The importance of the quality of the therapeutic relationship on outcomes of behaviour change interventions has also been recognised [19]. Therapeutic alliance includes affective aspects of the professional-patient relationship (such as empathy, rapport and warmth) and instrumental aspects (such as agreement on goals and tasks). Baldwin and colleagues highlighted the impact of therapeutic alliance in weight management outcomes, and noted importance of the professional's contribution to developing this alliance [20].
Weight management interventions require professionals to skilfully select and deliver elements in line with evidence and an individual's needs. The importance of initial training and continuing professional development has been highlighted [5]. Key features to promote fidelity (defined as the degree to which an intervention is delivered as intended) are staff training, supervision and an intervention manual [21]. Failure to implement the intervention as designed can result in a 'Type III error' , where study results do not reflect the effects of the planned intervention [22]. Fidelity includes exposure, adherence to content and quality of delivery [21]. It is commonly assessed by trained observers, either live or from recordings [16,21,23]. For example, one study of fidelity in a behaviour change intervention for diabetes found that staff training improved motivational interviewing spirit [24].
In a 12-month weight loss intervention trial for obese and overweight volunteers, a third of the intervention group achieved clinically significant weight loss (5% or more of their baseline weight) [25]. The present study was designed to examine intervention fidelity, to explore whether differences in intervention delivery may have contributed to variability in intervention group outcome.

Study aim
To investigate weight loss intervention fidelity through assessing the content and process of advisor-participant consultations. Specifically, to establish whether: (i) intervention topics and activities were delivered as intended; (ii) advisors' consultation style was consistent with approaches to support lifestyle behaviour change, in particular, using a motivational interviewing approach and establishing a therapeutic alliance.

Design
This was an independent evaluation examining fidelity of a multi-component weight loss intervention delivered by health advisors to participants with weight categorised as overweight and obese, in the intervention arm of a pragmatic randomised control trial in primary care. This was a descriptive, observational study, conducting a process evaluation using independent, blind ratings of recorded advisor-participant consultations. Fidelity of intervention content (scheduled topics and activities) and process (motivational interviewing and therapeutic alliance) were examined. Recordings were taken from the mid-point of the intervention to: (i) assess the therapeutic relationship that had developed, (ii) reduce the influence of participants' and advisors' awareness of intervention outcome (i.e. final weight change).

Participants
Participants were adults attending the 12-month Camden Weight Loss programme during a two year research period. Consultations were recorded during a five month period. Written consent was obtained from all participants. Out of 191 participants who received the intervention, 104 audio or video-recordings were obtained for 42 participants during the recording period. Including only participants for whom final weight outcomes were available resulted in 34 participants. Of these, recordings from the three mid-intervention sessions were available for 27 participants. Due to problems with sound quality, recordings from five participants were excluded, resulting in a total sample of 22 participants (12% of 191).
The 22 participants were 10 women and 12 men, predominantly White British/White Other (17 participants), with a mean age of 53 years (range 26-80 years) and a mean body mass index at baseline of 32.6 (range 25.2-45.1). At outcome (12 months), 11 had achieved clinically significant weight loss (5% or more of baseline weight) and 11 had not. The 22 participants did not significantly differ from the other 169 trial participants in the intervention arm for: age, baseline weight, waist or body mass index, or final weight loss, but were more likely to be male (12/22 compared to 42/169, Chi 2 (1) = 8.5, p < 0.01) and to complete more sessions (mean 10.9 compared to 7.4, t(158) = 3.5, p < 0.01).

The weight loss intervention
The Camden Weight Loss programme was offered to adults with weight categorised as clinically overweight or obese in primary care practices in a research trial [25]. The trial aimed to develop a locally delivered weight loss intervention, in line with the National Health Service Health Trainers Initiative [26], drawing health advisors from local communities, who are trained to support people in adopting healthier lifestyles by using psychological techniques to promote behaviour change. These techniques include supporting others to: choose a behaviour to change, set 'SMART' goals, plan behaviour change, improve confidence, review behaviour change, and embed behaviour change into their lifestyle [26]. The intervention was devised as a multi-component programme to promote behaviour change in line with National Institute for Health and Care Excellence guidance [27] and based on behaviour change models (Social Cognitive Theory, Goal Setting, Systems Thinking) [28][29][30]. Baseline and final weight were measured by research staff. The results of the randomised controlled trial of 381 participants, which included the 191 participants in the intervention arm, are published elsewhere [25].
Six advisors with a background in health care or exercise were trained to deliver a structured one-to-one intervention. The recordings included five of the advisors: one nurse, two osteopaths and two qualified personal fitness trainers, one of whom also had training in nutrition (CYQ Central YMCA Qualification Level 3 Award). Each participant was allocated to one advisor and were scheduled to attend 14 sessions, lasting 30 min per session, over 12 months in a primary care setting. The session length was intended to enable more in-depth discussion of weight management than is possible in a standard National Health Service primary care consultation (10 min), whilst being delivered to participants in their local practice. Sessions 9, 10 and 11 straddled the intervention mid-point (6 months).
Advisors attended two days of training, including: (i) the intervention design and rationale (ii) effective behaviour change strategies and principles of motivational interviewing (iii)simulated practice in setting weight loss goals, talking about weight and behaviour change and addressing difficult issues.
Advisors were given a detailed manual listing the goals and content of each session, including handouts for participants in some sessions, and a 20-page booklet on Helping People Change Behaviour, including worked examples of techniques for: motivational interviewing, agenda setting, assessing importance and confidence, listening and informing. During the intervention, advisors attended additional group meetings, including further training in motivational interviewing techniques, and met with research staff individually to discuss intervention progress and any issues in intervention delivery.
Each consultation included a review of progress, recording and reviewing pedometer counts, taking weight and waist measurements, reviewing weight loss progress, introducing a new topic, goal setting, making an action plan and confirming the next appointment. The review included discussing the participant's experience and success with the previous session's topic. The intervention schedule is shown in Table 1. Sessions were delivered on a regular schedule with tapering frequency: fortnightly for 12 weeks, 3-weekly to 27 weeks, 4-weekly to 35 weeks and a 12-week interval to the last session. Further details of the intervention are published elsewhere [25]. The topics addressed in sessions 9, 10 and 11 were: positive and negative thinking, responding to situations where you might 'slip up' , social eating, and staying on course in the long term.

Treatment fidelity
Checklists were devised for the three sessions by itemising the session content from the manual, using the same wording for items repeated across sessions. Eight topics were included in every session: (1) reviewing overall progress, (2) reviewing previous topic and handouts, (3) reviewing pedometer counts and physical activity, (4) taking weight and waist measurements, (5) reviewing weight change, (6) presenting new topic, (7) setting goals, making action plans and assigning home activities, (8) setting date of next appointment. Due to the detail in the manual, the initial checklists contained 41, 40, and 52 items respectively for the three sessions. The final checklist for session 9 is shown in Table 2 as an example.
The checklist was piloted using a scoring key of 0 (not done), 1 (partially done) and 2 (completely done) for most of the items (e.g. 'feedback is given on performance'), with some simple items (e.g. 'waist circumference is measured') assessed on a binary scale of 0 (not done) and 1 (done). However, low frequency and brevity of advisor behaviours observed during piloting indicated that measurement was better suited to assessing the presence of behaviours in comparison to expected content, rather than a combination of presence and quality. The scoring key for all items was converted to a binary scale (done/not done). An additional category ('not recorded') noted where items could not be rated due to poor sound quality. A rater crib sheet specified item content, including strategies or examples the advisors had been encouraged to use.

Motivational interviewing
The Motivational Interviewing Treatment Integrity Scale [31] assesses adherence to and competence in using motivational interviewing, with good inter-rater reliability reported [32,33]. Global ratings are made for five dimensions: Evocation, Collaboration, Autonomy/ Support, Direction and Empathy on a 5-point scale, and a summary score: Spirit of Motivational Interviewing. Behaviour counts are made for seven aspects: Giving Information, Closed Questions, Open Questions, Simple Reflections, Complex Reflections, Motivational Interviewing Adherent Behaviours and Motivational Interviewing Non-adherent Behaviours. Further summary scores are also computed.

Therapeutic alliance
The 14-item Alliance scale of the Primary Care Therapy Process Rating Scale [34] assesses the quality of the professional-patient therapeutic bond in psychological interventions conducted in primary care settings. It was designed for research into treatment fidelity and process-outcome relationships and has good internal consistency (Cronbach's alpha 0.88). Items are scored  The appointment for the next session is confirmed on a 7-point scale, with anchors at four points (not at all, somewhat, considerably, extensively).

Rater training
Raters were blind to participants' weight loss outcomes. The raters (LA and GB) practised using consultations not included in the analysis (five for treatment fidelity, 18 for motivational interviewing and 10 for therapeutic alliance) and discussed discrepancies. The raters independently rated consultations in batches of five and reconvened to discuss discrepancies. For motivational interviewing, a third rater (LN) independently rated six consultations and met with the raters to discuss discrepancies. For therapeutic alliance, a third rater (EG) independently rated three consultations and met with raters to discuss discrepancies and provide additional examples of ratings. Raters coded each consultation four times: once for treatment fidelity, twice for motivational interviewing (global ratings followed by behaviour counts) and once for therapeutic alliance.

Treatment fidelity analysis
Inter-rater reliability Overall and specific agreement, for positive and negative agreement, were calculated for each item [35,36]. Overall agreement was 0.82 (i.e. for 770/937 decisions, the raters agreed that the item had been done or not done). Items with low inter-rater reliability (defined as agreement in less than 70% of decisions) or with too many missing (due to issues with sound in the recording) were excluded. The items deleted were: (1) initial items, as advisors did not necessarily start the recording immediately, (2) items that could not be reliably recorded from audio-only recordings, e.g.

Fidelity identified from the recordings
Overall, a mean of 41% of scheduled content was addressed, with session totals of 39, 35 and 49% respectively ( Table 3). The amount of content addressed per participant ranged from 24 to 54% (SD 10%). Content included most frequently was: taking weight and waist measurements (98%), setting date of next appointment (86%), reviewing general progress (85%), and reviewing weight change (84%). The most frequent items were 'giving encouragement' and 'showing appreciation of participant's efforts' (95% in reviewing general progress and 88% in reviewing pedometer counts and physical activity).
Topics with lower or more variable frequency were: reviewing participant's use of the previous session's topic and handouts (8%), reviewing pedometer use, reviewing step counts and physical activity (excluding the item about appreciation of participant's efforts) (19%), setting goals, developing action plans and assigning home activities (24%), and presenting the new topic, which varied from 21% for session 10 (Social Eating) to 79% for session 11 (Staying on Course).

Motivational interviewing Inter-rater reliability
For the global dimensions, using the categories described by Cicchetti [37] the intra-class correlation coefficients (two-way random, testing for consistency) were excellent for Direction, fair for Empathy, and poor for Evocation, Collaboration, Autonomy/Support and Spirit of Motivational Interviewing (Table 4).
For the behaviour counts, the intra-class correlation co-efficients showed excellent reliability for Giving Information, Simple Reflections, Complex Reflections and Motivational Interviewing Adherent, good reliability for Closed Questions, fair reliability for Open Questions and poor reliability for Motivational Interviewing Non-adherent.

Motivational interviewing identified from the recordings
The scale authors suggested that a mean score of 3.5 indicates 'beginning proficiency' and 4.0 indicates 'competency' for the global dimensions [31]. The mean scores for Evocation, Collaboration, Autonomy/Support and Spirit of Motivational Interviewing fell below the threshold for 'beginning proficiency' , and Empathy was at 'beginning proficiency' ( Table 5). The mean score for Direction was high, indicating that advisors maintained focus on the target topic of weight loss.
Mean total questions asked by the advisors was 4.8 (SD 3.0) and mean total reflections was 5.9 (SD 4.5). Summary scores for the behaviour counts fell between the scale authors' suggested scores for 'beginning proficiency' and 'competency' for Reflection to Question ratio and Percent Open Questions, and below the threshold for 'beginning proficiency' for Percent Complex Reflections and Percent MI-Adherent.

Therapeutic alliance
Inter-rater reliability and internal consistency Excellent internal consistency was found (Cronbach's alpha 0.92). Intra-class correlation coefficients (Table 6) showed excellent reliability for two items (warmth and empathy), good reliability for four items (involvement, rapport, client self-discloses thoughts and feelings, and client and therapist agree on the kind of changes to make), fair reliability for five items (supportive encouragement, client expresses emotions, client works actively with therapist's comments, client and therapist share same sense about how to proceed, and client and therapist agree on salient themes), and poor reliability for the remaining three items.
Therapeutic alliance identified from the recordings Mean total score was 4.1 (SD 0.8) indicating that the consultations were being rated around the mid-point, between the anchor points of 'somewhat' and 'considerably'.
Of the 11 items with fair to excellent reliability, items with a mean score above the mid-point were: supportive encouragement (mean 5.0, SD 1.

Relationship of process measures to weight outcome
Using independent t-tests to compare the 11 participants who had lost 5% or more of their baseline weight with the 11 participants who had not, there was no difference between the groups for: (i) consultation length, (ii) percentage of total content covered, (iii) motivational interviewing: Motivational Interviewing Spirit, percentage open questions, percentage complex reflections, percentage Motivational Interviewing Adherent, (iv) therapeutic alliance total score (Table 7).

Discussion
Intervention fidelity should be improved by providing advisors with training, supervision and supporting materials [21]. Nonetheless, observed adherence to intervention content was lower than expected. Paradoxically, having detailed session content may cause a conflict between achieving an intervention that can be consistently delivered and one that can be realistically delivered. The development of the fidelity measure revealed a relatively high number of items for the intended consultation length. However, advisors were not routinely using the full time allocation, with the average duration of the consultations being a third less than the time scheduled. This suggests that health advisors were selective in delivering content. Certain elements were performed consistently, such as reviewing general progress and taking weight and waist measurements, whilst others were  performed inconsistently, such as reviewing participants' use of information from the previous session and goal-setting. The findings indicated that advisors focused more on practical elements and education than on exploring participants' perspectives. The latter is potentially more challenging, despite the availability of time and relationship continuity. Notwithstanding the detailed guide to content, differing levels of skill are required across intervention components. This may highlight a limitation of interventions designed to be delivered by trained advisors rather than by traditionally trained health professionals, in that elements of the intervention requiring more advanced psychological consultation techniques were not delivered, despite the availability of time.
The advisors knew they were being recorded, indeed, they switched on the recording equipment, as is common in UK primary care settings. During the five-month recording period, all consultations were recorded (whether or not at the mid-point of the intervention), to 'normalise' the routine of recording, of which a fifth (22/104) were analysed. The raters reported that the advisors appeared to have a 'routine' for the consultations and that the language used (for example, in beginning the consultation and initiating the 'taking measurements' task) indicated that this was a routine familiar to the participants. This suggests that the aim of capturing a well-developed relationship and consultation routine was achieved by recording at the intervention mid-point. Whilst it cannot be ruled out, there was no evidence to suggest that the advisors were behaving differently whilst being recorded.
Higher inter-rater reliabilities were achieved for 'basic' and specific skills in motivational interviewing (e.g. whether the advisor maintains a focus on the target topic, demonstrates an empathic approach, or asks closed questions). The authors of the scale noted that it performs better for rating 'entry level' than expert therapeutic behaviours, and specifically, for measuring empathy and micro-skills (such as using open and closed questions) rather than advanced skills (such as creating a discrepancy between client values and behaviours or eliciting change talk) [32]. The findings of the present study are consistent with this. It is, however, easier to code a behaviour reliably when it is present, as behaviours which appear to meet the criteria can be scrutinised and any discrepancies between raters discussed.
The findings suggested that the advisors were operating at 'entry level' proficiency in motivational interviewing, which is consistent with the advisors' level of experience and skill. Advisors consistently demonstrated an empathic approach and maintained a focus on the topic of weight loss. This concurs with the results about fidelity, which also found the advisors to be consistently encouraging and supportive. However, more advanced therapeutic skills in motivational interviewing were not observed. This 'layering' of skills in motivational interviewing, with increasing complexity requiring considerable experience and training, is consistent with other research [24].
In terms of therapeutic alliance, higher inter-rater reliability was achieved for affective qualities of the relationship (e.g. involvement, warmth, rapport and empathy) than specific skills (e.g. client works actively with therapist's comments). Consistent with the findings from the other two measures, advisors demonstrated 'entry level' proficiency, achieving higher ratings for aspects of the quality of the interpersonal climate, such as supportive encouragement, involvement, warmth, rapport and empathy.
Overall, the findings demonstrated that certain aspects of the intervention were consistently delivered. Participants' weight was checked, information was provided, and sessions maintained a focus on the target outcome. Furthermore, these tasks were conducted in the context of a warm and supportive advisor-participant relationship. Interviews with participants in the Camden Weight Loss trial reported elsewhere revealed that the most valued aspects of the intervention were the relationship they formed with the advisor, followed by regularity of meetings [38]. The aim of the trial was to examine the feasibility and effectiveness of a weight management programme which was centrally organised but locally delivered in a primary care setting. Other research examining consultations in primary care in which weight management is discussed has demonstrated the importance of training health professionals in weight management interventions [18]. The results of the present study, however, highlight the importance of ongoing training and supervision. Multi-component weight management interventions comprise a spectrum of tasks and skills at varying levels of sophistication, which take time to acquire and develop.
An important determinant of intervention delivery is the congruence between the aims of the intervention and the experience and skill of the provider. One solution may be to use recorded consultations during supervision to provide feedback about fidelity and discuss strategies the advisor might use to achieve the intervention aims. In addition to providing ongoing training and supervision, another solution might be to alter the complexity of the intervention over time, as advisors' experience and skill increase.
This study had several limitations. Consultation recordings were not available for all participants in the study, as they were gathered during a five-month period and participants varied in their start date for the 12-month intervention. Recordings were also subject to the vagaries of the primary care settings, including technical failure. The present sample attended a greater number of sessions compared to others in the intervention group, as those included necessarily continued to at least session 9, and were more likely to be male, although there is no clear explanation for this (there was no gender difference in the number of sessions completed). The small sample size made it difficult to assess the impact of variation in fidelity on intervention outcome (final weight change). Nonetheless, the observed consultations appeared to provide a representative picture of the intervention as delivered in practice.
The health advisors in the study were recruited and trained as recommended by the National Health Service Health Trainers Initiative [26]. However, the study did not examine delivery of the intervention by other types of advisors, such as professionals trained in primary care or psychological interventions. Advisors with different backgrounds and experience may have delivered the intervention differently.

Conclusions
The results of randomised controlled trials and the effectiveness of complex interventions addressing behaviour change are dependent on the fidelity of the intervention delivered. Obesity statistics indicate that weight management interventions will continue to be required for the foreseeable future, emphasising the need for effective intervention delivery. This study has demonstrated that an independent process evaluation can identify the components of a complex intervention which are and are not reliably delivered. As these interventions are complex and layered, advisors delivering such interventions require considerable support, training and ongoing supervision to support those attempting to achieve significant weight loss.

Funding
Camden Primary Care Trust (NHS Camden) funded the intervention from which the data were gathered. The funding source had no role in the design or conduct of the study; collection, management, analysis or interpretation of the data and preparation, review or approval of the manuscript.

Availability of data and materials
The data used during the current study are not publicly available as they consist of audio-and video-recordings of consultations during an intervention from which individual health advisors and/or volunteers receiving the weight loss intervention might be identifiable. Confidentiality of the data was stipulated in the study protocol, as approved by the local research ethics committee.
Authors' contributions LN, EG, KN and NT conceived and designed the study, KN and NT contributed to acquisition of data, LN, EG, LA and GB contributed to analysis of the data; LN drafted the article, all authors contributed to the interpretation of the data, preparation of the manuscript and have read and approved the manuscript.
Ethics approval and consent to participate The project was approved by the London School of Hygiene and Tropical Medicine Observational/Interventions Research Ethics Committee, reference number 6356. Written consent was obtained from all participants.

Consent for publication
Not applicable.

Competing interests
The authors declare they have no competing interests.

Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.