- Research article
- Open access
- Published:
Can we achieve better trial recruitment by presenting patient information through multimedia? Meta-analysis of ‘studies within a trial’ (SWATs)
BMC Medicine volume 21, Article number: 425 (2023)
Abstract
Background
People need high-quality information to make decisions about research participation. Providing information in written format alone is conventional but may not be the most effective and acceptable approach. We developed a structure for the presentation of information using multimedia which included generic and trial-specific content. Our aim was to embed ‘Studies Within A Trial’ (SWATs) across multiple ongoing trials to test whether multimedia presentation of patient information led to better rates of recruitment.
Methods
Five trials included a SWAT and randomised their participants to receive a multimedia presentation alongside standard information, or standard written information alone. We collected data on trial recruitment, acceptance and retention and analysed the pooled results using random effects meta-analysis, with the primary outcome defined as the proportion of participants randomised following an invitation to take part.
Results
Five SWATs provided data on the primary outcome of proportion of participants randomised. Multimedia alongside written information results in little or no difference in recruitment rates (pooled odds ratio = 0.96, 95% CI: 0.79 to 1.17, p-value = 0.671, I2 = 0%). There was no effect on any other outcomes.
Conclusions
Multimedia alongside written information did not improve trial recruitment rates.
Trial registration
ISRCTN71952900, ISRCTN 06710391, ISRCTN 17160087, ISRCTN05926847, ISRCTN62869767.
Background
Trials remain critical to evidence-based practice, but recruitment remains a significant challenge worldwide [1, 2]. Despite these challenges, the evidence base concerning effective recruitment strategies is weak, with the Cochrane review in this area reporting only 3 strategies with ‘high certainty evidence’ [3].
To rapidly develop the evidence base, a model has developed of testing promising recruitment strategies by embedding randomised tests in multiple ongoing trials (so-called co-ordinated ‘Studies Within a Trial’ or SWAT). This model has been used by the START [4], TRECA [5] and PROMETHEUS [6] research programmes and has contributed to major growth in the evidence base for recruitment and retention. One of the innovations tested by both the START and TRECA programmes was the potential for multimedia to provide improved information to trial participants and potentially enhance the likelihood that they would be successfully recruited to a trial.
The potential role of multimedia
Written information in trials has been criticised for length and complexity and a lack of clear structure to help patients find the information they need. Changes based on user testing and information design can produce information sheets that are easier for patients to understand [7,8,9], although our previous programme of co-ordinated SWATs found that such changes did not lead to improvements in recruitment [10].
Provision of audio-visual information about trials may be another way to improve the delivery of information and enhance patient decision-making. Previous studies suggest that audio-visual presentation leads to a small increase in patient understanding, but may have less effect on recruitment [11]. Multimedia information is defined by its use of more than one format, and in terms of patient information has generally entailed the digital presentation of a combination of written text, recorded speech, pictograms and video (including scenario portrayal and animations). It can increase levels of user attention and engagement, not only through the choice of format, which allows users to ‘personalise’ or tailor the information, but also through dual channel stimulation and efficient cognitive demands [12]. It may better meet the needs of an audience increasingly accustomed to obtaining information digitally. There is some evidence that this can increase comprehension within the research consent process [13,14,15].
Reviews of the impact of multimedia interventions on research participation have explored a variety of outcomes, including knowledge and understanding, recall, willingness to participate, perceptions of the value of research and decision-making outcomes. Only a small number of studies explored the effects of multimedia materials in the Cochrane review on improving recruitment to trials [16] and the overall conclusion was of uncertainty concerning the effects. Given the limited evidence base, further research through co-ordinated SWATs is clearly warranted. Testing the same strategy across multiple trials provides the opportunity to produce both a more precise estimate of effect and some exploration of whether effects vary by trial type. In this paper, we synthesise SWATs of multimedia use to test their effects on recruitment.
Methods
START was a programme of methodology research funded by the Medical Research Council, which aimed to (a) develop methods for design, delivery and reporting of SWATs [17] and (b) to deliver two sets of co-ordinated SWATs testing two recruitment interventions across multiple trials. The methods of the START programme have been published [4], followed by the first co-ordinated set of SWATs testing the effects of written information sheets optimised through user testing [10]. The current paper represents the second set of SWATs.
To recruit studies for each set of SWATs, we contacted chief investigators through the National Institute of Health and Care Research (NIHR) Health Technology Assessment and Efficacy and Mechanism Evaluation programmes or the Primary Care Research Network, selecting trials with at least 800 participants to be approached for recruitment where the design was amenable to the multimedia recruitment model. We aimed to recruit 6 host trials in which to embed our multimedia intervention.
Development of the multimedia intervention
Multimedia content was generated by team members, informed by a review of factors identified by patients as determinants of decisions about trial participation [18], as well as input from patient and public involvement (PPI) contributors and qualitative experts on patient health experiences.
Content included study-specific information (e.g. study purpose, risks) and generic information (e.g. confidentiality). PPI contributors and qualitative experts developed study-specific components involving bespoke themes such as investigator details and benefits of participation. Generic information components included information on informed consent, randomisation and confidentiality. Existing video clips of patients discussing their experiences of participation were edited for length and carefully matched to these components. Additional file 1 provides an example of the range of material presented, with text and video material in separate tabs for the particular trial (for example, information about this study’, ‘what will happen during the study’) and trials in general (‘why get involved’, ‘leaving a study’). The material was designed to provide a much more flexible set of options for patients in terms of how much information they accessed, and in what order, as well as being designed to be more accessible and engaging. The multimedia intervention was developed by a commercial company for use on a range of platforms including desktops and smartphones.
Access to the multimedia resource was provided within the patient information sheet, with a URL link and QR code to assist with easy access (see Additional file 2 for the presentation of the resource to patients). Although we randomised participants to access the multimedia, it was entirely the choice of the participant whether they actually engaged with the multimedia information (alongside the written information) as part of their decision-making process about the trial.
Methods of the SWAT
In each SWAT participants being approached to take part were randomised to receive the multimedia alongside written information or standard written information alone. Individual randomisation was used where possible to maximise power and precision and minimise selection bias, but we adopted cluster randomisation where preferred by the host trial for logistical reasons.
Outcome measures
The primary outcome was recruitment, defined as the proportion of participants recruited and randomised to a host trial following an invitation to take part. The denominator for the outcome was the total number of potentially eligible participants offered entry to the trial. Depending on the particular trial, this would include a mix of eligible and ineligible patients according to the formal inclusion and exclusion criteria.
Secondary outcomes were:
-
1.
Acceptance, defined as the proportion of potentially eligible participants who express interest in participating (i.e. posting a reply or attending a recruitment appointment). We anticipated that in some SWATs, the number of participants recruited to the host trial could be different from numbers of participants responding positively, due to eligibility criteria used in the host trial.
-
2.
Retention, defined as the proportion retained at primary outcome measurement endpoint of the host trial.
Ethical approval
START was approved by the National Research Ethics Service (NRES) Committee, Yorkshire and the Humber – South Yorkshire (Ref: 11/YH/0271) on the 5 August 2011. Each individual host trial had its own ethical agreement and registration.
Data analysis
Analyses of recruitment were conducted according to a statistical analysis plan. Outcomes were first described separately by study arm and then compared using logistic regression to estimate the between-group odds ratio and corresponding 95% confidence interval. The data from each SWAT were meta-analysed using the Stata metan command (Stata version 14.2) using random effects models based on likely clinical and methodological heterogeneity. Statistical inconsistency was quantified using the I2 statistic. In the meta-analysis, we used a two-stage strategy where each individual SWAT was analysed using the appropriate analysis methods (i.e. taking into account whether it was individually or cluster randomised) to generate trial-level summary statistics (e.g. odds ratios) first, and then the results from individual SWATs were combined across trials using the Stata metan command (Stata version 14.2).
We performed pre-specified subgroup analyses investigating differences between studies based on underlying recruitment rates (low defined as a recruitment rate of 5% or below in control group vs. higher rates). We hypothesised that when the baseline recruitment rate is low, the increase in the absolute recruitment rate associated with a recruitment intervention may be higher. A second planned analysis comparing patients with a known diagnosis versus participants ‘at risk’ was not conducted as it proved difficult to assign trials to the categories reliably.
Results
We originally recruited 6 trials for the SWATs. Only one trial has reported the individual SWAT evaluation [19]. Table 1 describes the characteristics of the host trials and the SWATs. One host trial was only able to report accurate data on 11/37 sites randomized [20] and was excluded from all the analyses (available data by arm are reported in Additional file 3). All host trials were individually randomised, but 2 SWATs used cluster randomisation (general practices, endoscopy units or week of recruitment) because this was operationally easier. One host trial included the same SWAT in two separate groups of practices [21], one group allocating patients on the basis of first contact letters and a second group allocating patients on reminder letters after they had initially been contacted.
Five host trials provided data on recruitment [21,22,23,24]. Access to multimedia resulted in little or no difference in recruitment rates (pooled odds ratio = 0.96, 95% CI: 0.79 to 1.17 p-value = 0.671, I2 = 0%) (Table 2 and Fig. 1).
Four host trials provided data on participant acceptance rates [21,22,23]. Access to multimedia resulted in little or no difference in the likelihood of responding positively to the invitation compared to participants receiving standard information (pooled odds ratio 0.98, 95% CI: 0.85 to 1.13, p value = 0.778, I2 = 0%) (Fig. 2 and Table 3).
Three SWATs provided data on retention [22,23,24]. Access to multimedia resulted in little or no difference in retention compared to participants receiving standard information (pooled odds ratio 1.07, 95% CI: 0.71 to 1.62, p value = 0.737, I2 = 44.7%) (Fig. 3 and Table 4).
There was no marked difference in intervention effects by baseline rate (odds ratio 0.94 95% CI 0.65 to 1.35 in low baseline trials compared with 0.99 95% CI 0.73 to 1.33 in high baseline trials).
Discussion
Summary
We tested the effects of access to multimedia information on trial recruitment and retention. In a number of SWAT evaluations that run through a diverse group of host trials, the intervention did not improve acceptance, recruitment or retention rates among participants.
Strengths and limitations of the study
This programme of coordinated SWATs was one of the first to be initiated, demonstrating the broad feasibility of this strategy as a model for the more rapid development of an evidence base.
As with most SWATs, there was no formal sample size calculation for individual trials. The host trials undertaking the SWATs were self-selected and therefore the studies on which the programme was run represent a relatively specific (though diverse) group of study contexts, with most patients in the older age groups. It is possible that the variation in those contexts was sufficient to give the recruitment strategy a fair test across multiple designs and populations, and there was limited evidence of significant variation in effect. Even the pooled analysis of data from six trials left some imprecision in the estimate of effect. We have provided broad details of the patient populations sought for each of the trials, but we do not have detailed information on the demographics of those who took part in the SWATs. It is important to note that the participants in the SWATs are not the same as those taking part in the trials, as many people participate in the SWATs without entering the trial. Collecting such information is complicated and potentially burdensome for trial teams. One study was unable to provide the necessary data for our main analysis, which reduced the sample size available. In our experience, problems with delivering SWATs are fairly rare but these difficulties do highlight that SWATs can stretch the resources of already busy trial teams.
Although we planned to assess the use of the multimedia intervention, including its various elements, to provide better context to our outcome data, an error in the web-hosting software meant no accurate data on use were available. The SWATs only provided patients with a link to the multimedia resource and did not actively encourage use. It is unclear whether the intervention failed because it was not accessed, or because access to the intervention had limited impact on patient decision-making. We were unable to assess whether uptake and engagement varied across trials, or between different patient groups in individual trials.
The study was conducted pre-pandemic and the increase in the use of remote tools (including in the delivery of trials) may impact the future effectiveness of multimedia presentations in the context of trial recruitment. The creation and dissemination of the evidence was far from rapid, given recruitment began in 2012. This reflects a number of issues, including the fact that some SWATs extended beyond the funded START programme itself (hampering the completion of the meta-analysis). Some individual SWATs took a significant amount of time to complete recruitment or provide recruitment and retention data. Development of SWAT processes since that time has highlighted the need for greater efficiency, permitting faster publication of individual studies and ‘living’ meta-analyses at the level of a recruitment or retention strategy to better inform the trials community.
The participating trials were led by experienced investigators and teams, so the standard information sheets may have already been well designed, leaving less scope for improvement through intervention. To simplify ethical approvals, we compared our intervention plus standard information with standard information alone, but this may have reduced the impact of the multimedia compared to a comparison of multimedia versus standard information. Further developments in technology and media may mean that future iterations of these types of interventions could include more features and greater interactivity which might enhance effects (albeit at increased cost). All the host trials were done in the UK, making it unclear how applicable this evidence is to other countries.
Study results in the context of the wider literature
We report here a linked series of pre-planned and co-ordinated SWATs testing the same recruitment intervention, rather than a retrospective systematic review of all relevant studies using this strategy. The studies reported here will eventually be integrated into the ongoing Cochrane review on strategies to improve trial recruitment [3], alongside similar data from studies outside the START programme. It is possible that access to multimedia has positive benefits on patient understanding, but that does not translate to improved recruitment. Assessing these sorts of impacts through a SWAT is difficult and qualitative research or process data may be required to explore such effects. It is also possible that multimedia and non-written information may be more effective for recruitment of some specific populations, for example some ethnic groups [25].
Most of the studies included in the current meta-analysis were restricted to adults. A linked study has explored the impact of multimedia on recruitment in younger populations. Data from three SWATs found that participants allocated to multimedia were more likely to be recruited to the host trial than those allocated to written information alone (OR 1.54; 95% CI 1.05, 2.28; p = 0.03), although multimedia did not show any impact on measures of decision-making, and the combination of multimedia and written information showed no comparative advantage [26]. Decisions about further SWAT evaluations of this technology can be based on published guidance [27], combining the results reported here, those in the TRECA study, and additional studies of this technology identified by the forthcoming Cochrane review update.
Conclusions
A co-ordinated programme of SWATs among multiple trials found little evidence that multimedia information alongside standard information had an impact on recruitment or other outcomes.
Availability of data and materials
The summary meta-analysis data is available from the corresponding author.
Abbreviations
- MRC:
-
Medical Research Council
- NRES:
-
National Research Ethics Service
- PIS(s):
-
Participant information sheet(s)
- REC:
-
Research Ethics Committee
- START:
-
Systematic Techniques for Assisting Recruitment to Trials
- SWAT:
-
Study-within-a-trial
References
Sully B, Julious S, Nicholl J. A reinvestigation of recruitment to randomised, controlled, multicenter trials: a review of trials funded by two UK funding agencies. Trials. 2013;14:166.
McDonald A, Knight R, Campbell M, Entwistle V, Grant A, Cook J, Elbourne D, Francis D, Garcia J, Roberts I, et al. What influences recruitment to randomised controlled trials? A review of trials funded by two UK funding agencies. Trials. 2006;7:9.
Treweek S, Lockhart P, PitKethly M, Cook J, Kjeldstrøm M, Johansen M, Taskila T, Sullivan F, Wilson S, Jackson C, et al. Methods to improve recruitment to randomised controlled trials: Cochrane systematic review and meta-analysis. BMJ Open. 2013;3: e002360.
Rick J, Graffy J, Knapp P, Small N, Collier D, Eldridge S, Kennedy A, Salisbury C, Treweek S, Torgerson D, et al. Systematic Techniques for Assisting Recruitment to Trials (START): study protocol and preliminary findings on a platform for nesting studies of recruitment interventions across multiple trials. Trials. 2014;15:407.
Martin-Kerry J, Bower P, Young B, Graffy J, Sheridan R, Watt I, Baines P, Stones C, Preston J, Higgins S, et al. Developing and evaluating multimedia information resources to improve engagement of children, adolescents, and their parents with trials (TRECA study): Study protocol for a series of linked randomised controlled trials. Trials. 2017;18(1):265.
Arundel CE, Clark L, Coleman E, Doherty L, Parker A, Torgerson DJ. Challenges and solutions to the implementation of studies within a trial: The experiences of the PROMETHEUS programme. Res Methods Med Health Sci. 2022;4(1):16–23.
Knapp P, Raynor D, Silcock J, Parkinson B. Performance-based readability testing of participant information for a Phase 3 IVF trial. Trials. 2009;10:79.
Knapp P, Raynor D, Silcock J, Parkinson B. Performance-based readability testing of participant materials for a phase I trial: TGN1412. J Med Ethics. 2009;35:573–8.
Knapp P, Raynor D, Silcock J, Parkinson B. Can user testing of a clinical trial patient information sheet make it fit-for-purpose? - a randomised controlled trial. BMC Med. 2011;9:89.
Madurasinghe VW, Bower P, Eldridge S, Collier D, Graffy J, Treweek S, Knapp P, Parker A, Rick J, Salisbury C, et al. Can we achieve better recruitment by providing better information? Meta-analysis of ‘studies within a trial’ (SWATs) of optimised participant information sheets. BMC Med. 2021;19(1):218.
Synnot A, Ryan R, Prictor M, Fetherstonhaugh D, Parker B. Audio-visual presentation of information for informed consent for participation in clinical trials. Cochrane Database Syst Rev. 2014;5:003717.
Hermann M. Dreidimensionale Computeranimation – neues Medium zur Unterstützung des Aufklärungsgesprächs vor Operationen Akzeptanz und Bewertung der Patienten anhand einer prospektiv randomisierten Studie – Bild versus Text. Chirurg. 2002;73(5):500–7.
Tait AR, Voepel-Lewis T, Levine R. Using digital multimedia to improve parents’ and children’s understanding of clinical trials. Arch Dis Child. 2015;100(6):589–93.
Hutchison C, Cowan C, McMahon T, Paul J. A randomised controlled study of an audiovisual patient information intervention on informed consent and recruitment to cancer clinical trials. Br J Cancer. 2007;95:705–11.
Knapp P, Mandall N, Hulse W, Roche J, Moe-Byrne T, Martin-Kerry J, Sheridan R, Higgins S. Evaluating the use of multimedia information when recruiting adolescents to orthodontics research: a randomised controlled trial. J Orthod. 2021;48(4):343–51.
Treweek S, Pitkethly M, Cook J, Fraser C, Mitchell E, Sullivan F, Jackson C, Taskila TK, Gardner H. Strategies to improve recruitment to randomised trials. Cochrane Database Syst Rev. 2018;2:00013.
Madurasinghe V, Sandra Eldridge on behalf of MRC START Group and Gordon Forbes on behalf of the START Expert Consensus Group. Guidelines for reporting embedded recruitment trials. Trials. 2016;17(1):27.
Sheridan R, Martin-Kerry J, Hudson J, Parker A, Bower P, Knapp P. Why do patients take part in research? An overview of systematic reviews of psychosocial barriers and facilitators. Trials. 2020;21(1):259.
Jolly K, Sidhu M, Bower P, Madurasinghe V, Eldridge S, Graffy J, Parker A, Knapp P, Torgerson D, Treweek S, et al. Improving recruitment to a study of telehealth management for COPD: a cluster randomised controlled ‘study within a trial’ (SWAT) of a multimedia information resource. Trials. 2019;20(1):453.
Bhandari S, Mehta S, Khwaja A, Cleland JGF, Ives N, Brettell E, Chadburn M, Cockwell P. Renin–angiotensin system inhibition in advanced chronic kidney disease. N Engl J Med. 2022;387(22):2021–32.
Fife-Schaw C, de Lusignan S, Wainwright J, Sprake H, Laver S, Heald V, Orton J, Prescott M, Carr H, O’Neill M. Comparing exercise interventions to increase persistence with physical exercise and sporting activity among people with hypertension or high normal blood pressure: study protocol for a randomised controlled trial. Trials. 2014;15(1):336.
Jolly K, Sidhu MS, Hewitt CA, Coventry PA, Daley A, Jordan R, Heneghan C, Singh S, Ives N, Adab P, et al. Self management of patients with mild COPD in primary care: randomised controlled trial. BMJ. 2018;361:k2241.
Thomas KS, Batchelor JM, Akram P, Chalmers JR, Haines RH, Meakin GD, Duley L, Ravenscroft JC, Rogers A, Sach TH, et al. Randomized controlled trial of topical corticosteroid and home-based narrowband ultraviolet B for active and limited vitiligo: results of the HI-Light Vitiligo Trial*. Br J Dermatol. 2021;184(5):828–39.
Hull MA, Sprange K, Hepburn T, Tan W, Shafayat A, Rees CJ, Clifford G, Logan RF, Loadman PM, Williams EA, et al. Eicosapentaenoic acid and aspirin, alone and in combination, for the prevention of colorectal adenomas (seAFOod Polyp Prevention trial): a multicentre, randomised, double-blind, placebo-controlled, 2 × 2 factorial trial. Lancet. 2018;392(10164):2583–94.
Dawson S, Banister K, Biggs K, Cotton S, Devane D, Gardner H, Gillies K, Gopalakrishnan G, Isaacs T, Khunti K, et al. Trial Forge Guidance 3: randomised trials and how to recruit and retain individuals from ethnic minority groups—practical guidance to support better practice. Trials. 2022;23(1):672.
Knapp P, Moe-Byrne T, Martin-Kerry J, Sheridan R, Roche J, Coleman E, Bower P, Higgins S, Stones C, Graffy J, et al. Providing multimedia information to children and young people increases recruitment to trials: pre-planned meta-analysis of SWATs. BMC Med. 2023;21(1):244.
Treweek S, Bevan S, Bower P, Briel M, Campbell M, Christie J, Collett C, Cotton S, Devane D, El Feky A, et al. Trial Forge Guidance 2: how to decide if a further Study Within A Trial (SWAT) is needed. Trials. 2020;21(1):33.
Acknowledgements
We would like to acknowledge all those in the trial teams who supported this programme of SWATs, as well as our public contributors (Ailsa Donnelly and Judith Hogg). We also thank Paul Wallace (original MRC START applicant) and Reason Digital for multimedia development. We thank the patients who participated in the GHT2000, PSM COPD, HI-Light, seAFOod and STOP-ACEi trials, the members of the Birmingham and Nottingham Clinical Trials Units, the Research and Development Team at Hull University Teaching Hospitals NHS Trust, and the staff members at all the participating trial centres.
Funding
The authors wish to acknowledge the MRC Methodology Research Programme which funds this research (MRC grant reference: G1002325). The MRC has no role in study design; collection, management, analysis and interpretation of data; writing of the report; or the decision to submit the report for publication.
Author information
Authors and Affiliations
Contributions
PB, SE, DC, JG, ST, PK, CS and DT were applicants on the original bid and contributed to design and delivery of the study. JR managed the programme of work with AP, and VM conducted the analyses with SE. KJ, CF, MS, MH, KS, AM, SB and EB designed and delivered the SWATs in host trials. PB, VM and PK drafted the paper. All authors read the manuscript, revised it critically, and approved the final manuscript.
Corresponding author
Ethics declarations
Ethics approval and consent to participate
The START programme of recruitment strategies was approved by the National Research Ethics Service (NRES) Committee, Yorkshire and the Humber – South Yorkshire (Ref: 11/YH/0271) on 5 August 2011. Host trials had their own approvals.
Consent for publication
Not applicable.
Competing interests
The authors declare that they have no competing interests.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Additional file 1.
Examples from the multimedia intervention.
Additional file 2.
Presentation of the resource to patients.
Additional file 3.
Raw data from STOP ACEI.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
About this article
Cite this article
Madurasinghe, V.W., Knapp, P., Eldridge, S. et al. Can we achieve better trial recruitment by presenting patient information through multimedia? Meta-analysis of ‘studies within a trial’ (SWATs). BMC Med 21, 425 (2023). https://0-doi-org.brum.beds.ac.uk/10.1186/s12916-023-03081-5
Received:
Accepted:
Published:
DOI: https://0-doi-org.brum.beds.ac.uk/10.1186/s12916-023-03081-5