How to present economic evaluations to non-technical audiences? Randomized trials with professionals and the general population

Abstract

Background

Cost-effectiveness analyses of health technologies have become a part of the decision-making process in healthcare policies. Nevertheless, economic results are not always presented in comprehensible formats for non-technical audiences, such as the general population, healthcare professionals or decision-makers. The purpose of this study was to observe which formats best convey the key message of an economic evaluation, and which are best received by two different audiences.

Methods

The summary of a hypothetical cost-effectiveness analysis was edited in different formats: infographic, plain language text and video-abstract for the general population; executive summary and policy brief for a specialized population, i.e. clinicians, clinical and non-clinical managers, or methodologists in health-technology assessment. Participants were randomly shown one of the formats, and data on objective and subjective comprehension, and perceived usefulness/acceptability were gathered by means of online questionnaires. Statistical differences between formats within each audience were analysed.

Results

In the general population (N = 324), objective comprehension was statistically significantly better for infographic than for video-abstract (p = 0.005), and for plain text than for video-abstract (p = 0.024). There were no differences in subjective comprehension, but video-abstract was considered statistically significantly more useful to understand the information than plain text (p = 0.011). In the specialized population (N = 100), no statistically significant differences were observed for objective and subjective comprehension, although policy brief was perceived as statistically significantly more useful than executive summary (p = 0.005).

Conclusions

A balance between effectivity of conveying the message and attractivity of the format needs to be sought, to facilitate non-technical audiences’ understanding of economic data and, consequently, perceive decision-making processes as more transparent and legitimate. The infographic and policy brief could be robust ways to present economic data to the general public and specialized audience, respectively.

Text box 1. Contributions to the literature
• There is evidence that presentation format influences the comprehension of clinical data, but evidence regarding economic data is lacking.
• This study shows how different ways of presenting cost-effectiveness results affect how well people understand and receive the information.
• Health economists need to find a good balance between the format’s attractiveness and its ability to convey the message.

Background

Economic evaluations of health technologies are part of key information used by healthcare authorities for decision-making on financing of alternative technologies. This guarantees that both effectiveness and costs are considered in the final decision. In decentralized systems such as Spain, these decisions are taken not only at high but also at lower levels, for example, hospitals or local health authorities, where decision-makers may not always have demonstrable expertise in health economics. At the same time, having a certain level of knowledge can help the general public better understand the decision-making process and the reasons behind why health technologies that are effective are not always financed by the National Healthcare Service [1, 2]. Naturally, these non-technical audience groups normally do not have the same technical knowledge as health economists.

Furthermore, health economists often do not think of non-technical audiences and present conclusions of their studies using language and formats which are difficult to understand for the economically illiterate. Although guidelines for economic evaluations provide recommendations regarding good practices for the presentation of results, these can fail to attain proper communication and transfer of information to non-technical audiences [3].

While it is the healthcare authority’s role to use research results and communicate the decision-making process, it is also critical that cost-effectiveness results be presented to meet the varying levels of economic literacy across different decision-making contexts. There is evidence that challenges exist when decision-makers need to understand technical concepts such as Quality Adjusted Life Years (QALY) or graphics, such as cost-effectiveness planes [2, 4, 5]. In Spain, decision-makers have reported difficulties in the comprehension of economic techniques, excessive theoretical content and lack of reality in results [5]. Interpreting results is even harder when the intervention evaluated is more effective but also costlier than the comparator. In these cases, the incremental cost-effectiveness ratio (ICER) must be calculated, which is expressed in euros (or any chosen currency) per a gained QALY. The final decision regarding whether an intervention is cost-effective or not relies on the cost-effectiveness threshold established, also called willingness to pay. Therefore, the economic evaluation user should be able to understand multiple concepts (QALY, ICER, threshold) to interpret the results correctly.

Methods used to transfer knowledge have commonly been applied to improve comprehension of research data by decision-makers, healthcare professionals and patients [6, 7]. General recommendations include the use of simple graphs and abstracts drawn up in plain language and containing straightforward information. Providing information that is straightforward but also solid, about results of economic evaluations aimed at non-technical audiences is a complex task [8]. A balance needs to be struck so that information is understandable but also upholds the integrity and precision of the economic evaluation. When overly simplified, technical information may lose some of the relevant nuances for decision-making. There are many ways to present scientific data, from simple texts, such as executive or plain language summaries [9, 10], to formats with graphic elements, such as infographics or policy briefs [6, 11], or even formats using audio or video components, such as video-abstracts or podcasts [12].

These needs and challenges justify the research efforts made to better communicate the results aimed at decision-makers as well as other non-technical audiences. Although previous studies have analysed the impact of several formats for communicating clinical results to different audiences [13,14,15], to the best of our knowledge, none has focused on the results of health economic evaluations. Therefore, the aim of this study was to design different formats to present an economic evaluation summary and observe which of them best conveys the key message and is best received by two different audiences: general and specialized public.

Methods

Aim, design and setting of the study

The main aim of this study was to evaluate the impact of different formats on understanding economic results in two non-technical audiences.

A hypothetical cost-effectiveness analysis was created only for the purpose of this study. This evaluated a new type of total hip prosthesis compared to a traditional prosthesis from the healthcare system perspective, using a Markov model. The hypothetical results revealed that the intervention was more effective and costlier than the comparator. However, only for patients aged < 65, the intervention was cost-effective, meaning that the ICER was below the cost-effectiveness threshold, estimated in Spain at €25,000/QALY [16]. For patients aged over 65, the ICER was higher than the threshold. Therefore, the intervention was not cost-effective for this patient subgroup. These results were summarized in several formats.

Two randomized trials were conducted with an online survey (one trial in each audience). The study was performed in Spain in 2023.

Participants

Different formats were presented to two different audiences:

• 1. General population: adult residents in Spain. Participants were recruited by a specialized company by means of an online panel, a pre-recruited group of individuals who have essentially agreed to take part in studies such as online surveys.

• 2. Specialized population in healthcare sector: adult residents in Spain who were medical doctors, health centres or unit managers or HTA methodologists. Participants were recruited by means of e-mail invitations sent by the researchers, and through an institutional social media call.

Intervention: formats of the economic evaluation summary

For the general population, three summary formats containing the same information were designed, using plain language according to guidelines [17]:

• Plain language summary: a 300-word text with no tables or graphics.

• Infographic: a visual representation combining graphic elements and text.

• Video-abstract: a two-minute-long video with illustrative images and spoken script.

For the specialized population, two formats were designed with information commonly used in cost-effectiveness analysis abstracts:

• Executive summary: a 470-word text with no tables or graphics.

• Policy brief: a concise three-page summary, with visual elements, tables and graphics for the results and further explanations of some key concepts (e.g. ICER, willingness to pay, etc.).

All formats drawn up in Spanish and their translation into English can be downloaded from the Zenodo repository [18].

Questionnaires and outcome measures

For each audience, a questionnaire was drawn up on the LimeSurvey platform [19]. The survey was anonymous and no personal data were requested. Study information was presented and informed consent was obtained before commencing the questionnaire.

The initial section of the questionnaire requested demographic data, professional background and format preferences. The second section randomly presented one of the formats (randomization was generated by the LimeSurvey platform). There was no time limit to read the information and the video-abstract could be viewed for a second time if required. Participants were not allowed to go back to the summary once they moved on to the next section.

Questions were then asked to collate data on objective and subjective comprehension and the format’s perceived usefulness and acceptability. Questions to measure these outcomes were based on similar tools used in previous research on comprehension of scientific information [14, 20]. Objective comprehension was defined as correctly answering all three questions about the content. Subjective comprehension was measured as the level of confidence in participants’ answers (how sure are you that your answers are correct?) on a five-point Likert-type scale (from “Not at all sure” to “Completely sure”). Participants were also asked to evaluate the usefulness of the format for comprehension of the information on a five-point scale (from “Not at all useful” to “Completely useful”). Eight additional questions on other subjective aspects were asked (satisfaction with the appearance, lack/excess of numeric information or explanations, appropriateness, etc.).

Before launching the survey, a pilot survey was performed with 42 participants from the general public and six specialists, in which correct comprehension and clarity of the questionnaire was confirmed and no further modifications of the questionnaires were necessary. The full questionnaire can be downloaded from the Zenodo repository [18].

Statistical analysis

Before commencing the trials, the required sample sizes were calculated using a 95% confidence and a minimum statistical power of 80% to detect a difference in average knowledge score between formats. We used findings from Buljan et al. [14] to define a standard deviation of 3.723 and 2.481 points for the general public and specialists, respectively. It was estimated at 97 and 43 individuals per format group for the general public and for specialists, respectively; i.e., the total required sample size was 291 and 86 participants from the general population and specialists, respectively.

Data for each population were analysed separately. Within each population, format groups were defined according to the format randomly presented to the participant (e.g. infographic group, plain text group, etc.). For categorical variables, both counts and percentages are shown. The proportion test was used to observe differences between types of formats. For numerical variables, the mean and the standard deviation are shown. Differences between formats within audience groups were analysed using student t-tests, analysis of variance (ANOVA) or Kruskal-Wallis test, according to the type of the variable and number of compared formats. The chi-squared test was used to determine differences between categorical variables and the Cramer’s V to determine the strength of association. P-values of 0.05 or lower were considered statistically significant. All statistical analyses and graphs were performed using R Statistical Software (v4.1.2; R Core Team 2021).

Results

Participants’ characteristics

The flow of participants in each trial is shown in Fig. 1. Demographic characteristics and format preferences are shown for each population in Tables 1 and 2.

Fig. 1

Flow of the study participants

General population

A total of 344 persons accessed the questionnaire and 324 completed all the questions. Of the full responses, 120, 114 and 90 were randomly shown the infographic, the plain text and the video-abstract, respectively (Fig. 1). Most participants had a tertiary education level and their profession was unrelated to either healthcare, research or economics.

There were no statistically significant differences between format groups, except for the audio format preferences (less preferred in the plain text group than in the other two groups). The video format was the most preferred format to receive scientific information in the three groups (Table 1).

Table 1 Participants’ characteristics and preferences. General population

Specialized population

A total of 116 specialists accessed the questionnaire and 100 completed all the questions. Of these, 46 and 54 were randomly shown the policy brief and the executive summary, respectively (Fig. 1).

There were no statistically significant differences between format groups. Most participants were clinicians. Although virtually half the sample considers that economic data on health technologies is more or less understandable, approximately 30% of participants believe it is hard to understand or incomprehensible. The specialized population prefers formats that include text with or without graphic elements, especially the scientific paper format, to receive research or HTA results, while audio-visual formats are less preferred (Table 2).

Table 2 Participants’ characteristics and preferences. Specialized population

Objective and subjective comprehension

In the general population, objective comprehension (correctly answered questions) appeared to be statistically significantly different between formats (p = 0.014). Two-by-two comparisons revealed statistical differences between infographic vs. video-abstract (p = 0.005), and plain text vs. video-abstract (p = 0.024), meaning that objective comprehension was highest in the infographic group (73.3% answered to all three questions correctly) and lowest in the video-abstract group (54.4%) (Table 3). However, no statistically significant differences were observed for subjective comprehension (confidence). The professional background of the participants (whether the profession was related to healthcare, research or economics) did not have a significant impact on objective or subjective comprehension (S1A in Supplementary Material).

In the specialized population, no statistically significant differences were observed for objective and subjective comprehension between format groups, although both measures were slightly better in the Executive summary group than in the Policy brief group. Statistical significance was not attained: 74.1% vs. 60.9% answered all three questions correctly; while 75.9% vs. 67.4% were quite or completely sure about their answers (Table 3). The professional profile of participants did not have a significant impact on objective or subjective comprehension (S1B in Supplementary Material).

Table 3 Objective and subjective comprehension in general and specialized population

Perceived usefulness of the format

In the general population, statistically significant differences between formats were detected for usefulness (p = 0.04). Two-by-two comparisons revealed that video-abstract was perceived significantly more useful than plain text (p = 0.011). No statistically significant differences were found in the rest of comparisons. Among specialists, policy brief was perceived as significantly more useful than executive summary (p = 0.005) (Table 4).

Table 4 Perceived usefulness of the presented format in general and specialized population

These results were partially confirmed in subsequent questions on satisfaction and acceptability of the formats (Figs. 2 and 3, and S2 in Supplementary Material). Participants in the video-abstract and infographic groups expressed greater satisfaction with the appearance of the format than in the plain text group (p = 0.026; p = 0.005, respectively) (question A1), and video-abstract and infographic was found more appropriate for this type of information than plain text (p = 0.003; p = 0.043, respectively) (question A4) (Fig. 2 and S2A in Supplementary Material).

Fig. 2

Participant-reported satisfaction and acceptability. General population. * Differences are statistically significant (p-values ≤ 0.05)

Specialists in the policy brief group found the format more appropriate than in the executive summary group (p = 0.010) (question B5). Moreover, more participants in the policy brief group felt that the information was easy to remember to answer the questions (p = 0.030) and understandable for someone with no knowledge of health economics (p = 0.005) (questions B6 and B7). No statistically significant differences were found for the rest of the questions (Fig. 3 and S2B in Supplementary Material).

Fig. 3

Participant-reported satisfaction and acceptability. Specialized population. * Differences are statistically significant (p-values ≤ 0.05)

Relationship between main outcomes

Relationship between subjective and objective comprehension was explored to confirm the assumption that people with more correct answers will have more confidence in how they responded. Indeed, the heat maps confirm this relationship (Figs. 4 and 5). In all three groups of the general public there was a statistically significant association between the outcomes, and approximately one third of people answered all three questions well and at the same time were “completely sure” about their answers. However, in the video-abstract group, there were more people who answered one, two or even all the questions incorrectly and were still “completely or quite sure” about their answers (26.6%), than in the other two groups (12.6% and 13.1% in the infographic and plain text groups, respectively). This could possibly mean that the video-abstract could give the audience a false conviction that they understood the results (Fig. 4). In the specialized audience, a statistically significant association between objective and subjective comprehension was observed only in the policy brief group (V = 0.434, p < 0.01). In the executive summary group, most of the sample answered all three questions correctly but was only “quite sure” about their answers (Fig. 5).

Fig. 4

Heatmap for the general population: Relationship between objective and subjective comprehension. The X axis represents the level of objective comprehension, i.e. number of correctly answered questions. The Y axis represents the level of subjective comprehension, i.e. how sure you are about your answers. V: Cramer’s V strength of association. Levels of significance: *** p < 0.001; ** p < 0.01; * p < 0.05

Fig. 5

Heatmap for the specialized population: Relationship between objective and subjective comprehension. The X axis represents the level of objective comprehension, i.e. number of correctly answered questions. The Y axis represents the level of subjective comprehension, i.e. how sure you are about your answers. V: Cramer’s V strength of association. Levels of significance: *** p < 0.001; ** p < 0.01; * p < 0.05

A relationship between objective comprehension and perceived usefulness could mean that if users feel comfortable with the format, they will get more right answers. This was confirmed for all three formats in the general public (Fig. 6). In the specialized population there was no statistically significant relationship between objective comprehension and usefulness, which could be explained by the absence of responses in low levels of both variables (Fig. 7).

Fig. 6

Heatmap for the general population: Relationship between objective comprehension and usefulness. The X axis represents the level of objective comprehension, i.e. number of correctly answered questions. The Y axis represents the level of perceived usefulness of the format to understand the information. V: Cramer’s V strength of association. Levels of significance: *** p < 0.001; ** p < 0.01; * p < 0.05

Fig. 7

Heatmap for the specialized population: Relationship between objective comprehension and usefulness. The X axis represents the level of objective comprehension, i.e. number of correctly answered questions. The Y axis represents the level of perceived usefulness of the format to understand the information. V: Cramer’s V strength of association. Levels of significance: *** p < 0.001; ** p < 0.01; * p < 0.05

Discussion

The main aim of this study was to measure the impact of the format on understanding the key messages of an economic evaluation by non-technical audiences. In addition, the perceived usefulness and acceptability of the formats was also explored. To the best of our knowledge, this is the first study analysing this impact in the context of health economic evaluations. The design of the formats reflected the target audience: for the general public, plain language was used in infographic, text and video format, while for professionals from the healthcare sector, more specialized language was used in executive summary and policy brief formats. A summary of a hypothetical economic evaluation was created for the purpose of this study, intentionally choosing a well-known disease and a situation in which the intervention is more effective and costlier than the comparator. It is, therefore, necessary to calculate the ICER and compare it to the cost-effectiveness threshold. Moreover, conclusions drawn were different for the two patient subgroups according to their age.

For the general population it was observed that participants understood the key messages significantly better from the infographic or plain text than from video-abstract. However, the perceived usefulness and acceptability was better for the video-abstract than for the plain text (not better than for infographic). But, our results also suggest that with the video-abstract, the audience could be wrongly convinced that they have a good understanding of the information. Based on this, the infographic could be a good way to present economic results to this audience, and if we prioritize comprehension over attractiveness, plain text could be an alternative to the infographic.

The specialized public had similar levels of comprehension after reading the executive summary or the policy brief (slightly better for the executive summary), but the policy brief was clearly perceived as more useful and convenient to convey the key messages. Therefore, in the absence of statistical differences in comprehension between formats, a policy brief may be the preferred choice. Interestingly, participants’ background in both populations did not have a significant influence on their objective or subjective comprehension.

Lastly, a relationship was observed between the main endpoints, i.e. objective vs. subjective comprehension, and objective comprehension vs. usefulness. This could indicate that the summaries were well designed and adequate for the target audience, especially in case of the general public. In this audience, our results confirm that having more correct answers relates to higher confidence and also to more usefulness perceived, for all three formats. The false conviction in understanding the results, that the video-abstract format appeared to give the audience, is not necessarily due to the format itself and should be studied further. Subjective comprehension, i.e. the level of confidence, cannot be compared between audiences because the objective questions and formats were different. However, specialists who answered all questions correctly did not show complete confidence in their answers, unlike the general public. Further qualitative research would be needed to clarify the reasons for these findings.

Visualization of information is a key factor in the popularization of technical-biomedical aspects [21]. Infographics have recently been included by the Cochrane Collaboration to present results from systematic reviews [11], in addition to scientific summaries, tables of results and summaries using plain language, as well as other innovative formats such as blogshots. Previous studies have developed similar analyses to ours but focused solely on clinical outcomes, whose conclusions on the comparison of different formats are summarized below. Current evidence about the use of infographic vs. other formats to present information is mixed [13, 22, 23]. There is evidence that the optimal format depends on the audience. For example, regarding systematic reviews it has been proven that professionals prefer information based on text, while the general public is more comfortable dealing with an infographic [13], which is in line with this study’s results. Similarly, when dealing with information about the risk-benefit ratio, parents prefer infographics over text and table [24].

Audiovisual format, such as video format or animation combined with sound, is a new way of communicating with the target population. Video-abstracts can include even more information in a concise format than infographics [25]. Some scientific journals have started offering video-abstracts as well as infographics and simple language abstracts for published articles [15]. This presentation has also been used for informed consent in clinical trials to improve participation and participant comprehension in regard to objectives, methods and potential benefits and risks [12, 26, 27]; and to illustrate information about the risks of an intervention, but without conclusive results [28]. Bredbenner et al. [15] presented summaries of HIV research and concluded that people value audiovisual formats better. In terms of comprehension of the information presented, this format, together with plain language summaries, received higher ratings than other formats. This study confirms that the video format is attractive for the general audience, but it may not be the most effective in conveying the key messages of economic evaluations. A combination of several formats, e.g. infographic and video-abstract, could further improve comprehension, but more research is needed.

It is fair to say that communication with the public extends beyond just simplifying the cost-effectiveness report itself. This also includes transparency about the processes involved in commissioning, conducting, and interpreting economic analyses. The complex task of decision makers to communicate not only the results, but also the methodology and decision-making processes, plays a critical role in ensuring the public’s understanding and trust.

Strengths and limitations

In the health economics field, there has been a lack of evidence on the effectiveness of different summary formats to convey key messages of economic evaluations. To the best of the authors’ knowledge, this study is the first to measure the objective and subjective comprehension of economic evaluation summaries.

One limitation is that the analysis here only identifies which format is better when a research team carefully creates content-identical summary formats following the rules set out by best practice guidelines. There is still further work to be performed to measure the comprehension of different types of economic results presented to specific audiences. Podcast was one type of summary format omitted in this study, but it is gaining in popularity.

Data were collected from a pre-recruited sample, which could differ from the general population in their characteristics, e.g. low representation of lower education levels. However, the randomization guaranteed that the groups were comparable between them. An online survey with a limited number of close-ended questions was used, although other techniques can be used to explore user preferences regarding format, such as interviews or focus groups, typical of qualitative research [29].

Conclusions

Adequate formats for presenting results would facilitate decision-makers to consider economic evaluation data. However, it is as important that other audiences, affected by these decisions, are also able to understand economic data and consequently perceive the decision-making processes as more transparent and legitimate.

To best communicate economic results, infographics could be a good way to present them to the general public. Among specialists, policy brief was preferred over executive summary. However, a good balance between the effectivity of conveying the message and attractivity of the format needs to be sought, especially for the general population.