• We are sorry, but NCBI web applications do not support your browser and may not function properly. More information
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptNIH Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
Patient Educ Couns. Author manuscript; available in PMC Dec 1, 2011.
Published in final edited form as:
PMCID: PMC2993846
NIHMSID: NIHMS238280

Relevance of CONSORT Reporting Criteria for Research on eHealth Interventions

Abstract

Objective

In 1996, 2001, and 2010, the Consolidated Standards of Reporting Trials (CONSORT) group released criteria for reporting critical information about randomized clinical trials (1, 2). These criteria were intended to improve the quality and completeness of reporting of RCTs in health care research. This paper discusses the relevance of the CONSORT recommendations for the reporting and design of eHealth research.

Methods

We reviewed the CONSORT recommendations and discussed their particular relevance to eHealth (electronic information, support and/or communication resources designed to promote health) research. This review focuses on such issues as recruitment and screening of participants, description of treatment elements, and reporting of outcome data and adverse events.

Results

eHealth research presents special challenges regarding the comprehensive and effective reporting of research information. However, the strategic application of CONSORT recommendations holds great promise for improving the quality and informativeness of eHealth research.

Conclusion

Investigators need to consider CONSORT recommendations at all stages of the research enterprise, including planning, execution and reporting in order to increase the informativeness of their research efforts.

Practice Implications

The recommendations contained in this paper have the potential to enhance the public health and scientific value of eHealth research.

1. Introduction

eHealth comprises stimuli, media, games, interventions, support and/or communication resources that are conveyed via electronic media and are designed to promote health. It is likely that eHealth interventions will become increasingly important in the delivery of health care in the foreseeable future. Multiple factors will contribute to this trend. First, because of time and economic constraints on health care delivery systems, health care providers may be unable to provide all the information that patients desire and need in a timely manner via more traditional, face-to-face approaches. In addition, as numbers and complexities of treatments escalate for different medical conditions, the amount of relevant information is also escalating, making it more difficult to provide complete information. In addition, traditional models of patient-clinician relationships appear to be changing, with patients and families often wanting to obtain information that supplements what they get from their clinician. Furthermore, there are roles that clinicians simply cannot perform as well as computers: linking patients to others who are facing the same problem, integrating large amounts of complex information, weighting it optimally based on characteristics of the patient, and presenting it over time or at critical times.

Recent data reveal that 74% of the U.S. population (220,000,000 people) use the Internet (3) and this figure will no doubt continue to grow as resource development and new technology make the high-speed Internet more user-friendly and accessible. Health information and support is essentially omnipresent on the Internet. Internet sites and delivery platforms can: convey a vast amount of information in a timely manner, be highly responsive to the patient's intentions and desires, be updated rapidly, be accessed across time and location, and be offered at negligible cost on a per user basis. In addition, because only a single intervention need be prepared for all users (including any tailored elements), every patient can receive an intervention designed by an expert, which might increase effectiveness. In sum, electronic media can make bulletin boards, blogs, instant messaging, e-mail, Web conferencing, interactive communication services, games and other resources readily available to individuals – providing them with interventions and communication with both experts and their peers.

Because of these significant advantages, numerous eHealth interventions have already been developed and are available to millions of individuals (4-11). Such interventions are widely used. Data published in 2006 show that 63.7% of the online population in the U.S. have looked for health information for themselves or others at least once in the previous 12 months (12). Such a trend may augur well for health in the U.S. and elsewhere, if eHealth interventions are truly effective.

While eHealth interventions are now widely used, relatively little well-controlled research exists that characterizes their efficacy or effectiveness1. Thus, on the one hand, patients and other individuals spend countless hours using such interventions, and developers devote substantial time and resources in creating and updating these interventions. On the other hand, research has not yet revealed: 1) which sorts of eHealth interventions are most effective, 2) for whom they are most effective, and finally, 3) how or why such interventions work, if indeed they do work.

In some ways, it should be easier to identify efficacious eHealth interventions than it is to identify efficacious psychosocial therapies delivered person-to-person. This advantage might accrue due to the greater standardization possible with programmed interventions. Face-to-face interventions might be delivered very differently across either time or therapists, making it difficult to disentangle content and stylistic factors that might influence a patient's outcomes. Programmed interventions permit programmatic testing of the impacts of specific types of contents, specific content “doses,” and specific delivery strategies. Thus, the widespread use of eHealth interventions, the feasibility of conducting research on such interventions, and the relative dearth of well-controlled outcome trials, all argue for greater programmatic research on such treatments.

1.1 Reporting Research on eHealth Interventions

In 1996, the Consolidated Standards of Reporting Trials (CONSORT) group released criteria for reporting critical information about randomized clinical trials (RCTs; (2). These reporting criteria were intended to improve the quality and completeness of reporting of randomized controlled trials in health care research. In 2001, the CONSORT group released a revised version of such criteria along with explanations (1). Subsequently, research has shown that the use of such criteria is associated with improvements in the quality of reports of RCTs (13, 14). In addition, research suggests that poor quality RCTs – ones that do not meet adequate standards for design and execution – yield biased estimates of effectiveness (14). The inspiration for this line of work is the notion that a clear set of reporting standards will not only enhance the value of reports of RCTs (15), but also may enhance the design and conduct of such trials. This work is supported by the view that “well designed and properly executed RCTs provide the best evidence on the efficacy of health care interventions” (1).

Reporting standards for eHealth RCTs should be similar to those recommended for other sorts of RCTs. However, eHealth interventions and research present special features, opportunities, and challenges that suggest the need for a focused appraisal of the CONSORT recommendations. The CONSORT statement has proven to be sufficiently influential to cause several medical specialties to offer their own calls for standards in reporting on clinical trials (e.g., (16). Inspired by the CONSORT statement, we similarly provide recommendations for eHealth clinical trials in the hope of raising the quality of studies in this area.

2. Methods

2.1 CONSORT criteria

This paper is intended to alert researchers and others to important methodological issues and reporting concerns relevant to eHealth trials. Our discussion of these issues will be guided by the criteria raised in the 2001 CONSORT statement (1) and any relevant changes indicated in the recent 2010 update (17). This paper is intended to discuss the relevance of these principles to eHealth research. The aims of this paper are both to influence reporting of eHealth research and also to serve as a resource for eHealth researchers as they design their research studies.

Table 1 presents the CONSORT criteria (17). This table directs the investigator's attention to design and reporting factors that should be considered and reported in RCTs. Some topics raised in the CONSORT statements are not discussed, or discussed only briefly, because their application to eHealth research is straightforward. Topics are discussed in order in which they are listed in Table 1, which is organized by sections of an empirical RCT report (with topic label and number indicated in parentheses).

Table 1
CONSORT Selected Criteria Adapted for eHealth RCTs (From Schulz et al., 2010)

2.2 Introduction: Background and Objectives (Scientific background and explanation of rationale, 2a; Specific objectives or hypotheses, 2b)

Researchers need to state clearly the objectives of their research. This refers not only to the hypotheses invoked, but also to the ultimate objectives. Is the eHealth intervention intended to be a stand-alone intervention or is it to be incorporated into a broader health care program? Is the intervention intended for a particular patient population? Is the intervention ultimately intended to be cost-effective relative to other interventions – if so, relative to what interventions in particular? Answers to these questions will enable the reader to determine how similar the current evaluation context is to the ultimate, intended use of the intervention, and the extent to which outcome measures assess relevant domains.

Researchers should not only provide the information discussed above, but they should also provide a clear rationale for the choices that they made in designing the eHealth intervention for the specific health problems, populations, or outcomes that they intended to address: i.e., why did they select the type of goals, delivery platform, and presentation styles that they did? That is, the researcher should ideally provide the reader with a rationale that explains why the intervention form and content were chosen to achieve the goals identified by the researcher.

The rationale should often include why a particular population is targeted for the research and intervention, and the relevance of this to the design of the intervention and how relevant participant/person factors might drive treatment use and outcomes in eHealth research vs. traditional delivery systems. For instance, while outcomes in traditional interventions might be related to the tendency or ability to form a meaningful or warm interpersonal relationship with the therapist, success in an eHealth intervention might reflect the ability to use computers well, literacy, availability of naturally occurring social support, access to efficient applications (e.g., with high-speed Internet access), outside time obligations that might compete with heavy use of the Internet, lack of prior exposure to Internet interventions (there is a greater novelty effect), and so on. Thus, the rationale should indicate the individuals for whom the intervention is intended, why it is especially appropriate for those individuals, and how this conceptualization or model influenced the choice of measured participant characteristics. As noted above, these characteristics might not be those typically measured in research, and their assessment should allow the researcher to mount ancillary research on subpopulations and suggest the population to which the research findings might be extrapolated (see point 18, Table 1). The important point is that the rationale should focus on population issues as well as other design considerations.

If the researchers have proposed relevant theory, they should describe the use parameters that they believe should lead to optimal outcomes. For instance, what is the optimal timing of use? If the intervention is designed to encourage smoking cessation, is it important for the user to begin using the program before the quit day? And, if the user is successful in quitting, how long should the user continue to use the program after s/he has quit? What instructions or recommendations were given to the user regarding timing, frequency, or heaviness of use? In the case of an eHealth intervention for breast cancer, when should the woman gain access to the intervention: as soon as possible after diagnosis or after initial treatments are finished? In addition, what does the researcher's theory say about use-benefit relations: e.g., is there a minimum dose (overall duration of use) that is needed to obtain benefit? Is there some pattern of use or combination of services that is targeted? Consideration of these characteristics of eHealth interventions can help the researcher sharpen his or her thinking about a tested intervention as well as improve communication of trial results. Thus, while most trials list major hypotheses or objectives (e.g., 18), there may be benefit to elaborating such discussion.

2.3. Methods: Trial Design (Description of trial design (such as parallel, factorial) including allocation ratio, 3a)

While this CONSORT reporting element refers in particular to the experimental design used (e.g., fractional factorial design), it seems especially important and difficult in eHealth research to select a control condition that reflects the objectives of a study (point 2b above) and then to describe it adequately (see point 5 below). Choice of a control or comparison condition demands that the researcher identify the features or facets of an intervention that s/he wishes to test (see also (4). For instance, unguided access to the Internet has been used as a control condition (19). Presumably, this controls for access to the Internet per se, and the availability of computer equipment. This also may control for such nonspecific effects such as positive expectations. However, it is important to note in such cases that the control condition is messy; the control subject may be exposed to many putative “active” elements of treatment. For instance, the eHealth intervention to be evaluated may contain person-to-person support services and up-to-date information. These services may also be readily available on the Internet for the specific disease or condition that is targeted. In this case, the control condition is actually “controlling for” many active elements of treatment. This would be acceptable if the researcher clearly believed that the evaluated treatment were sufficiently different in some way so as to make it highly distinct from other Internet resources. For instance, the investigator may surmise that while the basic services comprised by an intervention are not unique, the intervention's interface and access to services will make the intervention superior (see 20). Otherwise, the researcher is courting Type II error and possibly promoting false inferences based upon his/her research; i.e., that is, the researcher would mistakenly infer that components of his/her eHealth intervention (person-to-person support via discussion groups) do not produce significant health benefits. Unfortunately, using statistical control methods to partial variance related to fortuitous resource/intervention use by control subjects will result in confounded inferences (21).

It is also important to note that the impact of a control condition (to say nothing of an active intervention) may be influenced by population characteristics such as baseline access to resources. For instance, if computer provision and Internet access are used as a control condition, this might have very different effects on low-SES as compared to middle-class populations. The latter might have pre-existing Internet access so that the control treatment affords no material benefit. In the former case, however, the control condition might constitute an important new resource. Researchers may, therefore, attempt to provide information on the relevant resources that participants have prior to study participation.

Other sorts of control or comparison conditions are possible. One might compare eHealth interventions with traditional information resources such as books or pamphlets, or with treatment-as-usual, or with treatment that meets some standard-of-care. These choices should be driven by the inferences that the investigator wishes to make. This latter point is crucial. For example, if the investigator wishes to address the question of whether Internet delivery of information is superior to a book as a delivery system, s/he would try to arrange an experimental test so that control group members receive the same information as experimental participants, but a book(s) is the sole delivery system. However, eHealth investigators often confound a design intentionally. The investigator might decide that it makes no sense to use a computer platform to deliver the same material that would be suitable for a book. Features of computer delivery might afford much more diversity in content, tailoring of content, and so on (e.g., iPhone delivery with visual presentation and audio backup). In this case, the investigator is interested in the pragmatically important question of how well the “best book” does vis-à-vis the “best” computer intervention. This may be an important question, but the book vs. computer comparison per se does not allow the investigator to make strong inferences about whether the special content of the computer system was effective, or whether any differences in outcomes could be attributed to the different delivery systems alone. This design by itself affords little insight into critical intervention features. The point is, as is the case with all research, that the investigator must exercise care in making inferences from results, ensuring that the inferences drawn are permitted by the design.

A researcher may wish to address a very pragmatic question: e.g., does encouragement to use a particular eHealth intervention produce results superior to those produced by patients' fortuitous and normal access to information and resources (access that occurs outside of experimental influence, perhaps via the person's own initiative or via the influence of treatment personnel or family members). Note that in this case, strong inferences cannot be made about effect sizes of types of eHealth intervention components, since exposure to such components may be spread across both the experimental and control conditions.

In considering comparison conditions, it should also be noted that there may be differences in the uniqueness of an eHealth-based intervention as compared to a traditional clinical intervention. For instance, if an RCT involves pharmacotherapy, it is likely that this is a very distinct experience for the participant and one that does not occur outside the experimental context. However, with eHealth interventions, the participant may be exposed to many similar interventions – as Internet resources become more common, and more commonly used, a person may readily access a “therapy” experience that is very similar to the one tested in the RCT. Thus, it would be desirable if the researcher could provide some data on the extent to which participants used similar sorts of eHealth interventions beyond those “prescribed” in the RCT.

Additionally, the researcher might wish to use research findings to make a policy case: e.g., an eHealth intervention should be used in lieu of a traditional or usual care intervention (see discussion of point 2b above regarding objectives). In this case, the traditional therapy would need to be used as a comparison condition. Obviously these different control conditions may exert profound effects on the apparent efficacy of the experimental intervention. Thus, while an eHealth intervention may appear efficacious relative to unstructured Internet use, it may perform less well than a normal face-to-face intervention (22). Global statements about efficacy will be possible only once an intervention has been compared with multiple control conditions.

Researchers may also wish to make inferences about particular aspects of an eHealth intervention. For instance, they may wish to determine if a particular platform characteristic (e.g., high-speed Internet access) or a presentation feature (e.g., design of web pages, use of games) is efficacious. In such cases, the researcher might provide users in control/comparison conditions with the eHealth intervention that is devoid only of the to-be-tested feature or characteristic. Similarly, some eHealth interventions are supplemented by intervention components that are delivered via non-web media (e.g., written handouts, audiotapes: (23). If the researcher wishes to make inferences about the eHealth intervention per se, such non-web components must be provided to control or comparison participants.

The key is that the researcher must clearly identify the particular intervention feature(s) that s/he is interested in testing, and then explain why the control condition is appropriate for supporting strong inferences regarding the targeted feature.

A final word about control conditions: regardless of whether a “treatment-as-usual” control condition is used, the researcher should disclose fully the sorts of interventions that are provided participants in addition to the targeted eHealth intervention. eHealth interventions may not be intended to serve as stand-alone treatments; therefore, it is common for users to receive considerable additional intervention. Thus, if an eHealth intervention for asthma is being tested, the researcher should characterize the sorts of asthma treatments that the participants are receiving in addition to the experimental eHealth interventions. It is conceivable that eHealth interventions will exert very different effects as a function of the intensity or type of adjuvant treatments (e.g., interact with such treatments).

2.4a. Participants (Eligibility criteria for participants, 4a)

The researcher should consider and disclose all exclusion and inclusion criteria used and justify any criteria that are not obvious. It is important to note that de facto exclusion may occur in eHealth research: if recruitment occurs over the Internet, it perforce means that all participants will have access to the Internet. Such de facto selection criteria should be clarified for the reader.

In any area of applied research, a conflict often exists between internal validity and experimental control (lack of bias and accrual of high-quality data) on the one hand and external validity (generalizability to the target population) on the other hand. Many eHealth interventions are designed for use by large segments of the population, which would naturally foster recruitment of “all comers” via Internet recruitment. However, characteristics of a population recruited in this manner might reduce the sensitivity of the Internet evaluation due to heterogeneity of the sample population. Moreover, it might be difficult to get some recruitment information that would allow the researcher to complete the recruitment diagram in Figure 1; e.g., an eHealth study may involve self-determination of eligibility or involve screening and consent only for assessments, and in either case, it might be difficult to determine the proper denominator for enrollment figures (also see (4).

Figure 1
Recruitment Consort Flowchart

To the extent that trial participation is taken to reflect interest in a treatment, or treatment acceptability, it is vital that the researcher disclose all of the steps taken to effect recruitment. There is also some evidence that type of recruitment channel can affect the use or success of the intervention (24, 25). The researcher should disclose steps such as Web-based advertisement, personal or telephone contacts by clinic personnel, and so on. The researcher should present a figure that depicts flow of participants through all stages of a randomized trial (see example in Figure 1; (17). The researcher needs to justify the particular population recruited and the recruitment strategy used, and provide a clear rationale for the approach adopted.

2.4b. Participants (Settings and locations where the data were collected, 4b)

In addition to obvious information (e.g., collaborating clinics), it may be helpful to report how institutional affiliations are displayed to potential participants on eHealth media. Affiliations with prestigious hospitals or universities might affect volunteer rates, use, and reactions with regards to an intervention. Any feature or activity of affiliated institutions that might have affected the eHealth intervention use or effects should be reported.

2.5. Interventions (The interventions for each group with sufficient details to allow replication, including how and when they were actually administered, 5)

2.5a. Revision and updating

Accurate and thorough characterization of an eHealth intervention can be complex and difficult to achieve. Internet interventions are typically designed to be updated or changed over time. This may range from the updating of medical information to the development or improvement of services. In a thorough description, the researcher should disclose which services were routinely updated and how frequently this occurred. In addition, the researcher should disclose major changes or revisions of services that occurred over the course of the study, when and why such revisions occurred, the percentage of users who were exposed to the different service versions, and, if possible, whether such revisions were associated with different use patterns or effects. Finally, a major concern of eHealth users is the accuracy and quality of information provided by the service (26). Authors should provide information on quality assurance methods.

2.5b. Delivery platform

eHealth interventions are multidimensional. One dimension is the hardware platform via which the intervention is delivered. For instance, interventions might be delivered via laptop or desktop computers, or even via mobile platforms such as personal digital assistants or smart phones. Type of delivery platform is critical since it might affect amount and timing of use, the expense of the intervention and functionality. In addition, the type of Internet access is critical since speed of Internet access can affect the complexity of material that can be accessed, the speed with which it is accessed, and ultimately, the acceptability of the intervention (27). Also, the investigators should reveal how participants obtained access to the platform and Internet (e.g., DSL, cable, satellite, and how the platform was obtained: e.g., a new free, dedicated computer might inspire higher use).

2.5c. Presentation strategy

Presentation strategy refers to a set of topics ranging from design of web pages, presence of hyperlinks to other resources, to use of games or simulated virtual reality environments (28). The design and aesthetics of the web pages is an important dimension of an eHealth intervention that can affect use and impact (29). Principles or strategies used in page design might be disclosed, as could basic information such as average amount of text on pages. In order for readers to have a clear sense of the nature of such features, it would be desirable for researchers to ensure that readers could examine samples or portions of eHealth interventions, perhaps via web addresses made available in the published article or a screen shot. In any case, researchers should strive to ensure that readers have available to them at least a subset of pages comprised by an evaluated eHealth intervention.

2.5d. Content

Researchers should certainly disclose information about the type of content comprised by eHealth interventions. Content information should include the types of services that were provided and the targets or intended proximal or mediational effects of the services (e.g., provision of accurate information, training of new skills such as social skills or decision-making skills, or provision of social support). In addition, the researcher should reveal characteristics of the content presentation such as whether dates were included on the materials to give it a sense of currency. Finally, the researcher should inform the reader how the content was developed; i.e., the strategy used to develop and test the intervention content (e.g., focus groups, peer review) and the qualifications of the content developers.

2.5e. Intervention dose

eHealth interventions typically allow more individualization of the user experience and intensity of use – the developer may allow tailoring to individuals, and participants themselves can determine use patterns (there is typically no therapist present to guide use). With most traditional (non-eHealth) interventions, there is considerable formal structure to the intervention – a set number of sessions lasting so many minutes and occurring on a particular time schedule. However, with eHealth interventions the patient often can access the intervention 7 days per week, 24 hours per day, for many months – or never. Presumably, differences in use patterns are automatically collected in most eHealth trials. The upshot here is that evidence of intervention exposure or use, typically requires more extensive reporting in eHealth vs. other sorts of intervention research. For instance, while “session” length may be predetermined in many traditional RCTs, this factor may vary greatly in eHealth research. Of course, reporting such information is complex in an eHealth RCT since one must decide such things as how to define a “session” or use occasion. So, the researcher should not only supply the relevant use data, but also the definitions that guided the collection of data.

2.5f. Instructional strategy

Depending upon the hypotheses addressed by the researchers, it may be important to provide even finer grained detail about the nature of the eHealth intervention. For instance, skill training might occur via different specific instructional methods: e.g., case-based learning, self-assessment questions, or simulation (30, 31). This is a relatively neglected characterization of eHealth interventions and there is evidence that instructional strategy can significantly influence outcomes and distinguish one eHealth intervention from another in important ways (32, 33).

2.5g. Adjuvant training, prompts, and interveners

In most interventions, therapists are trained to deliver the intervention, but any training participants receive is part of the intervention per se (e.g., depressed individuals are trained to change the way they think about stressful events). However, in eHealth interventions, there may be participant training that falls outside intervention use. For instance, participants may need training in use of a computer, logging-on to the Internet, or even specialized training in some features of a program. Moreover, such training may be given, or be available, at any point during the intervention period. Further, training may be accomplished via phone, in-person, interactive CD-ROM, or in an online demonstration. Researchers should disclose all such training so the reader can determine how much support is needed to utilize the tested intervention.

In addition, researchers may use prompts (e.g., letters, emails, or phone calls) to encourage use of the intervention. The types and frequency of such prompts need to be reported to permit replication and to help gauge whether obtained use rates reflect spontaneous vs. prompted use.

Finally, some interventions link participants with experts or counselors. In this case, the training and professional status of the human adjuvant should be disclosed since the success of the intervention may depend as much on the training and expertise of the human as on the properties of the programming (34).

2.6. Outcomes (Completely defined pre-specified primary and secondary outcome measures, including how and when they were assessed, 6a)

While medical interventions may have single pre-eminent outcome (e.g., reduced hypertension, lowered lipid levels) that are of pre-eminent importance, many eHealth interventions do not. For instance, an eHealth intervention for breast cancer patients (35) might have as its goals: improvement in quality of life, reduction of negative affect or depression, greater knowledge (and confidence in knowledge) regarding the disease and treatment options, improved decision-making on the part of the patient, reduced health-care expenditures and utilization, fewer somatic symptoms, greater satisfaction with health care, more effective utilization of health care resources (e.g., more efficient visits), and so on. This multiplicity of outcomes is probably the norm rather than the exception with most eHealth interventions. Therefore, it is incumbent that the researcher take steps to reduce the threats posed by multiple outcomes and that these steps be accurately reported. The steps may include: 1). Report clearly the outcomes that were identified, a priori, as primary and secondary. This could serve as a basis for deciding which tests should be conducted with alpha protection and would also alert the reader to the tests that were most strongly supported by theory or that have greatest clinical importance (e.g., (36). 2). Analyze latent variables that capture reliable covariance in the outcomes and reduce the number of tests conducted. 3). Disclose tests conducted, but whose results are not necessarily reported, so that the reader has a sense of the population of outcomes that was analyzed. Of course the investigator should do his or her best to prioritize outcomes to identify the smallest number of highly meaningful ones.

In addition to investigating theory-based outcomes, the researcher may wish to examine more peripheral dimensions of the intervention. Investigators may want to ask questions about difficulties in gaining Internet access, ease of use, speed of information access, and so forth. It is possible that the different effects of contrasted eHealth interventions have more to do with such peripheral features than with hypothesized content differences. Finally, investigators may wish to gather subjective ratings of factors such as general satisfaction with the eHealth application, interface, frustration with program bugs, server downtime, or impatience with the system. Such might be especially useful when standardized content is used but presented via different delivery systems.

In closing, while some methodologists recommend having only a single primary outcome (18), there may be circumstances where the eHealth researcher cannot legitimately prioritize outcomes that concisely. In these cases, it is especially incumbent upon him/her to defend the outcomes selected and discuss the relation of these outcomes with the initial experimental hypotheses.

2.7. Blinding (If done, who was blinded after assignment to interventions [for example, participants, care providers, those assessing outcomes, and how; 11a)

Blinding issues are especially relevant to eHealth RCTs. For instance, researchers should be specific about how those who are doing data analyses, and those who are scoring assessments or collecting use-data (data on intervention use), are shielded from access to information about group assignment. For instance, if data are coded and gathered by user name or password, it is likely that some research staff will be able to link such identifiers with service access or use (e.g., users of discussion groups). In addition, it may be that clinicians (e.g., an oncologic nurse) may be able to determine whether a patient is receiving an experimental intervention. For instance, the patient may reveal the nature of his/her treatment in clinic visits, or the intervention may involve alerts or communications to the clinician. It is possible that such information could influence the treatment that a patient receives; the possibility of such a lack of blinding with regards to adjuvant health care systems or personnel should be noted. The collection of information on the patient's health care treatment would reveal whether the members of different experimental conditions actually received different treatments (this could reflect a failure of blinding or a legitimate downstream effect of the eHealth intervention.) Finally, while it is feasible to blind both patients and researchers to medication status (active vs. placebo), it is virtually impossible to blind participants to the sort of eHealth intervention they are receiving.

3. Results

3.1. Participant flow (For each group, the numbers of participants who were randomly assigned, received intended treatment, and were analyzed for the primary outcome, 13a)

It is important to determine the proportion of a presenting patient population that finds an intervention acceptable and is exposed to the intervention. Figure 1 is adapted from the CONSORT recommendations and depicts the data that should, if possible, be supplied regarding an eHealth RCT. These figures may arise from somewhat different sources in eHealth trials than in traditional RCTs. For instance, if open recruitment is conducted (e.g., via the Internet), it is quite difficult to determine how many individuals were initially exposed to recruitment information about the intervention. (While visits to a recruitment website can be tallied, this could reflect multiple visits by the same individuals.) This would make it difficult to determine the proportion of individuals who enter the study once it is presented to them. It is also important to know what screening occurred prior to subjects having the study presented to them – were there any factors that culled potential participants prior to study entry or that limited the generalizability of the findings? With closed recruitment it is possible to compare participants to nonparticipants in the same closed system (HMO members: (4), but even here it is sometimes difficult to know which members were exposed to the recruitment material.

The CONSORT statement also calls for reporting of treatment completion and attrition across all stages of the intervention (see Figure 1). Therefore, it may be important to decide if there is any level or intensity of intervention use or exposure that constitutes treatment completion. This determination is more difficult than with most traditional interventions where a standard course of therapy is prescribed (e.g., medication use for 8 weeks). Investigators must decide whether treatment completion is a meaningful concept in eHealth RCTs, and whether it should be instantiated in terms of number of services used, total time of use (e.g., number of minutes), or use over some number of weeks (e.g., (37). Measurement of eHealth use can be quite complex (4), which can compromise both hypotheses and tests of use-outcome relations. If no explicit standard of treatment completion is adopted, investigators may have little basis for defining attrition. Attrition could always be defined on the basis of formal withdrawal from the treatment program – but such withdrawals are often rare relative to those who simply cease participation.

If it is difficult to define either a dose of treatment that constitutes treatment completion or to clearly define attrition, it may be best for researchers to merely provide extensive documentation of treatment exposure across the group of participating individuals and be clear about exactly how use or engagement was measured: page views, viewing durations, program visits, measures of practice, composite measures, and any cut-offs designed to improve measure validity (4).

3.2. Recruitment (Dates defining the periods of recruitment and follow-up;14a)

Investigators should provide dates for critical events in an RCT trial (17). These would include dates for the trial launch, the pattern of recruitment over time (e.g., a breakdown of number of participants recruited in each of the study duration quartiles), the completion of trial recruitment, the length of the intervention, and how long the time period is for following up with study participants. As noted earlier, eHealth RCTs are often conducted in a world of flux (e.g., rapid technological change). Therefore, in addition to dates, the investigators could indicate where critical secular events fell during the study. These critical events might occur in at least three domains: 1). Changes in the treatment or assessment of the health condition that is the target of the intervention (e.g., FDA approval of a new medication, new clinic procedures), 2). Significant changes in Internet resources related to the target condition (introduction of a new widely available website), and 3). Changes in computer hardware or Internet delivery resources such as speed (e.g., slower speed dial-up vs. high-speed DSL or cable) or type of access (e.g., wired vs. wireless). The investigator could evaluate whether such events were related to outcomes. Such information will help the reader evaluate the context in which the eHealth intervention was tested.

3.3. Numbers analyzed (For each group, number of participants [denominator] included in each analysis and whether the analysis was by originally assigned groups, 16)

The problem of what to use as a denominator in outcome analyses is a crucial one in eHealth trials. For instance, while thousands of individuals may be exposed to an eHealth intervention, this may provide a very imprecise estimate of those for whom the intervention is relevant (e.g., targeted disease sufferers vs. the curious), those who actually use the intervention, and those who can generate meaningful outcome data. Because users or participants have such control over exposure to the intervention, investigators may be tempted to analyze data only from those individuals who were actually exposed to (i.e., used) the intervention. It is important that investigators realize that if they engage in such a practice, they can no longer describe their trial as adhering to the intent-to-treat principle (18). Perhaps the best strategy is to follow the general CONSORT principle of reporting N's across a range of study participation thresholds. That is, to the extent possible, investigators should report the number of individuals exposed to the intervention invitation, the number who formally consented, and the number who used the intervention beyond a limited number of strategically selected use levels (determined a priori), but the intent-to-treat principle should be used in primary analyses.

Non-response bias is an especially critical issue in eHealth research since it may occur for many reasons unrelated to outcome status. Non-response during follow-up in many clinical trials may be attributed to a poor outcome; e.g., in smoking cessation research (38, 39). However, in eHealth research, non-response may be due to such factors as spam filters, email address changes, or because there is little personal contact with intervention staff (40); participants may not develop loyalty to the intervention or research program. Finally, participants of eHealth intervention studies may misinterpret a follow-up query as intervention-related assessment, and not respond because they feel that they are no longer in need of treatment. Thus, meaningful levels of nonresponse should encourage investigators to report outcomes in multiple ways (e.g., survival methods with proper censoring, data imputation procedures, (41) as appropriate (18).

3.4. Harms (important harms of unintended effects in each group, 19)

Researchers need to disclose steps taken to ensure that all participants receive treatment that meets standards of care, and the steps taken to reduce iatrogenic effects of treatment. With respect to the latter point, some eHealth studies provide access to the Internet as a comparison treatment, or as a component of an active intervention. However, it is well known that the Internet can be a source of inaccurate, and possibly harmful information (e.g., (42, 43). In theory, individuals could come in contact with harmful information because of their research participation.

One potential source of harm (that may be particularly relevant in eHealth research) is the disclosure of personal health information or other private information through intervention use: perhaps in a discussion group or through writing exercises. The apparent impersonal nature of computer interaction can induce some individuals to reveal more information than they would reveal in face-to-face encounters (44). In addition, a research nonparticipant might access a participant's resource data by obtaining the participant's password or because the participant has failed to log-out of the resource. Participants' personally identifiable data and information about their use of the eHealth system could also be accessed by many sophisticated (and potentially malicious) Internet based tools such as spyware, cookies, bug exploitation, remote3 searching, data logging and deep packet inspection technology. http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=1071667. Additionally some spyware can interfere with the operation of the computer. Researchers should reveal steps taken to prevent such harms, whether or not such harms occurred, and the systems in place to detect such harms.

In cases of eHealth research where investigators may not have face-to-face contact with participants, steps should be taken to determine whether a potential participant is a minor. Also, researchers should consider how they will solicit and respond to questions about research participation during the consent process. These processes can be more challenging in eHealth research, especially where recruitment is via the Internet and the investigator has little or no face-to-face contact with participants (see (45) for discussion of these issues and potential strategies). Face-to-face contact may affect the amount or accuracy of data that may be collected (e.g., collection of vital signs; (46), participant reactivity to assessment burden, and decisions to participate in the research. Therefore, the investigator should report the nature of any face-to-face elements of the research procedures so the reader may evaluate their potential impact.

Investigators should report adverse events that occurred in the trial and attempt to determine whether these events can be attributed to either adjuvant medical treatments or to the eHealth intervention. This distinction may not be straightforward. For instance, the likelihood of suffering an adverse event due to the medical adjuvant treatment might be affected by effects of an eHealth intervention: e.g., less reliance on the clinician, confusion because of too much or conflicting information. It is necessary to assess possible untoward effects if the researcher has a hope of arriving at a meaningful appraisal of the harm/benefit ratio of such an intervention. This is especially important for eHealth interventions given credible routes via which harm might be done: viz. replacing/dislodging other health care, reducing naturally occurring (nonintervention) social support (47); although cf. (48), Internet overuse (e.g., displaces sleep or reduces physical activity), encountering erroneous or misleading information over the Internet, lack of understanding of complex health care information, and so on. The researcher should report any measures that were used to tap potential negative outcomes of treatment so that the reader can gauge the harm/benefit ratio.

The researcher should disclose any steps that s/he has taken to reduce the likelihood or detection of such harm (e.g., education and training in site evaluation, availability of a hotline).

4. Discussion and conclusion

4.1. Discussion

It should be noted that the CONSORT criteria (as well developed as they are) cover only a subset of issues related to intervention evaluation. Russell Glasgow pointed out that demonstrating whether an intervention is efficacious may be insufficient to determine its real world potential. Issues such as translatability and public health impact are very important but rarely the focus of eHealth evaluations. Glasgow and colleagues' framework, RE-AIM, (49) suggests that eHealth evaluations should assess: Reach into the target population, Efficacy, Adoption by target settings, consistency of Implementation and Maintenance of effects in individuals and organizations. They use the example of an intervention that has large impact in terms of reach and efficacy in one school, but that cannot be widely adopted because most schools do not have the resources to adopt, implement and maintain the system.

It is also the case that reporting CONSORT information is only one step that investigators can take to communicate important clinical trial information. Investigators should also consider registering eHealth trials, which would accomplish important aspects of disclosure (e.g., (17). It would reveal primary and secondary funding sources, a responsible person to contact regarding more information about the trial, research ethics review information, relevant study dates, and so on (16). The National Library of Medicine within the NIH operates a registry that is free, highly accessible, open to all registrants and verifies registered information (go to: www.clinicaltrials.gov). This would help researchers learn results of studies that are completed but not published (e.g., due to negative findings).

4.2. Conclusion

The recommendations made in this paper encourage evaluators to consider carefully their evaluation design and to be explicit in describing what they did and why. One key issue faced is how to package this very detailed description in a way that still allows the whole publication to fit within the increasingly more stringent 2,000-3,000 word limitations of many journals. Many aspects of our recommendations are nonobligatory and will not be relevant to some eHealth studies. This will reduce the reporting burden. In addition, authors may make some of the information available via addenda that can accompany the manuscript when it is submitted for review. This information could also be made available to readers via a website sponsored either by the authors or by the journal. Even if it is not possible to include all of this information in the article or addenda, it is worthwhile for scientists to consider these issues as they apply to eHealth research.

4.3. Practice implications

Increased attention to these unique characteristics of eHealth research and the recommendations contained in this paper have the potential to enhance the public health and scientific value of eHealth research.

Acknowledgment

Timothy B. Baker has served as an investigator on research projects sponsored by pharmaceutical companies including Pfizer, Glaxo Wellcome, Sanofi, and Nabi. No other author had potential conflicts to disclose. The study was funded by P50 CA095817-05 from the National Cancer Institute.

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

1Certainly, high quality research exists on eHealth interventions (e.g., 11). However, given the paucity of formal randomized trials of eHealth interventions and the fact that many have clear limitations (e.g., sample size; 6), the growth in experimental evaluations of eHealth interventions has not kept pace with the growth in their use.

References

1. Altman DG, Schulz KF, Moher D, et al. The revised CONSORT statement for reporting randomized trials: Explanation and elaboration. Ann Intern Med. 2001;134:663–94. [PubMed]
2. Begg C, Cho M, Eastwood S, et al. Improving the quality of reporting of randomized controlled trials. The CONSORT statement. J Amer Med Assoc. 1996;276:637–9. [PubMed]
3. Demographics of internet users. 2008. Accessed July 21, 2009, at http://www.pewinternet.org/Static-Pages/Trend-Data/Whos-Online.aspx.
4. Danaher BG, Seeley JR. Methodological issues in research on web-based behavioral interventions. Ann Behav Med. 2009;38:28–39. [PMC free article] [PubMed]
5. Griesam N, Meyer C, Haug S, et al. Diversity and accessibility of smoking cessation aids in the internet. Gesundheitswesen. 2008;70:372–6. [PubMed]
6. Klemm P, Bunnell D, Cullen M, Soneji R, Gibbons P, Holecek A. Online cancer support groups: a review of the research literature. Comput Inform Nurs. 2003;21:136–42. [PubMed]
7. Kreps GL, Neuhauser L. New directions in eHealth communication: opportunities and challenges. Patient Educ Couns. 2010;78:329–36. [PubMed]
8. Norman GJ, Zabinski MF, Adams MA, Rosenberg DE, Yaroch AL, Atienza AA. A review of eHealth interventions for physical activity and dietary behavior change. Am J Prev Med. 2007;33:336–45. [PMC free article] [PubMed]
9. Strecher V. Interactive health communications for cancer prevention and control. In: Miller SM, Bowen DJ, Croyle RT, Rowland JH, editors. Handbook of cancer control and behavioral science: A resource for researchers, practitioners, and policymakers. American Psychological Association; Washington, D.C.: 2009. pp. 547–58.
10. Murray E, Burns J, See TS, Lai R, Nazareth I. Interactive Health Communication Applications for people with chronic disease. Cochrane Database Syst Rev. 2005;(4):CD004274. [PubMed]
11. Strecher V. Internet methods for delivering behavioral and health-related interventions (eHealth) Annu Rev Clin Psychol. 2007;3:53–76. [PubMed]
12. Hesse BW, Moser RP, Rutten LJ, Kreps GL. The health information national trends survey: research from the baseline. J Health Commun. 2006;11(Suppl 1):vii–xvi. [PubMed]
13. Kane RL, Wang J, Garrard J. Reporting in randomized clinical trials improved after adoption of the CONSORT statement. J Clin Epidemiol. 2007;60:241–9. [PubMed]
14. Moher D, Schulz KF, Altman DG. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. Ann Intern Med. 2001;134:657–62. [PubMed]
15. Coyne JC, Lepore SJ, Palmer SC. Efficacy of psychosocial interventions in cancer care: evidence is weaker than it first looks. Ann Behav Med. 2006;32:104–10. [PubMed]
16. Elliott TR. Registering randomized clinical trials and the case for CONSORT. Exp Clin Psychopharmacol. 2007;15:511–8. [PMC free article] [PubMed]
17. Schulz KF, Altman DG, Moher D. CONSORT 2010 statement: Updated guidelines for reporting parallel group randomized trials. Ann Intern Med. 2010;152:726–32. [PubMed]
18. Moher D, Hopewell S, Schulz KF, et al. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. Brit Med J. 2010;340:c869. [PMC free article] [PubMed]
19. Gustafson DH, Hawkins R, McTavish F, et al. Internet-based interactive support for cancer patients: Are integrated systems better? J Commun. 2008;58:238–57. [PMC free article] [PubMed]
20. Gustafson DH, Hawkins RP, Boberg EW, et al. CHESS: 10 years of research and development in consumer health informatics for broad populations, including the underserved. Int J Med Inform. 2002;65:169–77. [PubMed]
21. Miller GA, Chapman JP. Misunderstanding analysis of covariance. J Abnorm Psychol. 2001;110:40–8. [PubMed]
22. Newman MG, Kenardy J, Herman S, Taylor CB. Comparison of palmtop-computer-assisted brief cognitive-behavioral treatment to cognitive-behavioral treatment for panic disorder. J Consult Clin Psychol. 1997;65:178–83. [PubMed]
23. White J, Jones R, McGarry E. Cognitive behavioral computer therapy for anxiety disorders: A pilot study. J Ment Health. 2000;9:505–16.
24. Glasgow RE, Nelson CC, Kearney KA, et al. Reach, engagement, and retention in an Internet-based weight loss program in a multi-site randomized controlled trial. J Med Internet Res. 2007;9:e11. [PMC free article] [PubMed]
25. Thompson D, Baranowski T, Cullen K, et al. Food, fun, and fitness internet program for girls: pilot evaluation of an e-Health youth obesity prevention program examining predictors of obesity. Prev Med. 2008;47:494–7. [PubMed]
26. Hesse BW, Arora NK, Burke Beckjord E, Finney Rutten LJ. Information support for cancer survivors. Cancer. 2008;112(11 Suppl):2529–40. [PubMed]
27. Shaw BR, Gustafson DH, Hawkins R, et al. How underserved breast cancer patients use and benefit from eHealth programs: Implications for closing the digital divide. Am Behav Scientist. 2006;49:1–12.
28. Cook DA. Learning and cognitive styles in web-based learning: theory, evidence, and application. Acad Med. 2005;80:266–78. [PubMed]
29. Experts vs. online consumers: A comparative credibility study of health and finance Web sites. 2002. Accessed July 21, 2009, at http://www.consumerwebwatch.org/pdfs/expert-vsonline-consumers.pdf.
30. Alur P, Fatima K, Joseph R. Medical teaching websites: do they reflect the learning paradigm? Med Teach. 2002;24:422–4. [PubMed]
31. Merrill MD. Instructional strategies and learning styles: Which takes precedence? In: Reiser R, Dempsey J, editors. Trends and issues in instructional design and technology. Prentice Hall; Columbus, OH: 2002. pp. 99–106.
32. Cook DA, Dupras DM. A practical guide to developing effective web-based learning. J Gen Intern Med. 2004;19:698–707. [PMC free article] [PubMed]
33. Shaw BR, Han JY, Baker T, et al. How women with breast cancer learn using interactive cancer communication systems. Health Educ Res. 2007;22:108–19. [PubMed]
34. Davidson KW, Goldstein M, Kaplan RM, et al. Evidence-based behavioral medicine: what is it and how do we achieve it? Ann Behav Med. 2003;26:161–71. [PubMed]
35. Gustafson DH, Hawkins R, Pingree S, et al. Effect of computer support on younger women with breast cancer. J Gen Intern Med. 2001;16:435–45. [PMC free article] [PubMed]
36. Hosmer DW, Lemeshow S. Applied Logistic Regression, Second Edition. 2nd ed. John Wiley & Sons, Inc.; New York: 2000.
37. Han JY, Hawkins RP, Shaw B, Pingree S, McTavish F, Gustafson DH. Unraveling uses and effects of an interactive health communication system. Journal of Broadcasting and Electronic Media. 2009;53:112–33. [PMC free article] [PubMed]
38. Hall SM, Delucchi KL, Velicer WF, et al. Statistical analysis of randomized trials in tobacco treatment: longitudinal designs with dichotomous outcome. Nicotine Tob Res. 2001;3:193–202. [PubMed]
39. Oka RK, Katapodi MC, Lim JW, Bacchetti P, Froelicher ES. Quantifying smoking cessation outcomes: from The Women's Initiative for Nonsmoking Study (X): methodological implications. Nurs Res. 2006;55:292–7. [PubMed]
40. Glasgow RE, Mullooly JP, Vogt TM, et al. Biochemical validation of smoking status: pros, cons, and data from four low-intensity intervention trials. Addict Behav. 1993;18:511–27. [PubMed]
41. Allison PD. Missing data techniques for structural equation modeling. J Abnorm Psychol. 2003;112:545–57. [PubMed]
42. Crocco AG, Villasis-Keever M, Jadad AR. Analysis of cases of harm associated with use of health information on the internet. J Amer Med Assoc. 2002;287:2869–71. [PubMed]
43. Meric F, Bernstam EV, Mirza NQ, et al. Breast cancer on the world wide web: cross sectional survey of quality of information and popularity of websites. Brit Med J. 2002;324:577–81. [PMC free article] [PubMed]
44. Rosen CS, Henson BR, Finney JW, Moos RH. Consistency of self-administered and interview-based Addiction Severity Index composite scores. Addiction. 2000;95:419–25. [PubMed]
45. Mathy RM, Kerr RL, Haydin BM. Methodological rigor and ethical consideration in Internet-mediated research. Psychotherapy: Theory, Research, Practice, Training. 2003;40:77–85.
46. Glasgow RE, Lichtenstein E, Marcus AC. Why don't we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness transition. Am J Public Health. 2003;93:1261–7. [PMC free article] [PubMed]
47. Kraut R, Kiesler S, Mukhopadhyay T, Scherlis W, Patterson M. Social impact of the internet: What does it mean? Communications of the ACM. 1998;41:21–2.
48. Kraut R, Kiesler S, Boneva B, Cummings J, Helgeson V, Crawford A. Internet paradox revisited. Journal of Social Issues. 2002;58:49–74.
49. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;89:1322–7. [PMC free article] [PubMed]
PubReader format: click here to try

Formats:

Related citations in PubMed

See reviews...See all...

Cited by other articles in PMC

See all...

Links

  • PubMed
    PubMed
    PubMed citations for these articles