Logo of jmirEditorial BoardMembershipSubmitCurrent IssueArchiveReviewSearchAboutJMIR PublicationsJMIR
J Med Internet Res. 2011 Apr-Jun; 13(2): e32.
Published online Apr 14, 2011. doi:  10.2196/jmir.1624
PMCID: PMC3221377

Effectiveness of a Web-based Intervention Aimed at Healthy Dietary and Physical Activity Behavior: A Randomized Controlled Trial About Users and Usage

Monitoring Editor: Gunther Eysenbach
Reviewed by Miriam Wanner and Marieke Verheijden

Abstract

Background

Recent studies have shown the potential of Web-based interventions for changing dietary and physical activity (PA) behavior. However, the pathways of these changes are not clear. In addition, nonusage poses a threat to these interventions. Little is known of characteristics of participants that predict usage.

Objective

In this study we investigated the users and effect of the Healthy Weight Assistant (HWA), a Web-based intervention aimed at healthy dietary and PA behavior. We investigated the value of a proposed framework (including social and economic factors, condition-related factors, patient-related factors, reasons for use, and satisfaction) to predict which participants are users and which participants are nonusers. Additionally, we investigated the effectiveness of the HWA on the primary outcomes, self-reported dietary and physical activity behavior.

Methods

Our design was a two-armed randomized controlled trial that compared the HWA with a waiting list control condition. A total of 150 participants were allocated to the waiting list group, and 147 participants were allocated to the intervention group. Online questionnaires were filled out before the intervention period started and after the intervention period of 12 weeks. After the intervention period, respondents in the waiting list group could use the intervention. Objective usage data was obtained from the application itself.

Results

In the intervention group, 64% (81/147) of respondents used the HWA at least once and were categorized as “users.” Of these, 49% (40/81) used the application only once. Increased age and not having a chronic condition increased the odds of having used the HWA (age: beta = 0.04, P = .02; chronic condition: beta = 2.24, P = .003). Within the intervention group, users scored better on dietary behavior and on knowledge about healthy behavior than nonusers (self-reported diet: χ2 2 = 8.4, P = .02; knowledge: F1,125 = 4.194, P = .04). Furthermore, users underestimated their behavior more often than nonusers, and nonusers overestimated their behavior more often than users (insight into dietary behavior: χ2 2 = 8.2, P = .02). Intention-to-treat analyses showed no meaningful significant effects of the intervention. Exploratory analyses of differences between pretest and posttest scores of users, nonusers, and the control group showed that on dietary behavior only the nonusers significantly improved (effect size r = −.23, P = .03), while on physical activity behavior only the users significantly improved (effect size r = −.17, P = .03).

Conclusions

Respondents did not use the application as intended. From the proposed framework, a social and economic factor (age) and a condition-related factor (chronic condition) predicted usage. Moreover, users were healthier and more knowledgeable about healthy behavior than nonusers. We found no apparent effects of the intervention, although exploratory analyses showed that choosing to use or not to use the intervention led to different outcomes. Combined with the differences between groups at baseline, this seems to imply that these groups are truly different and should be treated as separate entities.

Trial registration

Trial ID number: ISRCTN42687923; http://www.controlled-trials.com/ISRCTN42687923/ (Archived by WebCite at http://www.webcitation.org/5xnGmvQ9Y)

Keywords: Randomized controlled trial, usage, eHealth, intervention, attrition, Internet, adherence, retention

Introduction

The increasing prevalence of overweight is a problem in modern society. It is closely related to a number of chronic conditions such as type 2 diabetes mellitus and places a great burden on the health care system. Losing weight and especially preventing weight regain is challenging. It might be more cost-efficient to prevent people from becoming overweight by focusing on healthy dietary and physical activity (PA) behavior [1-3]. To achieve this goal, interventions aimed at the general public are needed that must not only inform people about the risks of unhealthy dietary and physical activity habits but must also stimulate people to adopt healthier behaviors related to diet and physical activity [2,4]. Previous research has shown that tailored and interactive interventions can achieve this goal [2,4-7]. The Internet provides an opportunity for these interventions to reach a broad population. Besides, by using a Web-based application, the content of the intervention can be tailored to the users, and the intensity can be varied according to the needs and wishes of these users [8-9]. Research has already shown the potential of these applications for the achievement of weight loss and weight management [6,10-14]. However, most studies are focused on applications aimed at treatment or secondary prevention. Many questions remain about the users and the effectiveness of Web-based applications for the prevention of health problems by stimulating healthy behaviors.

The problem of attrition [15] poses a threat to most eHealth interventions but might pose an even bigger threat to Web-based interventions for prevention, considering that people who do not experience an urgent health problem might be less internally motivated to change their behavior [16]. Until recently, the characteristics of the users and nonusers of Web-based applications have gained only very limited attention [17-19]. It is important to know who the users of these interventions are. This knowledge helps us identify important factors in the dissemination of these interventions and the characteristics of intended users who are not reached [20]. Moreover, recent studies indicate that people react differently to motivational and persuasive strategies, which might make the need for examining user characteristics even more essential [21]. A recent review by Christensen and colleagues [22] emphasized the need for a theoretical framework to increase our understanding of attrition. They proposed using the framework adopted by the World Health Organization (WHO) [16] (ie, five dimensions of adherence: health system factors, social and economical factors, therapy-related factors, condition-related factors, and patient-related factors) and mention the possible potential of behavior theories. Furthermore, research into the reasons for use of Web-based eHealth applications can give us valuable information on what the users hope to accomplish and how the application can assist them. In addition, usability and satisfaction with an application can play an important role in the extent to which such applications are ultimately used [15,23].

We incorporated the WHO framework and behavior theories in a study of use of the Healthy Weight Assistant (HWA), a Web-based lifestyle intervention. We considered the influence of social and economic factors (demographics), condition-related factors (ie, general practitioner [GP] visits, having a chronic condition, and self-reported and self-rated dietary and PA behavior), patient-related factors or constructs identified by behavior change theories (ie, knowledge, attitude, and self-efficacy) [24-25], and reasons for use and satisfaction with the intervention.

Additionally in this study, we assessed the effectiveness of the intervention using self-reported dietary and PA behavior as primary outcome measures because the intervention was aimed at improving health behavior. We included secondary outcome measures that are known determinants of behavior change. We also chose to include measures of knowledge, attitude, and self-efficacy [24-25]. Self-rated behavior and insight into behavior were included as secondary outcome measures because one of the goals of the intervention was to improve insight into one’s own behavior.

Consequently, our research questions were: What characteristics of participants are related to the use of the HWA intervention? What effects does the HWA intervention have on the primary and secondary outcome measures?

Methods

Recruitment and Design

Participants were recruited through advertisements about an online lifestyle intervention in local newspapers, supermarkets, and on health-related websites. Permission of an ethics review board for the study was not required because, according to the Dutch law, nonintrusive interventions conducted with healthy adults do not require approval from an ethics board. In total, 297 respondents expressed interest in using an online lifestyle intervention and satisfied our inclusion criteria (body mass index [BMI] 18.5 - 28.0 kg/m2, Dutch-speaking). The inclusion criterion for BMI was chosen to reflect the target group of the intervention under investigation. The sample used in this study was a self-selected convenience sample. Enrollment took place beginning November 1, 2008, and ending December 31, 2008. All participants were randomly assigned to either the Web-based lifestyle coach or a waiting list. A total of 150 participants were allocated to the waiting list group, and 147 participants were allocated to the intervention group. Participants filled out online questionnaires before the 12-week intervention period started and again after the intervention period ended. The posttest questionnaire was available for all respondents for a period of 3 weeks beginning February 27 and ending April 16. After the intervention period, respondents in the waiting list group could use the intervention. The flowchart of the study can be found in Figure 1.

Figure 1
Flowchart of the study

Randomization

Randomization took place 1 week before the start of the intervention period. We used block randomization with blocks of 4 participants, stratified on age, sex, and education. The randomization scheme was created by a computer application and carried out by a member of the research team. Participants who filled out demographic information were randomized. Only respondents who completed the pretest questionnaire were included; therefore, 28 respondents were excluded. Participants were not blinded to randomization outcome but received an email with information on when and how they were able to access the Healthy Weight Assistant (HWA) after filling out the pretest questionnaire.

Intervention

The Healthy Weight Assistant (HWA) is a Web-based lifestyle intervention developed by the Netherlands Nutrition Centre, which is a government-funded organization focusing on increasing the knowledge of consumers about the quality of food and encouraging consumers to eat healthily and safely. The goal of the HWA is to support people with a healthy weight and people who are slightly overweight (ie, BMI 18.5-28.0 kg/m2) to maintain and achieve a healthy weight. The aim is not to achieve a given weight loss, but to support the achievement of healthy dietary and PA behavior. Therefore, the focus was broader than only energy balance-related behavior. The target group was selected by the Netherlands Nutrition Centre according to their BMI classification. The theoretical basis for behavior change via the HWA is the transtheoretical model [26], which entails that the participants are addressed according to the stage of change in which they find themselves when starting the application. The researchers were not the leading party in the design of the HWA but have done earlier research on the application. This previous study employed user-centered evaluation methods and has led to slight alterations in the design of the application in order to increase users’ motivation to keep using the HWA and their motivation to change behavior [27].

The HWA consists of 4 steps, which are marked in the application by a “to-do list” and tabs in the “diary” (Figure 2). When users enter the program for the first time, they start by assessing their baseline status. In this step, users answer questions about their body weight, dietary behavior, physical activity behavior, and emotions concerning these behaviors. This results in tailored advice that can be applied in the next steps of the application. The second step is motivation. Users are asked about their motivation to change behavior, and the application assists them in making these motivations clear to themselves, thereby also focusing on clarifying their emotions related to behavior. The third step is called difficult moments. Users are encouraged to reflect on their difficult moments (ie, moments at which it is tempting to engage in unhealthy behavior) and to provide solutions for these moments. The HWA coaches the user throughout this step by giving automated tailored feedback based on input of the users. The final step is goal setting and monitoring achievement of goals. Users are coached to set useful and realistic goals and can opt to receive a weekly email reminder on these goals. Additionally, users can give feedback on the achievement of their own goals and access an overview of previous goals. The news section of the HWA is regularly updated, and when users exit the application, random hints are displayed. Other content is static. The HWA is designed to be used at regular intervals. The intended use is one or multiple visits within a short period of time to complete the first 3 steps. For the last step, the intended use is once a week to once a fortnight over a longer period of time. For the research period, the HWA was only available to the participants. After this period, the application was made openly accessible through a website.

Figure 2
The Healthy Weight Assistant

Waiting List

We made use of a waiting list control group. Participants randomized in this group received an email newsletter every 3 weeks, but no access to the HWA during the intervention period. The newsletter contained general information about the study and about the University of Twente. Furthermore, it contained leisure tips, but it contained no information on healthy lifestyle. After the intervention period, participants in the waiting list group received access to the HWA. Participants in the intervention group also received the newsletter every 3 weeks.

Research Instruments

Online questionnaires were used to assess pretest and posttest values. Education was self-reported and recoded into the following three categories: low (primary and lower vocational education), moderate (secondary and middle vocational education), and high (higher vocational and university education). BMI (kg/m2) was calculated using self-reported weight and length. Dietary behavior was measured using a 14-item self-report questionnaire of the Netherlands Nutrition Centre, based on the Netherlands classification model [28]. This questionnaire has not been validated but was used because of the applicability to the standards used by the Netherlands Nutrition Centre [29]. These standards are based on a report of the Health Council of the Netherlands, which is the basis of nutritional education in the Netherlands [30]. This questionnaire classifies respondents as unhealthy (not complying to the standards on all aspects), improvable (complying with the standards on some aspects), and healthy (complying with the standards on all aspects). This classification entails that respondents in the healthy category have limited room for improvement because they already comply with all of the standards. We have included a translation of this questionnaire in Multimedia Appendix 1. Physical activity behavior was measured according to the Dutch Standard for Healthy Physical Activity, using a validated 4-item self-report questionnaire [31]. This questionnaire classifies respondents into two categories, unhealthy (not complying with the standards) and healthy (complying with the standards). Again, this classification entails that respondents in the healthy category have limited room for improvement because they already comply with the standards. We have included a translation of this questionnaire in Multimedia Appendix 2. Self-efficacy for diet and PA were both measured using a 3-item questionnaire with a 5-point Likert scale ranging from 1 (very high) to 5 (very low) [32]. Knowledge was assessed using a 10-item true/false questionnaire based on the Netherlands classification model [28] for diet and a 10-item true/false questionnaire for physical activity based on the Dutch Standard for Healthy Physical Activity [33]. The total scores of these questionnaires range from 1 (very poor) to 10 (excellent). Attitude was measured using a 5-item questionnaire on health consciousness attitude and a 6-item questionnaire on health-oriented beliefs; all questions used a 5-point Likert-scale ranging from 1 (very unfavorable) to 5 (very favorable). These questionnaires were based on the research of Dutta-Bergman [34] and adapted to the Dutch situation. Self-rated behavior (henceforth self-rating) was assessed by 2 items, 1 on self-rated diet and 1 on self-rated PA, both using a scale from 1 (very poor) to 10 (excellent). Insight into behavior was calculated by comparing self-reported and self-rated diet and PA based on the classification used by Ronda et al [35]. Self-rating was recoded into categories to match the categories of self-reported behavior. Therefore, self-rated diet was recoded into three categories (1-4: unhealthy; 5-7: improvable; 8-10: healthy) and self-rated PA was recoded into two categories (1-5: unhealthy; 6-10: healthy). Respondents who did not meet the criteria for recommended healthy behavior but rated their own behavior as healthy were classified as overestimators. Respondents who did meet the criteria for healthy behavior but rated their behavior as unhealthy were classified as underestimators. The remaining respondents were considered to have had realistic insight into their behavior. Pretest and posttest questionnaires were identical except for the following additional items at posttest: the number of newsletters received and opened (waiting list group) and satisfaction with the HWA (intervention group). Satisfaction was measured using 4 items with a 5-point Likert scale ranging from 1 (very negative) to 5 (very positive) on user friendliness, usefulness, recommending to others, and willingness to continue using the HWA [36].

In addition to the online questionnaires, the HWA stored every log-on by a participant. These log files were used to attain the usage of the HWA, that is, the number of times each respondent logged on to the HWA within the intervention period.

Electronic Surveys

SurveyMonkey was used for the electronic data collection [37]. The first page of the survey consisted of an informed consent. By agreeing to participate, participants were led to the actual questionnaire. Data was protected following the security measures of SurveyMonkey [38]. Moreover, no personal identifying information apart from an email address was collected. Our survey was pretested using 5 nonparticipants comparable to the participants of the study. Feedback from the pretest was implemented in the final survey.

Our format of data collection was an “open survey” [39] posted on a website. The survey was only accessible through our research website for respondents who satisfied our inclusion criteria. The initial contact mode was through online and offline advertisements for research into an online lifestyle coach. It was mandatory for participants to fill out the questionnaire to be included in the study. We offered no incentives to participate other than the use of the lifestyle coach. The pretest questionnaire was available for 8 weeks; the posttest questionnaire was available for 3 weeks. We used randomization of items for Likert-type questions with no specific order. The number of items was 42, divided over 5 screens. All questions were mandatory except comment boxes. Respondents were able to review and, if necessary, change previous answers until they had submitted the completed questionnaire.

We were not able to record unique site visitors or survey visitors. The completion rate was 90% (269/297). To prevent multiple entries from the same person we used cookies that were stored when visiting the first page and were valid for 14 days. Also, we checked IP addresses. Entries from the same address with identical sex and birth date were checked for completeness. The most complete entry was saved, or, in case of equal completeness, the first entry was saved.

Participants

Previous research on the HWA using the same research instrument on self-reported dietary behavior yielded information on the mean and standard deviation of this primary outcome measure (mean 62.9, SD 8.43) [27]. To be able to measure a meaningful difference (3.5 points) we needed a detectable effect size of 0.4. When testing at the .05 level, and, using a power of 80%, we calculated that we needed a sample size of 200 (100 per group).

Analyses

Statistical analyses were performed using SPSS Statistics 17.0 (IBM Corporation, Somers, NY, USA). We used the multiple imputation (MI) feature of SPSS Statistics 17.0 to handle missing data of posttest nonrespondents. Demographic variables and baseline outcome measures were used as predictors in the imputation model. We used an iterative Markov chain Monte Carlo method, which is the fully conditional specification. In addition, five imputed datasets were generated on which the effectiveness analyses were performed. When possible, pooled outcomes were used for the analyses; otherwise, the five estimates were combined into a single overall estimate following the MI inference rules of Rubin [40].

Differences between users and nonusers within the intervention group were assessed using Pearson's chi-square and analysis of variance testing. Furthermore, regression analysis was used to see whether characteristics predicted use of the intervention.

Effectiveness of the intervention was assessed by intention-to-treat (ITT) using effect sizes and odds ratios. Additionally, exploratory analyses were performed on pretest and posttest scores of all participants combined and separately for the control group, the users, and the nonusers of the intervention using regression analyses and effect sizes. All reported P values are 2-tailed. We used no statistical measures to correct for multiple testing. Effect sizes for differences in means are presented as Cohen’s d and effect sizes for nonparametric variables are presented as r, calculated from the z scores of the Wilcoxon signed rank test [41].

Results

Response Rates

Of the 269 enrolled respondents (those who completed the pretest questionnaire), 159 respondents filled out the posttest questionnaire (response rate = 59%, 159/269). The response was significantly lower in the intervention group (51%, 65/127) than in the control group (66%, 94/142) (P = .01). There were baseline differences between responders (ie, respondents who filled out the posttest questionnaire) and research dropouts on outcome variables. As shown in Table 1, dropouts scored significantly lower on attitude and self-rating. In addition, within the intervention group, only 48% (30/62) of dropouts used the HWA as opposed to 78% (51/65) of responders (χ2 1 = 12.424, P < .001).

Table 1
Baseline differences on outcome variables between responders and dropouts

Descriptive Analyses of Baseline Variables

As shown in Table 2, most of the respondents in this study were female (177/269, 66%) and in the highest education category (143/269, 53%). Mean age was 41.5 years (SD 13.5). There were no significant differences between the intervention and control group on demographic variables and reasons for use. On outcome variables, there was one significant difference at baseline, that is, respondents in the intervention group scored significantly higher on self-efficacy than respondents in the control group. Mean scores were respectively 2.2 (SD 0.6) versus 2.1 (SD 0.6) (F 1,267 = 4.109, P = .044). The most frequently mentioned reason by respondents for wanting to use the application was to gain more insight into their own lifestyle.

Table 2
Baseline demographics and reasons for use

Users and Nonusers

Respondents in the waiting list (control) condition reported to have opened a mean of 3.4 (SD 1.2) out of 5 newsletters. From the log files of the HWA, we know that 81 of the 127 (64%) respondents in the intervention group used the HWA at least once, while 49% (40/81) of these used the application only once. The respondent that used the HWA most frequently used it 13 times during the intervention period of 12 weeks. The median number of times HWA was used was 1.0. Of the 127 respondents in the intervention group, 4 (3%) used the application at least the intended number of times within the intervention period (ie, once a fortnight or 6 times during the 12-week period). Satisfaction with the application was assessed within the posttest questionnaire. We used only the data provided by 50 respondents who filled out the posttest questionnaire and who had accessed the HWA at least once in the intervention period. These results are depicted in Table 3. The overall mean satisfaction score for these 50 respondents was 3.0 (SD 0.74) on the 5-point scales where 1 = very negative and 5 = very positive. A score of 3.0 lies within the neutral category.

Table 3
Satisfaction with the Healthy Weight Assistant (n = 50)

Baseline differences between respondents in the intervention group who used the application (users) and the respondents in this group who did not use the HWA at least once (nonusers) are depicted in Table 4.

Table 4
Baseline differences between users and nonusers in the intervention group

Overall, at baseline, users were healthier (scored better on dietary behavior and had a chronic condition less often) and were more knowledgeable about healthy behavior. Furthermore, users seemed to underestimate their behavior more often than nonusers, and nonusers seemed to overestimate their behavior more often than users.

To assess whether variables of the framework proposed in the introduction could be used to predict if respondents were going to use the HWA, we performed an exploratory logistic regression using the factors from the framework (social and economic, condition-related, patient-related or constructs from behavior change theories, and reasons for use). Results of this logistic regression (Table 5) showed that one variable within the social and economic factor (ie, age) and one variable within the condition-related factor (ie, chronic health condition) significantly contributed to the model. The model showed that increased age and not having a chronic condition increased the odds of having used the application (Cox & Snell R 2 = .24, Nagelkerke R 2 = .32, Model χ2 18 = 34.15, P = .01).

Table 5
Logistic regression model to predict usage of the HWA

Furthermore, we performed a linear regression to investigate whether satisfaction with the intervention HWA predicted the number of logins (Table 6). The model showed that satisfaction did not predict frequency of use (R 2 = .05, adjusted R 2 = .04).

Table 6
Linear regression on satisfaction predicting number of log-ins to the Healthy Weight Assistant

Effectiveness

In addition, ITT analyses were performed on all outcome variables (Table 6). We found a significant but very small effect on attitude (d = 0.08) favoring the intervention group. None of the other variables showed a significant effect of the intervention.

Complementary to the ITT analyses, we performed analyses comparing the differences of the control group with the differences of the users (results not shown). These analyses did not yield any significant effects and were comparable to the results of the ITT analyses, although the effect sizes were generally larger.

Table 7
Intention-to-treat (ITT) analyses

For the group as whole (independent of randomized condition), there were significant differences between pretest and posttest scores. With respect to diet (effect size r = −0.12), physical activity (effect size r = −0.09), and self-rating (effect size d = 0.21) the study seemed to have had a positive influence, although the effect was small (data not shown). These differences could not be attributed to the intervention according to the ITT analyses. As mentioned in the previous paragraph, our results showed differences between users and nonusers at baseline. Therefore, we performed exploratory analyses on these groups to investigate whether choosing to use or not to use the application led to different outcomes. Tables 8 and and99 show that we found significant differences in some groups but not in others. Contrary to what we expected, only the nonusers showed a significant improvement on diet (r = −0.23). Examining the data more closely revealed that the control group and, to a larger extent, the users also showed improvement, although this difference was not significant. The data showed that only the users significantly improved with respect to PA behavior (effect size r = −0.17). The control group showed a nonsignificant improvement, while PA behavior of the nonusers deteriorated, but the change was nonsignificant. With respect to attitude, the nonusers showed a significant improvement with a medium effect size (d = 0.28), although the absolute difference was small. With respect to self-efficacy, the control group and the nonusers showed deterioration (effect sizes respectively d = 0.14 and d = 0.33), again with small absolute differences. Lastly, the data showed that users’ self-rated behavior was more favorable at posttest than at pretest. The size of this effect was small to medium (d = 0.27).

Table 8
Pretest and posttest values on outcome variables for control group, nonusers, and users
Table 9
Effect size (ES) of the differences between pretest and posttest values on outcome variables for control group, nonusers, and users

Discussion

The results showed that the HWA was not used as often as intended. Increased age and not having a chronic condition increased the odds of having used the application at least once. Moreover, users were healthier and more knowledgeable about healthy behavior than nonusers. The ITT analyses showed no apparent effects of the intervention; however, there were differences in the effect of the intervention on users and nonusers. With respect to dietary behavior and attitude, nonusers improved more than users, while with respect to physical activity and self-rated behavior the users improved more than nonusers. On self-efficacy, the control group and the nonusers showed deterioration from baseline to posttest.

Only 64% (81 out of 127) of the participants who received access to the HWA actually used the application. This finding is not unique to this study; for example, see [6,15,20,42]. This stresses an important aspect of Web-based interventions, that is, of the respondents who agree to participate in a study on a Web-based intervention, we can expect that a substantial percentage does not use the intervention at all. In addition, we saw that the HWA is not used as often as intended in the design of the application. Of the included social and economic factors of the proposed framework, only increased age increased the odds of having used the application. This finding might seem counterintuitive, but it concurs with recent findings on the motivation to use e-consultation [43], which showed that older people were more motivated to use e-consultation than younger people. With respect to the condition-related factors, the regression analysis showed that having a chronic condition decreased the odds of using the application. An explanation might be that the HWA was not developed for people with chronic conditions and no special attention is paid to the needs of people with chronic conditions. Therefore, these people might feel that the HWA does not suit their needs. Significant differences between users and nonusers on condition-related factors showed that users were healthier. A reason for this might be that people like to be rewarded for their healthy behavior and not confronted with their unhealthy behavior.

Additionally, users more often underestimated their dietary behavior (respondents who did meet the criteria for healthy behavior but who rated their behaviour as unhealthy were classified as underestimators), while nonusers more often overestimated their behavior. This shows that the people who could have benefited most from the HWA were less likely to use the application. Of the patient-related factors or constructs from behavior change theories, only knowledge showed a significant difference between users and nonusers. Users knew more about healthy behavior, which supports the notion that the people who could have benefited most from the HWA were least likely to use the application.

There were no differences related to the reasons for use between users and nonusers, and the different reasons do not explain whether respondents used the HWA or not. However, the reasons for use might play a role in the frequency of use. The most frequently mentioned reason for wanting to use the intervention was to gain insight into one’s own behavior (60%). It might be that this goal was reached after using the HWA once, and participants might not have felt the need to use the HWA again.

Interestingly, the intervention was specifically not made to help people lose weight, but this goal was mentioned by 40% of respondents. Respondents seemed to want a quick and short-term effect (to gain insight) and might not have been willing to use the intervention frequently to work on a long-term goal (eg, a healthier lifestyle). Satisfaction with the HWA was not associated with the frequency of use. However, overall, participants were not very satisfied with the HWA, which might have contributed to the relative low usage rates. To summarize, one of the social and economic factors (ie, age), condition related factors (ie, chronic condition, self-reported behavior, and insight into behavior), and one of the patient-related factors (ie, knowledge) were related to use of the system. Satisfaction and reasons for use provided more in-depth information related to the causes of the lack of adherence to the intervention.

At baseline, the intervention and control groups showed a significant difference in attitude. The absolute value of the difference was small, however, and we don’t consider it to be a meaningful difference. Therefore, we can argue that the groups were comparable at baseline. We found no meaningful significant effects of the intervention using ITT analyses. We did find that both the waiting list group and the intervention group showed significant improvement on behavior and a significantly more favorable self-rated behavior. This well-known Hawthorne effect [44] (ie, the effect on outcome through participation in research) might be explained by the increased attention participants paid to healthy behavior due to completing questionnaires on behavior and by increased awareness of current and desired behavior. Another explanation for the improvement in all respondents might be social desirability. Thinking of the intended behavior might have influenced the responses given in the posttest questionnaire. Considering the control group, the users, and the nonusers separately showed that only nonusers significantly improved on dietary behavior. This might be due to the large differences between users and nonusers at baseline. Users were already much healthier, and both groups improved, although at posttest, the nonusers were still less healthy than the users. It seems that a ceiling effect prevented the users from improving significantly while the nonusers had much room for improvement and, for that reason, showed significant improvement. On PA, we found that only the users of the intervention improved significantly, although the effect size was not very large (r = −0.17). The nonusers, who chose not to use the intervention, showed a decline in behavior while the control group showed improvement. Although these differences were not significant, this does point toward a difference between choosing not to use an intervention and not being able to use an intervention. However, these differences might also reflect social desirability because of the focus on PA in the intervention. Lastly, users judged their own behavior significantly more positively after the intervention period than before. None of the other groups showed this significant improvement. Summarizing, we found no apparent effects of the HWA, but it seems that having chosen to use or not to use the intervention led to different outcomes. Combined with the differences between groups at baseline, this seems to imply that these groups are truly different and should be treated as separate entities.

In this study, we were faced with substantial dropout and nonusage rates. High dropout rates are not uncommon in this field of research and have been said to be a major challenge [45,46]. Additionally, the reduction of nonusage rates is also a major challenge [15,47]. Faced with these challenges, it is important to note that in this study the groups of dropouts and nonusers overlapped, but were not the same. Almost half of the dropouts had been users, and there were also nonusers that were responders. Consequently, it is very important keep these two concepts apart.

Our results showed that the users of the HWA were healthier than nonusers, which is an unfortunate finding not unique to this study [18]. The group for which the intervention seemed to have been most useful, namely people who had room for improvement on both diet and physical activity, were less likely to have used the HWA. This tells us that we need to try different ways to entice potential users who could benefit from the HWA to become active users. More effort should be made to tempt the nonusers of the intervention to become users. One way to do this might be to make it as easy as possible to start using the application. The moment people are interested, they should be able to use the application. In our study, there was considerable time between expressing interest and being able to use the application. Moreover, participants had to check their email, click on a link, and create a profile. All these steps require effort and could thereby decrease the odds of using the intervention. Once participants become users, the application itself can stimulate adherence. This might be done by regularly providing new content, by including reminders (through email or text messaging), or by explicitly telling participants what is expected of them in terms of usage. In our view, including these aspects would have improved the HWA.

In this study, the frameworks used to predict usage and to study effectiveness seem to have been insufficient. From the WHO framework [16], some factors, especially condition-related, seem to have explanatory power but not enough to fully explain why participants choose whether to use an intervention. This might be due to the fact that the goal of the model is adherence to treatment and not adherence to technology. Moreover, attitude, self-efficacy, and knowledge do not contribute to a better understanding of the effects of the intervention. These variables from classic behavior change theories might not discriminate enough. To gain more insight into how online interventions can support people in changing their behavior, we should try to take into account the specific barriers and opportunities of eHealth interventions and integrate them into a comprehensive conceptual framework.

Limitations

A limitation of this study is the use of self-reported behaviors. Although we used questionnaires used in previous studies, there is a chance of biased results due to social desirability or lack of insight into behavior. As a consequence, a possible change in insight into behavior might not be reflected in our results. It could be that at baseline, participants provided optimistic self-reported behavior. Due to the intervention, the users might have provided more realistic self-reported behavior at posttest. Unfortunately, this potentially positive effect of the HWA could not be tested in this study. A second limitation is related to the participants in this study. Most respondents were female and highly educated. Various studies have reported overrepresentation of this group [6,18,48,49]. Nevertheless, the question remains whether these results can be generalized to the broader target population of the HWA. Another limitation of this study is that we measured the usage of the system as the number of log-ons per participant. What participants did while logged on and for how long they were logged on, we do not know. As more and more eHealth research takes the usage of the applications into account, it might be beneficial to standardize the assessment of usage. Furthermore, a limitation of this study is related to the response rate. Our overall response rate was quite low (59%), and we found significant differences between responders and nonresponders. We accounted for this bias by using multiple imputation procedures. However, imputing 41% of the data might have yielded unreliable estimates, although research has shown that imputing up to 58% can be more reliable than listwise deletion [40,50]. In our view, this study has provided valuable insights into the users of a Web-based intervention. However, had we been able to conduct this study again, we would have changed the way we dealt with several issues. First of all, we would have included a larger number of respondents to certify a sample size large enough to account for the high dropout and nonusage rates. Second, we would have tested and adapted the application during development so that we could have chosen the outcome measures and study period to better reflect the goals and expected effects of the application. Unfortunately, this was not possible in the current study, and this stresses the importance of a close collaboration between researchers and developers of eHealth interventions.

Future Work

Usage is a major issue in research into the effects of eHealth applications. More research is needed into transforming potential users into actual users and into keeping them engaged with the application and, thereby, stimulating them to keep using the intervention. Moreover, long-term research on the use of eHealth applications is needed to provide insight into the way usage fluctuates over time. From this study, we have gained insight into differences between users and nonusers, which can be seen as a first step to decreasing attrition. The next step might be found when looking at the opportunities technology has to offer. For example, several recent studies have shown beneficial effects of adding mobile technology [51-53] and devices that provide automated tailored feedback [54]. Additionally, the field of persuasive technology might provide us with insight into how technology as a medium can persuade and motivate users to change behavior [55,56].

Acknowledgments

We are grateful for funding from the Netherlands Nutrition Centre.

Abbreviations

BMI
body mass index
CI
confidence interval
ES
effect size
GP
general practitioner
HWA
Healthy Weight Assistant
ITT
intention-to-treat
OR
odds ratio
PA
physical activity
SE
standard error
WHO
World Health Organization

Multimedia Appendix 1

Translated questionnaire dietary behavior

Multimedia Appendix 2

Translated questionnaire physical activity behavior

Footnotes

Conflicts of Interest:

None declared

References

1. Glenny AM, O'Meara S, Melville A, Sheldon TA, Wilson C. The treatment and prevention of obesity: a systematic review of the literature. Int J Obes Relat Metab Disord. 1997 Sep;21(9):715–37. [PubMed]
2. Hardeman W, Griffin S, Johnston M, Kinmonth AL, Wareham NJ. Interventions to prevent weight gain: a systematic review of psychological models and behaviour change methods. Int J Obes Relat Metab Disord. 2000 Feb;24(2):131–43. [PubMed]
3. Doak CM, Visscher TL, Renders CM, Seidell JC. The prevention of overweight and obesity in children and adolescents: a review of interventions and programmes. Obes Rev. 2006 Feb;7(1):111–36. doi: 10.1111/j.1467-789X.2006.00234.x.OBR234 [PubMed] [Cross Ref]
4. Müller MJ, Asbeck I, Mast M, Langnäse K, Grund A. Prevention of obesity--more than an intention. Concept and first results of the Kiel Obesity Prevention Study (KOPS) Int J Obes Relat Metab Disord. 2001 May;25(Suppl 1):S66–74. doi: 10.1038/sj.ijo.0801703. http://dx.doi.org/10.1038/sj.ijo.0801703. [PubMed] [Cross Ref]
5. Kroeze W, Werkman A, Brug J. A systematic review of randomized trials on the effectiveness of computer-tailored education on physical activity and dietary behaviors. Ann Behav Med. 2006 Jun;31(3):205–23. doi: 10.1207/s15324796abm3103_2. [PubMed] [Cross Ref]
6. Rothert K, Strecher VJ, Doyle LA, Caplan WM, Joyce JS, Jimison HB, Karm LM, Mims AD, Roth MA. Web-based weight management programs in an integrated health care setting: a randomized, controlled trial. Obesity (Silver Spring) 2006 Feb;14(2):266–72. doi: 10.1038/oby.2006.34. http://dx.doi.org/10.1038/oby.2006.34.14/2/266 [PubMed] [Cross Ref]
7. Wammes B, Oenema A, Brug J. The evaluation of a mass media campaign aimed at weight gain prevention among young Dutch adults. Obesity (Silver Spring) 2007 Nov;15(11):2780–9. doi: 10.1038/oby.2007.330. http://dx.doi.org/10.1038/oby.2007.330.15/11/2780 [PubMed] [Cross Ref]
8. Neuhauser L, Kreps GL. eHealth communication and behavior change: promise and performance. Social Semiotics. 2010;20(1):9–27. doi: 10.1080/10350330903438386. [Cross Ref]
9. Griffiths F, Lindenmeyer A, Powell J, Lowe P, Thorogood M. Why are health care interventions delivered over the internet? A systematic review of the published literature. J Med Internet Res. 2006;8(2):e10. doi: 10.2196/jmir.8.2.e10. http://www.jmir.org/2006/2/e10/v8i2e10 [PMC free article] [PubMed] [Cross Ref]
10. Moore TJ, Alsabeeh N, Apovian CM, Murphy MC, Coffman GA, Cullum-Dugan D, Jenkins M, Cabral H. Weight, blood pressure, and dietary benefits after 12 months of a Web-based Nutrition Education Program (DASH for health): longitudinal observational study. J Med Internet Res. 2008;10(4):e52. doi: 10.2196/jmir.1114. http://www.jmir.org/2008/4/e52/v10i4e52 [PMC free article] [PubMed] [Cross Ref]
11. Svetkey LP, Stevens VJ, Brantley PJ, Appel LJ, Hollis JF, Loria CM, Vollmer WM, Gullion CM, Funk K, Smith P, Samuel-Hodge C, Myers V, Lien LF, Laferriere D, Kennedy B, Jerome GJ, Heinith F, Harsha DW, Evans P, Erlinger TP, Dalcin AT, Coughlin J, Charleston J, Champagne CM, Bauck A, Ard JD, Aicher K, Weight Loss Maintenance Collaborative Research Group Comparison of strategies for sustaining weight loss: the weight loss maintenance randomized controlled trial. JAMA. 2008 Mar 12;299(10):1139–48. doi: 10.1001/jama.299.10.1139. http://jama.ama-assn.org/cgi/pmidlookup?view=long&pmid=18334689.299/10/1139 [PubMed] [Cross Ref]
12. An JY, Hayman LL, Park YS, Dusaj TK, Ayres CG. Web-based weight management programs for children and adolescents: a systematic review of randomized controlled trial studies. ANS Adv Nurs Sci. 2009;32(3):222–40. doi: 10.1097/ANS.0b013e3181b0d6ef.00012272-200907000-00005 [PubMed] [Cross Ref]
13. Hunter CM, Peterson AL, Alvarez LM, Poston WC, Brundige AR, Haddock CK, Van Brunt DL, Foreyt JP. Weight management using the internet a randomized controlled trial. Am J Prev Med. 2008 Feb;34(2):119–26. doi: 10.1016/j.amepre.2007.09.026.S0749-3797(07)00646-0 [PubMed] [Cross Ref]
14. Neve M, Morgan PJ, Jones PR, Collins CE. Effectiveness of web-based interventions in achieving weight loss and weight loss maintenance in overweight and obese adults: a systematic review with meta-analysis. Obes Rev. 2010 Apr;11(4):306–21. doi: 10.1111/j.1467-789X.2009.00646.x.OBR646 [PubMed] [Cross Ref]
15. Eysenbach G. The law of attrition. J Med Internet Res. 2005;7(1):e11. doi: 10.2196/jmir.7.1.e11. http://www.jmir.org/2005/1/e11/v7e11 [PMC free article] [PubMed] [Cross Ref]
16. Sabaté E. World Health Organization. 2003. [2011-03-21]. webcite Adherence to long-term therapies: evidence for action. www.who.int/chp/knowledge/publications/adherence_report/en/
17. Wanner M, Martin-Diener E, Bauer G, Braun-Fahrländer C, Martin BW. Comparison of trial participants and open access users of a web-based physical activity intervention regarding adherence, attrition, and repeated participation. J Med Internet Res. 2010;12(1):e3. doi: 10.2196/jmir.1361. http://www.jmir.org/2010/1/e3/v12i1e3 [PMC free article] [PubMed] [Cross Ref]
18. Verheijden MW, Jans MP, Hildebrandt VH, Hopman-Rock M. Rates and determinants of repeated participation in a web-based behavior change program for healthy body weight and healthy lifestyle. J Med Internet Res. 2007;9(1):e1. doi: 10.2196/jmir.9.1.e1. http://www.jmir.org/2007/1/e1/v9i1e1 [PMC free article] [PubMed] [Cross Ref]
19. Ware LJ, Hurling R, Bataveljic O, Fairley BW, Hurst TL, Murray P, Rennie KL, Tomkins CE, Finn A, Cobain MR, Pearson DA, Foreyt JP. Rates and determinants of uptake and use of an internet physical activity and weight management program in office and manufacturing work sites in England: cohort study. J Med Internet Res. 2008;10(4):e56. doi: 10.2196/jmir.1108. http://www.jmir.org/2008/4/e56/v10i4e56 [PMC free article] [PubMed] [Cross Ref]
20. Glasgow RE, Nelson CC, Kearney KA, Reid R, Ritzwoller DP, Strecher VJ, Couper MP, Green B, Wildenhaus K. Reach, engagement, and retention in an Internet-based weight loss program in a multi-site randomized controlled trial. J Med Internet Res. 2007;9(2):e11. doi: 10.2196/jmir.9.2.e11. http://www.jmir.org/2007/2/e11/v9i2e11 [PMC free article] [PubMed] [Cross Ref]
21. Kaptein M, Eckles D. Selecting effective means to any end: Futures and ethics of persuasion profiling. Proceedings of the 5th International Conference Persuasive; Persuasive Technology; June 7 - 10, 2010; Copenhagen. 2010. [Cross Ref]
22. Christensen H, Griffiths KM, Farrer L. Adherence in internet interventions for anxiety and depression. J Med Internet Res. 2009;11(2):e13. doi: 10.2196/jmir.1194. http://www.jmir.org/2009/2/e13/v11i2e13 [PMC free article] [PubMed] [Cross Ref]
23. DeLone W, McLean E. Information systems success revisited. Proceedings of the Thirty-fifth Hawaii International Conference on System Science. Vol 8; International Conference on System Science; January 7 - 10, 2002; Hawaii. Los Alamitos, CA: IEEE Computer Society Press; 2002. http://doi.ieeecomputersociety.org/10.1109/HICSS.2002.994345.
24. Noar SM, Zimmerman RS. Health Behavior Theory and cumulative knowledge regarding health behaviors: are we moving in the right direction? Health Educ Res. 2005 Jun;20(3):275–90. doi: 10.1093/her/cyg113. http://her.oxfordjournals.org/cgi/pmidlookup?view=long&pmid=15632099.cyg113 [PubMed] [Cross Ref]
25. Fishbein M. Developing effective behavior change interventions: some lessons learned from behavioral research. NIDA Res Monogr. 1995;155:246–61. [PubMed]
26. Prochaska JO, Velicer WF. The transtheoretical model of health behavior change. Am J Health Promot. 1997;12(1):38–48. [PubMed]
27. Kelders SM, van Gemert-Pijnen JE, Werkman A, Seydel ER. Evaluation of a web-based lifestyle coach designed to maintain a healthy bodyweight. J Telemed Telecare. 2010;16(1):3–7. doi: 10.1258/jtt.2009.001003.16/1/3 [PubMed] [Cross Ref]
28. Netherlands Nutrition Center. 2005. [2010-06-18]. webcite Criteria for the nutritional evaluation of foods. http://www.voedingscentrum.nl/resources2008/Criteriaengelssite.pdf.
29. Voedingscentrum. 2009. [2011-02-10]. webcite Richtlijnen voedselkeuze. http://www.voedingscentrum.nl/resources2008/RICHTLIJNENVOEDSELKEUZE20091.pdf.
30. Health Council of the Netherlands. [2011-02-10]. webcite Guidelines for a Healthy Diet 2006. http://www.gezondheidsraad.nl/sites/default/files/2006@21N.pdf.
31. Douwes M, Hildebrandt VH. Vragen naar de mate van lichamelijke activiteit. Geneeskd Sport. 2000;33(1):9–16.
32. Sallis JF, Pinsky RB, Grossman RM, Patterson TL, Nader PR. The development of self-efficacy scales for health-related diet and exercise behaviors. Health Educ Res. 1988;3(3):283–292.
33. Kemper HGC, Ooijendijk WTM, Stiggelbout M. Consensus over de Nederlandse Norm voor Gezond Bewegen. TSG. 2000;78:180–183.
34. Dutta-Bergman MJ. Health attitudes, health cognitions, and health behaviors among Internet health information seekers: population-based survey. J Med Internet Res. 2004 May 28;6(2):e15. doi: 10.2196/jmir.6.2.e15. http://www.jmir.org/2004/2/e15/v6e15 [PMC free article] [PubMed] [Cross Ref]
35. Ronda G, Van Assema P, Brug J. Stages of change, psychological factors and awareness of physical activity levels in The Netherlands. Health Promot Int. 2001 Dec;16(4):305–14. http://heapro.oxfordjournals.org/cgi/pmidlookup?view=long&pmid=11733449. [PubMed]
36. Tsang MW, Mok M, Kam G, Jung M, Tang A, Chan U, Chu CM, Li I, Chan J. Improvement in diabetes control with a monitoring system based on a hand-held, touch-screen electronic diary. J Telemed Telecare. 2001;7(1):47–50. [PubMed]
37. Surveymonkey. [2010-06-18]. webcite http://www.surveymonkey.com/
38. Surveymonkey. [2010-06-18]. webcite How do you keep our data secure and where is it stored? http://help.surveymonkey.com/app/answers/detail/a_id/337/kw/security.
39. Eysenbach G. Improving the quality of Web surveys: the Checklist for Reporting Results of Internet E-Surveys (CHERRIES) J Med Internet Res. 2004 Sep 29;6(3):e34. doi: 10.2196/jmir.6.3.e34. http://www.jmir.org/2004/3/e34/v6e34 [PMC free article] [PubMed] [Cross Ref]
40. Rubin D. Multiple Imputation for Nonresponse in Surveys. New York, NY: John Wiley & Sons, Inc; 1987.
41. Rosenthal R. Meta-analytic Procedures for Social Research. Newbury Park, CA: Sage Publications, Inc; 1991.
42. Balmford J, Borland R, Benda P. Patterns of use of an automated interactive personalized coaching program for smoking cessation. J Med Internet Res. 2008;10(5):e54. doi: 10.2196/jmir.1016. http://www.jmir.org/2008/5/e54/v10i5e54 [PMC free article] [PubMed] [Cross Ref]
43. Nijland N, van Gemert-Pijnen JE, Boer H, Steehouder MF, Seydel ER. Increasing the use of e-consultation in primary care: results of an online survey among non-users of e-consultation. Int J Med Inform. 2009 Oct;78(10):688–703. doi: 10.1016/j.ijmedinf.2009.06.002.S1386-5056(09)00100-2 [PubMed] [Cross Ref]
44. McCarney R, Warner J, Iliffe S, van Haselen R, Griffin M, Fisher P. The Hawthorne Effect: a randomised, controlled trial. BMC Med Res Methodol. 2007;7:30. doi: 10.1186/1471-2288-7-30. http://www.biomedcentral.com/1471-2288/7/30.1471-2288-7-30 [PMC free article] [PubMed] [Cross Ref]
45. Murray E, Khadjesari Z, White IR, Kalaitzaki E, Godfrey C, McCambridge J, Thompson SG, Wallace P. Methodological challenges in online trials. J Med Internet Res. 2009;11(2):e9. doi: 10.2196/jmir.1052. http://www.jmir.org/2009/2/e9/v11i2e9 [PMC free article] [PubMed] [Cross Ref]
46. Couper MP, Peytchev A, Strecher VJ, Rothert K, Anderson J. Following up nonrespondents to an online weight management intervention: randomized trial comparing mail versus telephone. J Med Internet Res. 2007;9(2):e16. doi: 10.2196/jmir.9.2.e16. http://www.jmir.org/2007/2/e16/v9i2e16 [PMC free article] [PubMed] [Cross Ref]
47. Neve MJ, Collins CE, Morgan PJ. Dropout, nonusage attrition, and pretreatment predictors of nonusage attrition in a commercial Web-based weight loss program. J Med Internet Res. 2010;12(4):e69. doi: 10.2196/jmir.1640. http://www.jmir.org/2010/4/e69/v12i4e69 [PMC free article] [PubMed] [Cross Ref]
48. Binks M, van Mierlo T. Utilization patterns and user characteristics of an ad libitum Internet weight loss program. J Med Internet Res. 2010;12(1):e9. doi: 10.2196/jmir.1347. http://www.jmir.org/2010/1/e9/v12i1e9 [PMC free article] [PubMed] [Cross Ref]
49. Postel MG, de Jong CA, de Haan HA. Does e-therapy for problem drinking reach hidden populations? Am J Psychiatry. 2005 Dec;162(12):2393. doi: 10.1176/appi.ajp.162.12.2393. http://ajp.psychiatryonline.org/cgi/pmidlookup?view=long&pmid=16330613.162/12/2393 [PubMed] [Cross Ref]
50. King G, Honaker J, Joseph A, Scheve K. Analyzing incomplete political science data: An alternative algorithm for multiple imputation. American Political Science Review. 2001;95:49–69.
51. Patrick K, Raab F, Adams MA, Dillon L, Zabinski M, Rock CL, Griswold WG, Norman GJ. A text message-based intervention for weight loss: randomized controlled trial. J Med Internet Res. 2009;11(1):e1. doi: 10.2196/jmir.1100. http://www.jmir.org/2009/1/e1/v11i1e1 [PMC free article] [PubMed] [Cross Ref]
52. Tufano JT, Karras BT. Mobile eHealth interventions for obesity: a timely opportunity to leverage convergence trends. J Med Internet Res. 2005;7(5):e58. doi: 10.2196/jmir.7.5.e58. http://www.jmir.org/2005/5/e58/v7i5e58 [PMC free article] [PubMed] [Cross Ref]
53. Webb TL, Joseph J, Yardley L, Michie S. Using the internet to promote health behavior change: a systematic review and meta-analysis of the impact of theoretical basis, use of behavior change techniques, and mode of delivery on efficacy. J Med Internet Res. 2010;12(1):e4. doi: 10.2196/jmir.1376. http://www.jmir.org/2010/1/e4/v12i1e4 [PMC free article] [PubMed] [Cross Ref]
54. Richardson CR, Brown BB, Foley S, Dial KS, Lowery JC. Feasibility of adding enhanced pedometer feedback to nutritional counseling for weight loss. J Med Internet Res. 2005;7(5):e56. doi: 10.2196/jmir.7.5.e56. http://www.jmir.org/2005/5/e56/v7i5e56 [PMC free article] [PubMed] [Cross Ref]
55. Chatterjee S, Price A. Healthy living with persuasive technologies: framework, issues, and challenges. J Am Med Inform Assoc. 2009;16(2):171–8. doi: 10.1197/jamia.M2859.M2859 [PMC free article] [PubMed] [Cross Ref]
56. Fogg BJ. Persuasive Technology: Using Computers to Change What We Think and Do. San Francisco, CA: Morgan Kaufmann Publishers; 2003.

Articles from Journal of Medical Internet Research are provided here courtesy of Gunther Eysenbach

Related citations in PubMed

See reviews...See all...

Cited by other articles in PMC

See all...

Links

  • Cited in Books
    Cited in Books
    PubMed Central articles cited in books
  • PubMed
    PubMed
    PubMed citations for these articles

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...