How to do (or not to do)… Measuring health worker motivation in surveys in low- and middle-income countries

Step 1: conceptualizing motivation

Motivation is a complex construct as indicated in this definition:

‘Work motivation is a set of energetic forces that originate both within as well as beyond an individual’s being, to initiate work-related behaviour, and to determine its form, direction, intensity, and duration’ (Pinder 2008).

A list of the most prominent motivation theories is provided in Box 1. Motivation is usually either conceptualized as a unidimensional construct, where the focus is on the overall quantity of motivation available to drive behaviour (Gow et al. 2013; Hagopian et al. 2009); or it is conceptualized as a multidimensional construct, with an additional focus, for example, on the composition of qualitatively different types of motivation such as intrinsic and extrinsic motivation (Lohmann et al. 2016). For definitions of key terms such as ‘construct’ please refer to Box 2. In some cases, researchers may wish to capture multiple conceptualizations. The choice of approach depends on the research question and, in the case of programme evaluation, one’s theory about how a given programme will affect motivation.

The measurement of motivation is more difficult. A key question is whether to measure motivation itself (a ‘direct’ measure) by, e.g. seeing how a programme affects intrinsic motivation; or to instead measure the things that affect or are affected by motivation (‘proxies’, ‘indirect’ measures). Direct measures of motivation are typically derived through measurement scales within a survey or through qualitative methods (Inceoglu et al. 2012; Deci and Ryan 1985). For example, JL and ED examined whether financial incentives crowd out intrinsic motivation using measurement scales grounded in Self-Determination Theory in health worker surveys in Burkina Faso and Afghanistan (Lohmann et al. 2017; Dale 2014) (see Figure 1). Indirect measures can be equally derived through surveys or qualitative methods or through experimental games or observations of behaviour. For instance, the Franco framework, which has been widely used in health worker motivation studies in low- and middle-income countries (LMICs), measures determinants of motivation with a series of psychometric scales in a health worker survey, examining the individual (e.g. self-efficacy, desire for achievement), organizational (e.g. management support, financial rewards), and external level determinants (relations with the community/patients) (see (Franco et al. 2002; Mbindyo et al. 2009 b; Mutale et al. 2013; Morrison et al. 2015; Chandler et al. 2009). JB and AK used this approach to examine the effects of primary care reforms on motivation composition and levels in Tanzania. Also in Tanzania, Leonard and Masatu (2010) made use of the Hawthorne effect (i.e. performance impact of being observed) to investigate health workers’ intrinsic motivation. In choosing a motivation measure, it is important to consider whether and how a programme is likely to affect motivation and how this would affect worker performance. This paper outlines the steps in measuring health worker motivation with Likert-type psychometric scales, as part of surveys, and in analysing such data, using examples from our respective research and the wider literature.

Open in a separate window

Figure 1.

Conceptualizing motivation. This figure is adapted from Franco et al. 2002 to convey the determinants and outcomes of motivation and the dimensions of motivation within a multi-dimensional framework. M1–M5 are factors, which represent different dimensions of motivation. Within self-determination theory, these could be the following: M1—motivation factor 1 (e.g. integrated motivation), M2—motivation factor 2 (e.g. identified motivation), M3—motivation factor 3 (e.g. introjected motivation), M4—motivation factor 4 (e.g. external regulation), and M5—motivation factor 5 (amotivation) Tremblay et al. 2009.

Step 2: developing and pre-testing a tool

Having selected a conceptualization of motivation, the first step in developing a survey tool is to identify a set of questions to measure motivation, referred to as a measurement scale (DeVellis 2012; Fowler 2009). If the aim is to understand the composition of motivation, then it is helpful to anticipate potential motivation dimensions with reference to theory and the intervention in question (Prytherch et al. 2012). Focus group discussions or in-depth interviews with health workers can also help identify dimensions and ensure appropriate communication of these concepts in the local language (Box 3) (e.g. Agyepong et al. 2004; Sacks et al. 2015). Once relevant dimensions have been identified, these need to be formulated as questionnaire items (i.e. statements or questions). A good first step is to review existing scales (see Mbaruku et al. 2014; Hotchkiss et al. 2015; Bonenberger et al. 2014; Inceoglu et al. 2012 and Supplementary Annex S1) and to decide on positive or negative wording, response scales and the number of response options (Box 4). It is recommended to include a minimum of three items per dimension (Little et al. 1999; Guilford 1952), although with a new scale, 4–5 items are recommended as some items may not perform well. To enable subsequent validity checks (Step 5), it is also important to collect data on variables that are expected to be related to motivation or motivation dimensions, such as motivational outcomes, e.g. intention to quit or organizational commitment (Hagopian et al. 2009; Bonenberger et al. 2014), or measures of the knowledge practice gap (Leonard and Masatu 2010) or other performance measures. It is also important to collect data on variables which might influence provider responses to items (health worker or facility level characteristics) (Chandler et al. 2009; Mbaruku et al. 2014; Hotchkiss et al. 2015; Franco et al. 2004). Researchers also need to decide on the mode of survey administration (see Box 5). As with all surveys, it is recommended to pre-test the motivation measurement tool with a small sample of health workers (see Prytherch et al. 2012) and to proceed through Steps 4 and 5.

Step 3: sample size considerations and sampling

Sample size is a further consideration prior to survey administration. The techniques used to assess the validity of the motivation measure (Step 5) require certain minimum sample sizes, dependent on the number of dimensions, items and other factors (Kline 2010). Commonly used rules of thumb for factor analytical techniques are ‘no less than100 observations’ (Gorsuch 1983; Kline 1979; Kline 2010; MacCallum et al. 1999), with 50 observations often considered the absolute minimum (de Winter et al. 2009) for exploratory factor analysis; and 200 observations the minimum for confirmatory factor analysis (DeCoster 1998). If sub-group analysis is planned (e.g. comparing motivation between different cadres of health worker), these sample sizes should be achieved for each sub group. As in any other study, sample size requirements also depend on the planned substantive analyses (Step 7). The sample size requirements to analyse motivation determinants depends on the type of model used, with a standard linear regression model having lower sample size requirements than structural equation models (Wolf EJ et al. 2013). When considering the impact of a programme on motivation, power considerations are also important, but estimations of effect size tend to be difficult given that they are highly dependent on the motivation measure itself. Often, motivation surveys are administered to health workers who are present on the day of the facility visit. These health workers are likely more motivated than their counterparts who are not present at facilities, and it would be important, where possible, to also make provisions to interview health workers who are absent from facilities on the day of the visit.

Step 4: exploratory data analysis

Once the data have been collected, it is important to start with an exploration of the data, estimating mean and median scores and distributions for each item, and checking for missing data. The empirical literature on health worker motivation has tended to analyse Likert responses as continuous variables, given that the underlying motivation construct is assumed to be continuous. However, there has been some debate as to whether this approach is appropriate given the ordered categorical nature of Likert scales, though there is evidence it may make little difference in practice—see (Carifio and Perla 2008; 2007), for more discussion of this point.

A high level of missing values may indicate that an item was not well understood by respondents (Raykov and Marcoulides 2011; Little 1992). Where missing values exceed 10% researchers should weigh the option of dropping the item against maintaining measurement consistency across respondents. It is generally recommended to consider dropping items where >80% of respondents provide the same answer to a question as such items have little discriminatory value (Streiner et al. 2008). Direction of response for each item, particularly for those that were negatively worded, should also be checked for plausibility.

Step 5: assessing validity of motivation measures

Before using motivation measurements in core analyses, researchers should ensure the measures are valid or that they measured what was intended (DeVellis 2012; Fowler 2009).

If motivation is considered to be multidimensional, the first step in validating the measure is to determine the composition of motivation, or the underlying dimensions, to confirm or modify initial hypotheses. This is typically done with factor analytical techniques; either exploratory or confirmatory (see Box 6). Before doing so, it is important to check the factorability (or reducibility) of the data (e.g. inspect item correlation matrix; Bartlett test of sphericity test; Kaiser–Meyer–Olkin test (Yong and Pearce 2013)).

Exploratory factor analysis

When constructing new scales and/or applying them to novel contexts, researchers are often not entirely sure how many and which motivation dimensions the scale items measure. Unlike CFA, EFA does not impose any a priori assumptions on the number of motivation factors, and the assignment of items to factors. Rather, EFA is used to identify meaningful dimensions of motivation, and to determine which items measure which dimension, on the basis of respondents’ answer patterns to the scale items. EFA is sometimes used to generate a theory about the relevant dimensions of motivation that are then used in a CFA. With sufficient sample size, EFA can be performed on one part of the data, and the generalizability of the extracted factors can be determined using CFA on the other (Raykov and Marcoulides 2011).

Factor extraction

A variety of statistical approaches can be used to extract factors using EFA. Principal component analysis (PCA) and principal axis factoring (PAF) are the most common (Williams et al. 2012; DeVellis 2012). Rotation is used to simplify and clarify the results of EFA facilitating the identification of factors. There are two main types of rotation: orthogonal and oblique, with the main difference being that the latter allows for some correlation between factors whereas the former does not. The former has been widely used because it is believed to be simpler (e.g. Chandler et al. 2009). However, as motivation dimensions are unlikely to be unrelated (e.g. there will be some association between different factors, such as drug availability and supervision or management involvement in facilities), the latter approach is preferable.

Deciding how many factors and which items to retain

The full list of factors resulting from an unrestricted EFA will correspond to the number of items included. The researcher must decide how many to retain. This decision will be based in part on theoretical considerations: how many dimensions is it reasonable to expect? and whether the resulting factors can be readily named and described. The following can also help determine the number of factors: a common rule of thumb is to retain factors that have eigenvalues over 1 (the Kaiser criterion) (Hayton et al. 2004; Kaiser 1960); visually examine eigenvalue plots for the natural bend or break point in the data where the curve flattens out (Figure 2) (Cattell 1966; Chandler et al. 2009); examine the total variance explained (aim to explain 50–75% with the least number of factors). In practice if using the ‘factor’ command in Stata, there are different cut-off values for factor retention that are built into the software depending on the method of factor extraction selected.¹ We encourage researchers to think critically about how many factors make sense in their context rather than to blindly accept these arbitrary cut-offs.

Open in a separate window

Figure 2.

Scree plot for survey data collected in Tanzania. Based on visual inspection alone, 5 factors appear to be the turning point after which the plot levels off (though it does so again at 8). However, using the Kaiser criterion (retaining factors with an eigen value of 1 or more), 3 factors would be retained

It is also important to examine the factor loadings for each item. In EFA, all items will load on all factors to some degree. The aim is to determine which items are most indicative of which factors, based on the degree of factor loading, with 0.3 (Tabachnick and Fidell 2007) and 0.4 being commonly used (Chandler et al. 2009) as cut-off values for ‘substantive loadings’. Higher thresholds are recommended for small sample sizes. The ideal scenario is that each item has a substantive loading on only one factor and is conceptually close to the other items with substantive loadings on that factor. However, this is often not the case, and researchers will have to decide whether for instance to define a different number of factors or to eliminate items with low factor loadings. EFA is invariably an iterative process, as results change with the number of factors retained and items included.

Interpreting and naming factors

When interpreting and naming factors, it is important to refer back to the exact wording of the scale items and the aspects of motivation they were designed to measure. Often, the interpretation of a factor is relatively straightforward from the items loading on it. For example, in Tanzania, the following three items: availability of drugs, supplies and equipment at the facility, had substantive loadings on the same factor. All three clearly pertained to the ‘work environment’. It is possible that some items may not fit semantically with the factor they load on. For example, in the same Tanzanian study, 5 items loaded substantively on another factor. Four of the items were related to ‘management and supervision at the facility’, but one item did not appear to fit with that definition: ‘relationship with local leaders in the community’. One explanation for such cases is a divergence between respondent and researcher interpretation of an item. In this case, respondents may have considered community leaders together with managers given their joint involvement in facility governing committees. Interviews/focus groups can be used to shed light on respondent understanding and if the item is found to be related to the dimension it can be retained. Another reason for ‘lone items’ is that they are related to a sub-dimension of motivation that did not emerge as a separate multi-item factor simply because the scale contained only one item pertaining to it. In such cases, researchers must decide whether to keep the item as a (psychometrically suboptimal) single-item measure, or whether to drop it. In some cases, clusters of items that do not fit well together may be a statistical artefact: EFA groups items based on response patterns without considering how these items relate to each other semantically. The idea is that people respond similarly to items of similar content, because these items tap into the same construct. However, this may not be the case. For instance, a person might feel equally motivated by intrinsic and extrinsic factors and thus assign similar numeric values to related items. In an EFA, we might then end up with a one-factor solution combining extrinsic and intrinsic motivation items. However, this does not mean that extrinsic and intrinsic motivation is the same. Therefore, it is important to interpret factor analysis results together with theory and knowledge of the context.

Step 6: measurement reliability

Reliability refers to the extent to which the measurement scale produces similar results under similar conditions (DeVellis 2012). Internal consistency based on Cronbach’s α coefficient, the average correlation between items, is the most widely used statistic to assess measurement reliability. In recent years, however, psychometricians have cautioned against the use of α, for conceptual reasons (Yang and Green 2011) and due to its vulnerability to outliers, non-normal data, small number of items, and low variability in total scores (Greer et al. 2006; Cortina 1993; Sijtsma 2009; Cronbach and Shavelson 2004). Factor-analysis-based estimation of reliability is now preferred to Chronbach's alpha (Yang and Green 2011; Raykov and Marcoulides 2011). When estimating Chronbach's alpha, it is recommended to use the polychoric correlation matrix instead of a Pearson correlation matrix (Gadermann et al. 2012; Dale 2014). For multidimensional measures of motivation, Chronbach's alpha should be estimated for each dimension (Cortina 1993). A typical recommended cut-off level for α has been 0.70, however, as this parameter depends on the number of items among other things, this value should be treated cautiously.

Test–retest measures the degree to which health workers would provide the same responses to items in a repeat survey. In public health studies where the scale development is not the central focus, test–retest validation studies are unfortunately often not feasible for practical reasons. If a retest is possible, it is important to choose the time delay between test and retest in a way that the underlying construct measured with the scale can be assumed to have remained stable.

When motivation is to be compared across different subgroups (e.g. women vs men, doctors vs other cadres, different language groups, across countries), the scale should be tested for equal measurement properties across subgroups. If measurement invariance is not established, there is a risk that subgroup differences are not due to differences in motivation but to differences in the performance of the measure in the subgroups (Vandenberg and Lance 2000). Measurement invariance testing is usually done in a CFA framework (see Supplementary Annex S3).

Step 7: core analysis

Once validity and reliability are established, the motivation measure can be used within analysis, depending on the objective of the study. If the objective is to describe motivation levels, item responses can be combined into a composite score, typically calculated as the arithmetic mean of health worker responses. Means can be calculated either as unweighted means, i.e. all items have the same weight, or one can give more weight to some items than to others, which may be preferable if the EFA and/or CFA results show substantially different factor loadings between items. In such cases, the loadings can serve as weights. If the motivation measure was found to be multidimensional at the EFA/CFA stage, scores are calculated separately for each dimension. Researchers have sometimes also estimated an overall motivation score by combining item scores across dimensions, e.g. Gow et al. 2013; Bhatnagar and George 2016; Hagopian et al. 2009; Mbindyo et al. 2009 b. Where dimensions are deemed conceptually distinct, such practice makes limited sense and risks evening out important differences across dimensions. If researchers wish to capture overall motivation, a related item can be included in the measurement scale (e.g. ‘Overall, how motivated do you feel?’).

If the objective is to understand determinants or consequences of motivation, or changes in motivation over time, there are two main analytical options: using composite scores (‘manifest variables’), or using a latent variable approach where the relationship between motivation and other variables of interest are inferred directly from the scale items, without the estimation of composite scores.

Composite scores can be used as predictor or outcome variables in a regression model. However, much of the variance contained in the individual items is ‘averaged out’ by the calculation of a mean composite score (Borsboom 2006; Skrondal and Laake 2001).

With a latent variable approach in SEM, associations between motivation and other variables of interest are directly estimated from the items via the latent variable/s. This approach provides more accurate estimates of the relationship between motivation and other variables as all information contained in the dataset is preserved. However, large structural equation models are complex and difficult to handle and have large sample size requirements.

The research in Tanzania reported in this paper was funded by the European Union: Funded under: FP7-HEALTH: Project reference: 261349, as was the time of J.B., A.K., J.G., F.M., K.O. Examples from Burkina Faso reported in the paper were drawn from the impact evaluation of the Health Sector Results-Based Financing Program, funded by the World Bank through the Health Results Innovation Trust Fund. Contributions made by E.D. are based on her PhD dissertation submitted to Johns Hopkins University under the supervision of David H. Peters and with inputs and guidance from Qian-Li Xue, Sara Bennett, Kitty Chan, and Saifuddin Ahmed. The UK Department for International Development (DFID) as part of the Consortium for Research on Resilient and Responsive Health Systems (RESYST) supported the time of J.L., E.D. and JB writing the paper. J.B.’s time was also supported by the Research Council of Norway. The views expressed and information contained in it are not necessarily those of or endorsed by the funders, which can accept no responsibility for such views or information or for any reliance placed on them.

Conflict of interest statement. None declared.