Tracking health commodity inventory and notifying stock levels via mobile devices: a mixed methods systematic review

Abstract Background Health systems need timely and reliable access to essential medicines and health commodities, but problems with access are common in many settings. Mobile technologies offer potential low‐cost solutions to the challenge of drug distribution and commodity availability in primary healthcare settings. However, the evidence on the use of mobile devices to address commodity shortages is sparse, and offers no clear way forward. Objectives Primary objective To assess the effects of strategies for notifying stock levels and digital tracking of healthcare‐related commodities and inventory via mobile devices across the primary healthcare system Secondary objectives To describe what mobile device strategies are currently being used to improve reporting and digital tracking of health commodities To identify factors influencing the implementation of mobile device interventions targeted at reducing stockouts of health commodities Search methods We searched CENTRAL, MEDLINE Ovid, Embase Ovid, Global Index Medicus WHO, POPLINE K4Health, and two trials registries in August 2019. We also searched Epistemonikos for related systematic reviews and potentially eligible primary studies. We conducted a grey literature search using mHealthevidence.org, and issued a call for papers through popular digital health communities of practice. Finally, we conducted citation searches of included studies. We searched for studies published after 2000, in any language. Selection criteria For the primary objective, we included individual and cluster‐randomised trials, controlled before‐after studies, and interrupted time series studies. For the secondary objectives, we included any study design, which could be quantitative, qualitative, or descriptive, that aimed to describe current strategies for commodity tracking or stock notification via mobile devices; or aimed to explore factors that influenced the implementation of these strategies, including studies of acceptability or feasibility. We included studies of all cadres of healthcare providers, including lay health workers, and others involved in the distribution of health commodities (administrative staff, managerial and supervisory staff, dispensary staff); and all other individuals involved in stock notification, who may be based in a facility or a community setting, and involved with the delivery of primary healthcare services. We included interventions aimed at improving the availability of health commodities using mobile devices in primary healthcare settings. For the primary objective, we included studies that compared health commodity tracking or stock notification via mobile devices with standard practice. For the secondary objectives, we included studies of health commodity tracking and stock notification via mobile device, if we could extract data relevant to our secondary objectives. Data collection and analysis For the primary objective, two authors independently screened all records, extracted data from the included studies, and assessed the risk of bias. For the analyses of the primary objectives, we reported means and proportions where appropriate. We used the GRADE approach to assess the certainty of the evidence, and prepared a 'Summary of findings' table. For the secondary objective, two authors independently screened all records, extracted data from the included studies, and applied a thematic synthesis approach to synthesise the data. We assessed methodological limitation using the Ways of Evaluating Important and Relevant Data (WEIRD) tool. We used the GRADE‐CERQual approach to assess our confidence in the evidence, and prepared a 'Summary of qualitative findings' table. Main results Primary objective For the primary objective, we included one controlled before‐after study conducted in Malawi. We are uncertain of the effect of cStock plus enhanced management, or cStock plus effective product transport on the availability of commodities, quality and timeliness of stock management, and satisfaction and acceptability, because we assessed the evidence as very low‐certainty. The study did not report on resource use or unintended consequences. Secondary objective For the secondary objectives, we included 16 studies, using a range of study designs, which described a total of eleven interventions. All studies were conducted in African (Tanzania, Kenya, Malawi, Ghana, Ethiopia, Cameroon, Zambia, Liberia, Uganda, South Africa, and Rwanda) and Asian (Pakistan and India) countries. Most of the interventions aimed to make data about stock levels and potential stockouts visible to managers, who could then take corrective action to address them. We identified several factors that may influence the implementation of stock notification and tracking via mobile device. These include challenges tied to infrastructural issues, such as poor access to electricity or internet, and broader health systems issues, such as drug shortages at the national level which cannot be mitigated by interventions at the primary healthcare level (low confidence). Several factors were identified as important, including strong partnerships with local authorities, telecommunication companies, technical system providers, and non‐governmental organizations (very low confidence); availability of stock‐level data at all levels of the health system (low confidence); the role of supportive supervision and responsive management (moderate confidence); familiarity and training of health workers in the use of the digital devices (moderate confidence); availability of technical programming expertise for the initial development and ongoing maintenance of the digital systems (low confidence); incentives, such as phone credit for personal use, to support regular use of the system (low confidence); easy‐to‐use systems built with user participation (moderate confidence); use of basic or personal mobile phones to support easier adoption (low confidence); consideration for software features, such as two‐way communication (low confidence); and data availability in an easy‐to‐use format, such as an interactive dashboard (moderate confidence). Authors' conclusions We need more, well‐designed, controlled studies comparing stock notification and commodity management via mobile devices with paper‐based commodity management systems. Further studies are needed to understand the factors that may influence the implementation of such interventions, and how implementation considerations differ by variations in the intervention.


A B S T R A C T Background
Health systems need timely and reliable access to essential medicines and health commodities, but problems with access are common in many settings. Mobile technologies o er potential low-cost solutions to the challenge of drug distribution and commodity availability in primary healthcare settings. However, the evidence on the use of mobile devices to address commodity shortages is sparse, and o ers no clear way forward.

Primary objective
To assess the e ects of strategies for notifying stock levels and digital tracking of healthcare-related commodities and inventory via mobile devices across the primary healthcare system

Secondary objectives
To describe what mobile device strategies are currently being used to improve reporting and digital tracking of health commodities To identify factors influencing the implementation of mobile device interventions targeted at reducing stockouts of health commodities

Search methods
We searched CENTRAL, MEDLINE Ovid, Embase Ovid, Global Index Medicus WHO, POPLINE K4Health, and two trials registries in August 2019. We also searched Epistemonikos for related systematic reviews and potentially eligible primary studies. We conducted a grey literature search using mHealthevidence.org, and issued a call for papers through popular digital health communities of practice. Finally, we conducted citation searches of included studies. We searched for studies published a er 2000, in any language.

Library
Trusted evidence. Informed decisions. Better health.

Selection criteria
For the primary objective, we included individual and cluster-randomised trials, controlled before-a er studies, and interrupted time series studies. For the secondary objectives, we included any study design, which could be quantitative, qualitative, or descriptive, that aimed to describe current strategies for commodity tracking or stock notification via mobile devices; or aimed to explore factors that influenced the implementation of these strategies, including studies of acceptability or feasibility.
We included studies of all cadres of healthcare providers, including lay health workers, and others involved in the distribution of health commodities (administrative sta , managerial and supervisory sta , dispensary sta ); and all other individuals involved in stock notification, who may be based in a facility or a community setting, and involved with the delivery of primary healthcare services.
We included interventions aimed at improving the availability of health commodities using mobile devices in primary healthcare settings. For the primary objective, we included studies that compared health commodity tracking or stock notification via mobile devices with standard practice. For the secondary objectives, we included studies of health commodity tracking and stock notification via mobile device, if we could extract data relevant to our secondary objectives.

Data collection and analysis
For the primary objective, two authors independently screened all records, extracted data from the included studies, and assessed the risk of bias. For the analyses of the primary objectives, we reported means and proportions where appropriate. We used the GRADE approach to assess the certainty of the evidence, and prepared a 'Summary of findings' table. For the secondary objective, two authors independently screened all records, extracted data from the included studies, and applied a thematic synthesis approach to synthesise the data. We assessed methodological limitation using the Ways of Evaluating Important and Relevant Data (WEIRD) tool. We used the GRADE-CERQual approach to assess our confidence in the evidence, and prepared a 'Summary of qualitative findings' table.

Primary objective
For the primary objective, we included one controlled before-a er study conducted in Malawi.
We are uncertain of the e ect of cStock plus enhanced management, or cStock plus e ective product transport on the availability of commodities, quality and timeliness of stock management, and satisfaction and acceptability, because we assessed the evidence as very low-certainty. The study did not report on resource use or unintended consequences.

Secondary objective
For the secondary objectives, we included 16 studies, using a range of study designs, which described a total of eleven interventions. All studies were conducted in African (Tanzania, Kenya, Malawi, Ghana, Ethiopia, Cameroon, Zambia, Liberia, Uganda, South Africa, and Rwanda) and Asian (Pakistan and India) countries.
Most of the interventions aimed to make data about stock levels and potential stockouts visible to managers, who could then take corrective action to address them.
We identified several factors that may influence the implementation of stock notification and tracking via mobile device.
These include challenges tied to infrastructural issues, such as poor access to electricity or internet, and broader health systems issues, such as drug shortages at the national level which cannot be mitigated by interventions at the primary healthcare level (low confidence). Several factors were identified as important, including strong partnerships with local authorities, telecommunication companies, technical system providers, and non-governmental organizations (very low confidence); availability of stock-level data at all levels of the health system (low confidence); the role of supportive supervision and responsive management (moderate confidence); familiarity and training of health workers in the use of the digital devices (moderate confidence); availability of technical programming expertise for the initial development and ongoing maintenance of the digital systems (low confidence); incentives, such as phone credit for personal use, to support regular use of the system (low confidence); easy-to-use systems built with user participation (moderate confidence); use of basic or personal mobile phones to support easier adoption (low confidence); consideration for so ware features, such as two-way communication (low confidence); and data availability in an easy-to-use format, such as an interactive dashboard (moderate confidence).

Authors' conclusions
We need more, well-designed, controlled studies comparing stock notification and commodity management via mobile devices with paperbased commodity management systems. Further studies are needed to understand the factors that may influence the implementation of such interventions, and how implementation considerations di er by variations in the intervention. We are uncertain of the effect of this approach on stockout of cotrimoxazole because it is supported by very low-certainty evidence. We are uncertain of the effect of this approach on stockout of artemether-lumefantrine because it is supported by very lowcertainty evidence.

Proportion of healthcare workers who reported a stockout of drugs in the last 30 days
(stockout of oral rehydration salts (ORS)  We are uncertain of the effect of this approach on stockout of oral rehydration salts because it is supported by very low-certainty evidence. We are uncertain of the effect of this approach on stockout of zinc because it is supported by very low-certainty evidence.

Quality of stock management
Quality of data about stock management (assessed by the extent to which HSAs (intervention group participants) sent messages about the stocks on hand for all the products they managed) In the intervention group, an average of 85% (N = 393) of the health surveillance assistants (HSA) who managed relevant medicines reported completely on stock levels.
This outcome was not assessed in the comparison group. We are uncertain of the effect of this approach on quality of data about stock management because it is supported by very lowcertainty evidence.

Timeliness of stock management
Time between stock-level reporting and appropriate action (measured over an 18-month period (January 2012to June 2013 In the intervention group, health facilities took an average of 12.8 days to fulfil an order requested by the health surveillance assistants (lead time).
This outcome was not assessed in the comparison group. We are uncertain of the effect of this approach on the timeliness of stock management because it is supported by very low-certainty evidence.

Satisfaction and acceptability
Provider acceptability and satisfaction (proportion of participants who reported using the digital intervention) In the intervention group, the proportion of participants who reported using the digital intervention (cStock) as the primary means for ordering health products was 97% (N = 81).
This outcome was not assessed in the comparison group. We are uncertain of the effect of this approach on provider satisfaction with stock management because it is supported by very low-certainty evidence.

Resource use
No studies were identified that reported on this outcome

Unintended consequences
No studies were identified that reported on this outcome *The basis for the assumed risk (e.g. the median control group risk across studies) is provided in footnotes. The corresponding risk (and its 95% confidence interval) is based on the assumed risk in the comparison group and the relative effect of the intervention (and its 95% CI). a Shieshia 2014. Published and unpublished data. Study conducted in primary healthcare setting b Downgraded two levels for very serious concerns regarding risk of bias: unclear random sequence generation, allocation concealment, and blinding of participants not feasible given the intervention, unclear blinding of outcomes and incomplete outcome reporting c Downgraded one level for imprecision: small sample size d For this outcome, the number of study participants was based on a di erent study sample to the one used for the other outcomes. These data come from ongoing data (backend data in a digital system), and comprise of all the health workers who ever reported on stock levels e Non-comparable results, thus downgraded to very low Summary of findings 2. Primary objective: mobile stock notification with e ective product transport compared to standard care Mobile stock notification (cStock) with effective product transport (EPT) compared to standard care in primary healthcare settings Patient or population: healthcare workers and other health professionals involved in commodity and stock management Setting: primary healthcare settings in Malawi Intervention: mobile stock notification with effective product transport (cStock + EPT), which involved providing health surveillance assistants (HSA) with training and tools for bicycle maintenance We are uncertain of the effect of this approach on quality of data about stock management because it is supported by very lowcertainty evidence

Timeliness of stock management
Time between stock-level reporting and appropriate action (Measured over an 18-month period: January 2012to June 2013 In the intervention group, health facilities took an average of 26 days to fulfil an order requested by the health surveillance assistants (lead time).
This outcome was not assessed in the comparison group.
We are uncertain of the effect of this approach on the timeliness of stock management because it is supported by very low-certainty evidence

Provider acceptability and satisfaction
In the intervention group, the proportion of participants who reported using the digital intervention (cStock) as We are uncertain of the effect of this approach on provider satis-(Proportion of participants who reported using the digital intervention) the primary means for ordering health products was 91% (N = 78).
This outcome was not assessed in the comparison group.
faction with stock management because it is supported by very low-certainty evidence

Resource use
No studies were identified that reported on this outcome

Unintended consequences
No studies were identified that reported on this outcome The basis for the assumed risk (e.g. the median control group risk across studies) is provided in footnotes. The corresponding risk (and its 95% confidence interval) is based on the assumed risk in the comparison group and the relative effect of the intervention (and its 95% CI).
95% CI: 95% confidence interval; RR: risk ratio; CBA: controlled before-after trial GRADE Working Group grades of evidence High certainty. Further research is very unlikely to change our confidence in the estimate of effect. Moderate certainty. Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate. Low certainty. Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate. Very low certainty. We are very uncertain about the estimate. a Shieshia 2014 published and unpublished data. Study conducted in primary healthcare setting. b Downgraded two levels for very serious risk of bias concerns: unclear random sequence generation, allocation concealment, and blinding of participants not feasible given the intervention, unclear blinding of outcomes and incomplete outcome reporting c Downgraded one step for imprecision: small sample size d For this outcome, the number of study participants is based on a di erent study sample to the one used for the other outcomes. These data come from ongoing data (backend data in a digital system), and comprise all the health workers who ever reported on stock levels. 1 Infrastructural issues, such as challenges in charging phones, uploading and transmitting data, and loss of data due to poor access to electricity and poor or non-existent internet connectivity were identified as key barriers to implementation. User-friendly systems, built with user participation with easy-to-use interfaces were considered important to implementation.

10
The use of basic mobile phones or personal phones by health workers reduced challenges with data coverage and expense, and supported easier adoption of the intervention due to familiarity with the phones. Cochrane Database of Systematic Reviews

B A C K G R O U N D
Access to medicines and other health commodities remains one of the most serious global public health problems and results in critical gaps in delivery of healthcare services.

Description of the condition
Reliable availability of health commodities is fundamental to diagnosing and treating illnesses in primary healthcare settings. Health commodities include health products, health and medical supplies, and other items that may be needed for the provision of health services, including medicines; vaccines; medical supplies, such as contraceptives, dressings, needles, and syringes; and laboratory and diagnostic consumables (Tran 2015; WHO 2017). The World Health Organization (WHO) Global Strategy for Women's and Children's Health highlights the importance of equitable access to life-saving medicines and other health commodities (WHO 2010). A hallmark of functioning health systems is the availability of essential medicines in adequate amounts, appropriate dosage forms, and assured quality -at a price that is a ordable for the local community (Tran 2015; WHO 2016). However, stockouts of critical medical commodities, such as medicines, are widespread, especially in low-and middle-income countries (LMICs).
At least one third of the world's population does not have regular access to medicines, which makes health care highly inequitable (WHO 2011). A survey of the national AIDS programmes in 12 countries, by the Pan American Health Organization (PAHO), found that between January 2011 and April 2012, over 67% of the countries reported experiencing stockouts of at least one drug, lasting an average of 40 days each (Sued 2011). Another study, in Kenya, reported that over 75% of health facilities had shortages of one component of the combination of drugs used to treat malaria, while one in four reported a lack of all related drugs (Kangwana 2009). This lack of access to critical drugs, caused by a stockout, has profound e ects on the ongoing treatment of diseases. A study in Côte d'Ivoire reported that people who experienced interruptions in their HIV treatment, caused by drug shortages, were twice as likely to permanently discontinue treatment or die (Pasquet 2010). Drug stockouts have been linked to increases in morbidity and mortality across several disease states in low-resource settings, including malaria (Chuma 2010), HIV (Pasquet 2010), and the prevention or treatment of pregnancy complications (Hill 2006).
Lack of access to medicines and other health commodities is o en symptomatic of broader systemic problems. For example, access to medicines is determined by rational use of medicines, a ordable pricing, sustainable financing, and reliable health and supply systems (WHO 2004;WHO 2015). A reliable medicine supply system should include appropriate procurement and distribution. A good distribution system ensures timely availability of medicines across all levels of the healthcare system and prevention of stockouts (WHO 2017).

Description of the intervention
The rapid global expansion of mobile technology has provided a potential low-cost solution to the challenge of drug distribution and stockouts. Plummeting costs of mobile handsets and services have made mobile phone technology accessible to people living in rural and underserved areas. Mobile interventions may address stockouts of medicines and health commodities primarily through two strategies: supply chain management, and assessment and reporting of essential commodities (Mehl 2017 [pers comm]).
Supply chain management involves approaches for monitoring and reporting stock levels, consumption and distribution of medical commodities, as well as approaches to analyse and project usage of medical commodities. This can include the use of communication systems, such as short message service (SMS) and data dashboards, to manage and report on supply levels of medical commodities. Some specific examples where mobile tools may be used to improve supply chain management include tracking inventory of health commodities, notifying stock levels of health commodities, monitoring cold-chain sensitive commodities, and managing distribution of health commodities.
Digital approaches for assessment and reporting of essential commodities are o en used for reporting and tracking the authenticity and quality of medical commodities. This can include using mobile functions, such as barcode readers and short message service (SMS) communication to validate an authentication code on the drug packaging (Frøen 2016), as well as to report on adverse drug e ects. Specific examples where mobile devices may be used for assessment and reporting of commodities include reporting on stock levels, reporting counterfeit or substandard drugs, reporting adverse drug interactions, and registering licensed drugs and health commodities.

How the intervention might work
Mobile devices are being used for supply chain management to improve data visibility, improve decision-making, and help to address the availability of commodities. There is a substantial amount of variation in how such systems might operate. At the most basic level, interventions may involve citizens reporting counterfeit medicines, using SMS sent to a toll-free phone number. Such interventions may use a mobile product authentication (MPA) application, or a barcode that allows consumers to text a set of unique numbers to a toll-free phone number, to verify if a medicine is authentic. In response, consumers may receive a SMS that indicates the legitimacy of the medicine.
Other interventions may involve frontline healthcare workers or healthcare administrators in primary healthcare settings using mobile devices to collect data on stock levels, so that data can be instantly digitised and used to predict and prevent stockouts, and respond to drug shortages. More comprehensive interventions may aim to develop a technology-based system for reporting of drug stock levels, and change the culture around the use of data (on stock levels), and accountability for responding to projected shortages. For example, cStock is an open-source internet-accessible logistics management information system that targets the availability of health commodities at the community level, in Malawi (Shieshia 2014). Health surveillance assistants (HSAs), who typically deliver primary healthcare services in the community, send information about the amount of medicine stocks they have on hand, via a text message to a toll-free number. The logistics management system automatically collates this data from multiple HSAs, calculates the total quantities of commodities needed, and sends a text message to the HSAs when the medicines are available at the nearest health centre. These data are also available on a internet-accessible dashboard, with simple, easy-touse reports, showing stock levels, HSA reporting rates, and alerts from central and district level health managers. Making real-time

Why it is important to do this review
There is rapid progress in the use of mobile devices to address systemic challenges in the delivery of healthcare services. Despite the exponential growth of mobile device-based interventions and their potential, there remain several unanswered questions about the e ectiveness of such interventions. The reliable availability of essential medicines and health commodities is foundational to a responsive health system, and an area that is of much interest to governments, especially in LMICs. However, the evidence on the use of mobile devices to address drug and commodity shortages is sparse, and o ers no clear way forward. We are not aware of any existing systematic reviews that assess the e ectiveness of strategies to improve stock notification, through either digital or non-digital approaches. The WHO recently published guidelines to inform investments in digital health applications for strengthening health systems (WHO 2019). Through a consultative process, assessing the impact of mobile interventions to address stockouts was identified as one of the several areas to be included in the guidelines. This Cochrane Review is one of a suite of reviews that contributed to these guidelines. We aimed to assess the e ectiveness of using mobile devices to address stockouts of drugs and essential health commodities, and the acceptability, resource use, and unintended consequences of such interventions.

O B J E C T I V E S Primary
• To assess the e ects of strategies for notifying stock levels and digitally tracking healthcare-related commodities and inventory, via mobile devices, across the primary healthcare system.

Secondary
• To describe what mobile device strategies are currently being used to improve reporting and digital tracking of health commodities; • To identify factors influencing the implementation of mobile device interventions targeted at reducing stockouts of health commodities.

Primary objective
For the review's primary objective, we included these study designs: • Randomised trials; • Non-randomised trials; • Controlled before-a er studies, provided they had at least two intervention sites and two control sites; • Interrupted time series studies, if there was a clearly defined point in time when the intervention occurred, and at least three data points before and three a er the intervention.
We included published studies, conference abstracts, and unpublished data. We included studies, regardless of their publication status, or language of publication.

Secondary objectives
For the review's secondary objectives, we included any studies that used descriptive, qualitative, or quantitative methods to describe interventions that were aimed at improving stockouts of health commodities.

Types of participants
For the review's primary and secondary objectives, we included studies with these participants: • All cadres of healthcare providers (i.e. professionals, paraprofessionals, and lay health workers), or others involved in the distribution of health commodities, located at any level of the health system (e.g. administrative sta , managerial and supervisory sta in purchasing or distribution, or dispensary sta ); • Other individuals or groups involved in stock notification, monitoring, and tracking commodity inventories. These individuals or groups may be based in a primary healthcare facility or in the community, and must be involved in supporting the delivery of primary healthcare services. • Clients or recipients of health services

Types of interventions
For the review's primary and secondary objectives, we included interventions that were aimed at improving the availability of health commodities, including medicines and other medical supplies, using mobile devices for the delivery of primary healthcare services in healthcare facilities or in the community, if they involved one or both of the following: • Strategies for tracking health commodity inventory using mobile devices. Tracking health commodity inventory may have involved the use of databases and dashboards to manage the availability of health commodities and project availability of medical supplies. While some aspects of commodity tracking might have involved mobile devices, the data may have been linked to a logistics management information system (LMIS) or supply chain management system, where inventory levels and historic data were maintained on desktops; • Notification of stock levels conducted via mobile devices.
This may have involved the transmission of information on stock levels by health workers within healthcare facilities or by members of the community, to alert higher-level facilities about potential stock shortages. For example, health workers at facilities or dispensaries may have used text messaging, short message service (SMSa), or unstructured supplementary service data (USSD)-based systems to notify district or central authorities about stock levels. In some interventions of interest, notification of stock levels using mobile phones may have been a component of a broader strategy for tracking health commodities.

Library
Trusted evidence. Informed decisions. Better health.

Cochrane Database of Systematic Reviews
By primary healthcare services, we meant a combination of the following: • The first contact point of healthcare (Awofeso 2004), including care delivered at an individual or community level, or both, by individual healthcare providers or teams of providers, and intended to bring care to where people worked and lived (Muldoon 2006), or the co-ordination or provision of continuity of care, or both (WHO 2008); • Any rehabilitative, therapeutic, preventive, or promotional healthcare (Global Health Watch 2011).
The key comparison for this review was tracking commodity inventory and notifying stock levels via mobile devices compared with standard practice (i.e. non-digital strategies or no intervention).
We excluded: • Studies that focused on cold chain management only, and did not report on stock levels of the vaccines; • Studies where commodity tracking and notification of commodities was conducted on stationary computers or laptops only.
Where tracking or notification via mobile device, or both, was delivered as part of a wider package, we included the study if we judged the mobile component to be the major component of the intervention.

Primary objective
For the review's primary objective, we included studies that assessed the following outcome measures: • Availability of commodities, measured, for instance, as decreased stockouts, lead time for drug supply, availability at point of care; • Quality of data about stock management (accuracy of data, completeness of data); • Timeliness of stock level reporting, and time between receipt and reporting data regarding commodity status and appropriate action; • Provider acceptability or satisfaction with the intervention, measured with a validated scale if available; • Resource use (e.g. human resources or time, including additional time spent by providers when managing or transitioning dual paper and digital reporting systems; training, supplies, and equipment); • Unintended consequences that may result in the intervention having adverse e ects (these could include: misreading or misinterpreting the data; transmitting inaccurate data, for instance through so ware formatting errors; interrupted workflow due to infrastructure constraints for battery recharge and network coverage; decreased motivation or trust in the system by health workers, if stock replenishment is not reliable; loss or misuse of mobile device).

Search methods for identification of studies
We started the search in 2000. This was based on the increased availability and penetration of mobile devices in LMICs from 2000 onwards (ITU 2015).

Electronic searches
An independent Information Specialist (JE) developed the search strategies in consultation with the review authors.
We searched the following databases for primary studies, from 2000 to the date of search: • Appendix 1 lists the search strategies we used to search all the databases. Search strategies were comprised of keywords and controlled vocabulary terms. We did not apply any limits on language.

Trial registries
We searched for ongoing trials in the following trial registries, and contacted authors for further information and data, if available: • WHO International Clinical Trials Registry Platform (ICTRP; www.who.int/ictrp; searched 7 August 2019); • US National Institutes of Health Ongoing Trials Register ClinicalTrials.gov (www.clinicaltrials.gov; searched 7 August 2019).
We searched Epistemonikos (www.epistemonikos.org; searched 27 January 2020) for relevant systematic reviews and potentially eligible primary studies. Additionally, the WHO issued a call for papers through popular digital health communities of practice, such as the Global Digital Health Network, to identify additional primary studies and grey literature.

Grey literature
We searched www.mhealthevidence.org for grey literature. The search portal for mhealthevidence.org was more limited; therefore, we reviewed the titles and abstracts of all contributed literature that was not referenced in MEDLINE Ovid (searched 15 August 2017; the database was discontinued in 2018).

Library
Trusted evidence. Informed decisions. Better health.

Selection of studies
A core team of two authors (NH and HB), with assistance where necessary from one additional review author (SA), were responsible for the selection of studies. We downloaded all titles and abstracts retrieved by electronic searching to a reference management database and removed duplicates. Two review authors (NH and HB) independently screened titles and abstracts for inclusion for the primary and secondary objectives. We retrieved the fulltext study reports and publications for titles and abstracts that were assessed as potentially eligible. Two review authors (NH and HB) for the primary objective, and one review author for the secondary objectives (SA), independently screened the fulltext, and identified studies for inclusion, and recorded reasons for exclusion of the ineligible studies. We resolved any disagreement through discussion; if required, we consulted a third review author.
We listed studies that initially appeared to meet the inclusion criteria but that we later excluded in the 'Characteristics of excluded studies' table. We collated multiple reports of the same study, so that each study, rather than each report, was the unit of interest in the review. We also provided any information we obtained about ongoing studies. We recorded the selection process in su icient detail to complete a PRISMA flow diagram (Liberati 2009).

Data extraction and management
We modified the Cochrane E ective Practice and Organisation of Care (EPOC) standard data collection form and adapted it for study characteristics and outcome data (EPOC 2017a). We identified key characteristics of the intervention for abstraction based on the mHealth Evidence Review and Assessment (mERA) guidelines (Agarwal 2016). We piloted the form on one study in the review.

Primary objective
Two review authors (NH and HB) independently extracted the following study characteristics from the studies that were included for the primary objective: • general information: title, reference details, author contact details, publication type, funding source, conflicts of interest of study authors; • population and setting: country, geographical location (rural, urban, peri-urban), healthcare setting (e.g. facility-based, community-based); • methods: function of the intervention, study design, unit of allocation, study duration; • participant characteristics: type of user (role, if in the health system; length of training, if any), description of any other participants in the intervention, withdrawals; • interventions: intervention purpose, components, infrastructure to support the technology, type of technology (so ware platform), type of mobile device(s) used (smartphone, tablets with a screen size larger than 7 inches, feature phones that can run java applications, basic phone with SMS and call functions, laptops), mode of delivery, content of the intervention, participant and provider training, interoperability, compliance with national guidelines, data security, comparison, fidelity assessment, duration of intervention; • outcomes: primary and other outcomes specified and collected, time points reported, adverse events, results of any subgroup analyses.
We noted in the 'Characteristics of included studies' table if outcome data were reported in a way that was not usable.

Secondary objective
For the secondary objectives, we extracted all the information listed above, if available, to describe the intervention. To understand factors a ecting the implementation of relevant interventions, we had planned to use the Supporting the Use of Research Evidence (SURE) framework (SURE 2011; Glenton 2017); however, we found that the themes identified in the framework did not apply well to the contents of the included studies. We also explored the use of other implementation research frameworks, such as the consolidated framework for implementation research (CFIR; (Damschroder 2015), but found minimal overlap between the themes identified in the data. Therefore, we read and re-read the included studies to identify new codes to tag the abstracted data.

Assessment of risk of bias in included study for the primary objective
For the primary objective, two review authors (NH and HB) independently assessed the risk of bias for the included study, using the criteria outlined in the Cochrane Handbook for Systematic Reviews of Interventions Section 8.5 (Higgins 2011), and guidance from the Cochrane EPOC group (EPOC 2017b). We assessed risk of bias for the included controlled before-a er study using the following criteria: random sequence generation, allocation concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, selective outcome reporting, baseline outcomes measurement, similarity of baseline characteristics, and other bias.
We judged each potential source of bias as either high, low, or unclear, and provided a quote from the study together with a justification for our judgment, in Table 2. We considered blinding separately for di erent key outcomes where necessary (e.g. for unblinded outcome assessment, risk of bias for all-cause mortality may be very di erent than for a participant-reported pain scale). When considering treatment e ects, we took into account the risk of bias for the study that contributed to that outcome.

Assessment of methodological limitations of included studies for the secondary objectives
For the secondary objectives, the included studies comprised a multitude of study designs and study aims, including case studies that were primarily descriptive. We were unable to find an accepted tool designed to appraise methodological limitations that could accommodate this variation in study design. Therefore, we piloted a newly developed tool for assessing the methodological limitations of sources, such as programme reports, that do not use typical empirical research designs. Two review authors (SA and CG) independently assessed the methodological limitations of the studies using the Ways of Evaluating Important and Relevant Data (WEIRD) tool (Lewin 2019). The tool, which is currently being piloted in EPOC and other systematic reviews, is available in Appendix 2.

Cochrane Database of Systematic Reviews
For each item and question in the tool, the review author selected one of the following response options: • Yes -the item was addressed adequately in the source • Unclear -it is not clear if the item was addressed adequately in the source • No -the item was not addressed adequately in the source • Not applicable -the item is not relevant to the source being assessed The assessments for each WEIRD tool item for each relevant study are reported in Table 3.
Based on the assessments for each WEIRD tool item, we made an overall assessment of the methodological limitations of the source as follows: • Where the assessments for most items in the tool were 'yes' -no or few limitations • Where the assessments for most items in the tool were 'yes' or 'unclear' -minor limitations • Where the assessments for one or more questions in the tool were 'no' -major limitations For each source, our assessment of whether most of the WEIRD tool items were addressed or not was a judgement. To make these judgements as explicit and transparent as possible, we have provided explanations of our reasoning in Table 3.
We then used the overall assessment for each source as part of the GRADE-CERQual assessment of how much confidence to place in the findings for each secondary objective.

Measures of treatment e ect
For the review's primary objective, we report pre-intervention and post-intervention means and proportions for the intervention and comparison groups, where possible. We estimated the e ect of the intervention using risk ratios for dichotomous data, together with the appropriate associated 95% confidence interval (CI) and mean di erence.

Unit of analysis issues
For the controlled before-a er studies included in the review, we had planned to report cluster adjusted risk ratios and their 95% CIs. However, the analysis of the one included cluster trial was not adjusted for clustering, and no intracluster correlation coe icient (ICC) was available (Shieshia 2014). Therefore, we presented the results without a measure of variance or precision of e ect for outcomes for which there is a unit of analysis error (EPOC 2017c)

Dealing with missing data
We contacted investigators in order to verify key study characteristics and obtain missing outcome data where possible (e.g. when a study was identified as an abstract only).

Assessment of heterogeneity
We did not undertake a meta-analysis, as we only included one study for the primary objective.

Assessment of reporting biases
We did not explore reporting bias statistically, as we only included one study for the primary objective.

Data synthesis
We presented a narrative overview of the findings, together with tabular summaries of extracted data, for the primary objective. We used Mantel-Haenszel risk ratios to present results from dichotomous data, where su icient data were available.
As part of the data synthesis, we had planned to explore how we could integrate the findings from our primary objective with those of the secondary objective. However, this was not feasible, as only one study was eligible for the primary objective and we assessed the findings from the primary objective to be of very low certainty.
For the secondary objectives, we had originally planned to use the SURE framework. However, we found that the themes identified in the framework did not apply well to the contents of the included studies. Therefore, we applied a thematic analysis approach. We read and re-read the included studies, coded the data, and generated themes. We then identified common themes across all included studies, and consolidated themes where they had overlapping data, and divided themes further if the data captured disparate ideas. Thematic synthesis is a standard approach that has been used across several qualitative evidence summaries. We only reported themes emerging from the data; we did not apply any other organizing frameworks.
Once the review findings were completed, one author went through each finding, identified factors that may influence the implementation of the intervention, and developed prompts for future implementers. These prompts were reviewed by at least one other review author. These prompts are not intended to be recommendations, but instead, are phrased as questions to help implementers consider the implications of the review findings in their context. The questions are presented in the 'Implications for practice' section.

Subgroup analysis and investigation of heterogeneity
Sub-group analysis was not possible as we only included one study for the primary objective, and it did not have data relevant to any planned sub-group analyses.

Sensitivity analysis
We did not identify a su icient number of studies to perform sensitivity analyses.

Summary of findings and assessment of the certainty of the evidence
We created 'Summary of findings' tables for the main intervention comparison(s) and included the most important outcomes in order to draw conclusions about the certainty of the evidence within the text of the review: • Availability of commodities (e.g. proportion of health workers or facilities reporting drug stockouts, time between stockout and availability of commodities); • Quality of data about stock management (e.g. accuracy of data, completeness of data); For the primary objective, two review authors independently assessed the certainty of the evidence (high, moderate, low, or very low) using the five GRADE considerations (risk of bias, consistency of e ect, imprecision, indirectness, and publication bias) (Guyatt 2008). We used methods and recommendations described in Section 8.5 and Chapter 12 of the Cochrane Handbook for Systematic Reviews of interventions (Higgins 2011), and the Cochrane EPOC worksheets (EPOC 2017d), and used GRADEpro so ware (GRADEpro GDT). We provided justification for decisions to downgrade or upgrade the ratings using footnotes in the table.
We used plain language statements to report these findings in the review (EPOC 2017e).
For the secondary objectives, two authors (SA, CG) used the GRADE-CERQual approach to assess our confidence in each finding (Lewin 2018). GRADE-CERQual assesses confidence in the evidence, based on the following four key components: methodological limitations of included studies; coherence of the review finding; adequacy of the data contributing to a review finding; and relevance of the included studies to the review question. A er assessing each of the four components, we made a judgement about the overall confidence in the evidence supporting the review finding. We judged confidence as high, moderate, low, or very low. The final assessment was based on consensus among the review authors. The GRADE-CERQual evidence profile tables supporting the assessment of confidence in each finding can be found in Table  1.

Results of the search
We conducted a systematic literature search to August 2019. We identified a total of 4886 references a er removing duplicates. We excluded 4778 references for the primary and secondary objectives, following a review of the titles and abstracts. We retrieved the full texts of 92 articles for the primary and secondary objectives for detailed eligibility assessment.
We included one study that fulfilled our inclusion criteria for the review's primary objective (Shieshia 2014, published and unpublished data).
We included 16 papers that fulfilled our inclusion criteria for the review's secondary objectives, including the one study that was also included for the primary objective.
We excluded 76 articles for reasons described in Figure 1. We did not identify any ongoing studies. Cochrane Database of Systematic Reviews

Primary objective
We included one controlled before-a er study that met our inclusion criteria for the primary objective: to assess the e ects of the intervention (Shieshia 2014). We determined that the study was a controlled before-a er study, based on our assessment of published and unpublished data. We present key characteristics of the included studies in the Characteristics of included studies table.

Secondary objective
We included sixteen studies that fulfilled our inclusion criteria for the review's secondary objectives; 13 studies were peer-reviewed articles and 5 were published reports. These described a total of 11 interventions targeted at stock notification and digital tracking of healthcare commodities. See Characteristics of included studies.
Several studies described interventions that were implemented in multiple countries. For example, one intervention called 'SMS for Life', described by four studies, was tested as a pilot in Tanzania

Interventions
The study that addressed the primary objective aimed to improve reporting, collation, and visibility of stock data. Shieshia 2014 describes two interventions, both with a common mobile web-based reporting system called 'cStock'. cStock is used for community-level reporting of stocks for 19 health products managed by health surveillance assistants (HSAs). In addition to cStock, one of the interventions included an enhanced management (EM) component, comprised of quality improvement teams that used data supplied by cStock, to monitor performance of the supply chain and make informed supply chain decisions (Comparison 1). The second intervention combined cStock with e icient product transport (EPT), which consisted of training all HSAs on bicycle maintenance, and providing a basic tool kit (Comparison 2).
The interventions described in all included studies for the primary and secondary objectives were targeted at notifying and managing

Outcomes
Shieshia 2014, included under the primary objective, reported on the availability of seven medicines for the treatment of childhood illnesses, timeliness of reporting on stock levels, and the acceptability of digital intervention to providers. Stock availability was measured in two ways: (1) Percentage of eligible HSA's who reported stockout of required medicines on the day of visit; and (2) Percentage of eligible HSA's who reported stockouts of specific medicines over the last 30 days. Timeliness of reporting on stock levels was only reported for the two intervention groups, and not for the comparison group. Acceptability of cStock was evaluated by looking at its level of routine use (e.g. HSA's who used cStock as the primary means for ordering health products), and benefits perceived by the users. Again, these results were reported only for the two intervention groups.
All the studies included for the secondary objectives described the interventions targeted at reducing stockouts with varying levels of clarity. None of the studies aimed to formally assess the barriers and enables of implementation.

Excluded studies
For the primary objective, we excluded 77 articles a er full-text screening for one of the following reasons: the article did not meet the criteria for study design (N = 33); the intervention did not include a mobile device component (N = 1); the intervention did not directly target stock notification or tracking (N = 39); the intervention was not used by a healthcare provider (N = 3); or the intervention was not used in primary care (N = 1). Details of 15 potentially relevant studies, which were excluded, are provided in the 'Characteristics of excluded studies' table.

Risk of bias in included studies for the primary objective
For the primary objective, we have presented the risk of bias assessments for the included study in Table 2  Cochrane Database of Systematic Reviews nature of the intervention, blinding was not possible. The study had di erent sample sizes for the di erent outcomes assessed, and more participants were included in the analyses at followup compared to the baseline. Methods for random sequence generation, allocation concealment, and blinding of outcome assessment were not described in the methods. Random sequence was generated by lottery among twelve socioeconomically and topographically comparable districts. Blinding was not feasible, as the intervention involved distribution of mobile phones. Outcome data were not reported for all participants.

Methodological limitations of included studies for the secondary objectives
For the secondary objectives, the included studies comprised a multitude of study designs. Some were case studies that described . All but two of the studies included for the secondary objective had significant methodological limitations -they did not include empirical data, and provided unclear descriptions of the source of the information, with limited evidence to support their findings (Atnafu 2017; Shieshia 2014). These studies described the interventions, and the conclusions were typically drawn from authors' experiences in implementing the intervention. We have reported our assessments for each WEIRD tool item and the overall assessment for each relevant study in Table 3.

E ects of interventions
See: Summary of findings 1 Primary objective: mobile stock notification with enhanced management compared to standard care; Summary of findings 2 Primary objective: mobile stock notification with e ective product transport compared to standard care; Summary of findings 3 Secondary objective: summary of findings

Primary objective
We included one study that met our primary objective: to assess the e ects of the intervention.
In Shieshia 2014, health surveillance assistants (HSAs) used their mobile phones for community-level reporting of data about nineteen drugs and products through a structured SMS -a system referred to as cStock. This was combined with two additional intervention components. In the enhanced management group (EM), district product availability teams were trained to use the data, monitor performance, and make informed supply chain decisions (Comparison 1). In the E icient Product Transport (EPT) group, HSAs received a toolkit and training on bicycle maintenance (Comparison 2).

Comparison 1: Mobile stock notifications with enhanced management (EM) compared to standard care
See Summary of findings 1

Availability of commodities
We are uncertain of the e ect of mobile stock notification with enhanced management on the availability of commodities (stockout of drugs in the last 30 days) compared to standard care (Shieshia 2014; very low-certainty evidence). Stockout of drugs in the last 30 days was measured for cotrimaxazole to treat bacterial infections (Analysis 1.1), artemether-lumefantrine to treat malaria caused by Plasmodium falciparum (Analysis 1.2; Analysis 1.3), oral rehydration drugs to treat dehydration (Analysis 1.4), and zinc to treat diarrhoea (Analysis 1.5).

Quality of stock management
Based on data from the intervention group only, we are uncertain of the e ect of mobile stock notification with enhanced management on quality of stock management (Shieshia 2014; very low-certainty evidence). Quality of data about stock management was assessed as the extent to which HSAs reported data about stocks that they had available. In the mobile stock notification with EM group, on average, 85% (N = 393) of the intervention group participants reported on stock levels for all the products that they managed.

Timeliness of stock management
Based on data from the intervention group only, we are uncertain of the e ect of mobile stock notification with enhanced management on timeliness of stock management (Shieshia 2014; very lowcertainty evidence). The e ect of the intervention on the timeliness of stock management was measured at the level of the health facilities. Health facilities in the stock notification with EM group took an average of 12.8 days to fill an order requested by healthcare providers.

Satisfaction and acceptability
Based on data from the intervention group only, we are uncertain of the e ect of mobile stock notification with enhanced management on provider satisfaction (Shieshia 2014; very lowcertainty evidence). Provider satisfaction with the intervention was evaluated based on routine use. Ninety-seven percent (N = 81) of HSAs in the stock notification with EM group reported using cStock as the primary means for ordering health products from the resupply point.

Resource Use
The included study did not report on the e ect of the intervention on resource use.

Unintended consequences
The included study did not report on the e ect of the intervention on unintended consequences.

Comparison 2: Mobile stock notification with e icient product transport (EPT) compared to standard care
See Summary of findings 2

Availability of commodities
We are uncertain of the e ect of mobile stock notification with EPT on the availability of commodities (stockout of drugs in the last 30 days) compared to standard care (Shieshia 2014; very low-certainty evidence) stockout of drugs in the last 30 days was measured for cotrimaxazole to treat bacterial infections (Analysis 2.1), artemether-lumefantrine to treat malaria caused by Plasmodium

Quality of stock management
Based on data from the intervention group only, we are uncertain of the e ect of mobile stock notification with EPT on the quality of stock management (Shieshia 2014; very low-certainty evidence).
Quality of data was measured as the extent to which HSAs reported data about stocks that they had available. In the mobile stock notification with EPT group, on average, 65% (N = 253) of the HSAs reported on stock levels for all the products that they managed.

Timeliness of stock management
Based on data from the intervention group only, we are uncertain of the e ect of mobile stock notification with EPT on the timeliness of stock management (Shieshia 2014; very low-certainty evidence). E ect of the intervention on the timeliness of stock management was measured at the level of the health facilities. Health facilities in the stock notification with EPT group took an average of 26 days to fill an order requested by healthcare providers.

Satisfaction and Acceptability
Based on data from the intervention group only, we are uncertain of the e ect of mobile stock notification with EPT on provider satisfaction (Shieshia 2014; very low-certainty evidence). Provider satisfaction with the intervention was evaluated based on routine use. Ninety-one percent (N = 78) of the HSAs in the stock notification with EPT group reported using cStock as the primary means for ordering health products from the resupply point.

Resource Use
The included study did not report on the e ect of the intervention on resource use.

Unintended consequences
The included study did not report on the e ect of the intervention on unintended consequences.

Current use of mobile strategies to improve reporting and digital tracking of health commodities
We included 16 studies that met the first of our secondary objectives; to describe how these types of mobile strategies are currently being used. These studies described eleven di erent interventions, all of which aimed to reduce stockouts. See Study characteristics for the secondary objectives in Characteristics of included studies. Figure 2 summarises the key intervention components that we identified in eight of these eleven interventions. The overarching purpose of each of these interventions was to make data about stock levels and potential stockouts of commodities visible to managers, who could then take corrective action to address them. Each of these interventions required the use of a mobile device by healthcare workers, either to report stock levels as a text message . It was intended that this webpage be monitored by a higher level (e.g. district level) health o icial, who was responsible for taking corrective action to address extant or expected stock or commodity shortages. Some authors reported instituting active measures to ensure that the online data were used and responded to in a timely fashion. Shieshia 2014 described how additional sta , with clearly defined roles, were hired to monitor the data and respond to them. In 'SMS for Life', weekly summary reports were provided to district medical o icers and pharmacists in addition to the dashboards, to support data use (Barrington 2010; Githinji 2013; Mikkelsen-Lopez 2014; WHO 2013).

Figure 2. Common key intervention components of interventions targeted at reducing stock-outs
The following features were salient to some of the eight interventions described in Figure 2.
• Reminders to healthcare workers to send reports. Three of the eight interventions described a feature that sent reminders to healthcare workers to send their weekly stock-level reports there were checks in place to ensure that the messages sent by the healthcare workers were free of error. In the case of an error in the format of the message, the health worker would receive an error message, advising them to correct their message (Asiimwe 2011; Barrington 2010; Githinji 2013). When there were no errors, the healthcare worker received a confirmation message, stating that their weekly stock report had been received (Asiimwe 2011; Barrington 2010; Chandani 2017; Githinji 2013; Shieshia 2014). • Multi-faceted interventions. Some of the interventions targeted at reducing stockouts were part of a broader multifaceted intervention. In addition to digital data reporting, aggregation, and visualisation, cStock invested in training district and central-level sta to use computers to access web-based dashboards for reporting, and quality improvement teams to use the data supplied by cStock to monitor performance of the supply chain and make informed supplychain decisions (Shieshia 2014). The LMS Suite, described by Stanton 2016, comprised three tools, each targeted at di erent aspects of managing cases of lymphatic filariasis, including: (1) 'MeasureSMS-MDA' to support text message mass drug administration (MDA) and reporting of cases that had been treated for elephantiasis using appropriate anti-parasitic medicines; (2) 'Measure-SMS-morbidity' to report on new cases of elephantiasis and their demographic information; and (3) 'Ly-MSS' lymphedema management support system aimed to maintain the supply chain of care packages (such as washbasins, towels, soaps, antibacterial cream).
Three of the eleven interventions did not include the key components described in Figure 2. One of these interventions was MomConnect, described by Barron 2016. MomConnect is a national-level service in South Africa that is targeted at connecting pregnant women to health services. Registration on MomConnect puts demographic and pregnancy-specific information about women on an electronic database. Women then receive free informational messages, as per their stage of pregnancy, till their infant is one year old. Using this service, women can also interact with a help desk, located in the South African Department of Health (DoH) to (1) answer a brief survey about the quality of services received at the health facility, using unstructured supplementary service data (USSD) at no cost to them; (2) ask any question related to their pregnancy, using a text message; and (3) log a complaint or a compliment, using a free of cost text message. The complaints could be about commodity or medicine shortages during their last visit to the health facility, and are routinely monitored and addressed by the help desk.
The second intervention was the palliative care management so ware reported in Namisango 2016. This intervention included data collection, an electronic health record for the patient, and functions for supply chain management and provider workplanning and scheduling. The health care provider could use a tablet computer to check their mobile application screen for which drugs were available in the pharmacy before prescribing the drug. Providers could enter patient and pharmacy data on the application. These data were linked to the database available to the pharmacist, who was responsible for tracking inventory based on the number of prescriptions written for a certain drug.
The third intervention was the use of bar codes to record and track health products reported in Hara 2017. They used an automatic identification and data capture system (AIDC), using barcodes based on global standard to improve end-to-end supply chain visibility of health commodities. The bar codes uniquely identified each health product and were linked to the batch and serial numbers of the products, and expiration dates. They described a process in which a smartphone application was used, in Ethiopia, to scan bar codes using the mobile phone's camera, and push the captured data to a central logistics management so ware. This made real time tracking of commodities feasible, and built e iciencies in the system to streamline the availability of products, where needed.

Factors influencing implementation
We also used these 16 studies to address the second of our secondary objectives: to identify factors influencing the implementation of mobile interventions targeted at reducing stockouts of health commodities. All but one of these studies lacked empirical data, and clear descriptions of the source of the information, resulting in limited evidence to support the findings (Shieshia 2014). The studies described the interventions used, and the conclusions were typically drawn from the study authors' experiences in implementing the intervention. We considered these limitations in our GRADE-CERQual assessment of confidence in our findings. We also noted in the results that the perspective of the findings is, in general, that of the study authors.
The study authors referred to several factors that may influence the implementation, uptake, or e icient use of interventions targeted at improving stock management. In Figure 3, we grouped these factors under three categories: macro-level factors that constitute a supporting ecosystem; programmatic factors associated with implementation; and factors directly pertaining to the technological component of the intervention. We also used these as a summary of findings for the secondary objective in Summary of findings 3. Cochrane Database of Systematic Reviews

Supporting ecosystem a. Infrastructure
Study authors identified several infrastructural issues that they suggested directly influence the implementation of mobile interventions, targeted at improving notification and tracking of commodities. These included problems with poor access to electricity, and poor or non-existent internet connectivity, leading to challenges in charging phones, uploading and transmitting data, and loss of data ( About one-fi h of all participants in one study reported challenges in sending text messages due to poor network (Shieshia 2014). Authors identified how some of these issues could be mitigated: using solar energy packs for charging phones (Namisango 2016), developing systems for working o line during internet outages (Namisango 2016), training health workers to resend their reports the next day, in case network coverage is not available initially (Asiimwe 2011), and taking steps upfront to ensure that network coverage is available within a few hours of the health facilities where the intervention is being implemented, so that reports can be submitted when health workers reach areas with better connectivity (Barrington 2010; Biemba 2017). However, health facilities that are responsible for receiving patient referral data from the community need to have regular connectivity (Biemba 2017).

b. Procurement and distribution systems
Study authors were concerned that digital stock notification systems used at the facility level could not mitigate several broader health system problems, including an underlying lack of stock at the national or district level, and a mismatch between national ordering routines and local needs (low-certainty evidence; Finding 2; Summary of findings 3). Cochrane Database of Systematic Reviews were identified as important considerations for implementation (Hara 2017). For example, the procurement of health commodities is influenced by donor policies. Mikkelsen-Lopez 2014 reported that during the implementation of their intervention 'SMS for Life', the Tanzanian government had reduced its malaria budget significantly, because over half the malaria drugs were provided by donors. However, a two-year delay in one of the donor funding cycles resulted in a national, critical, unanticipated drug shortage. Shortages at the national and regional levels cannot be mitigated by any corrective action taken at the district level. Such shortages also result in poor morale of managers (Githinji 2013).

Nationally instituted procurement and distribution systems (Chandani 2017), and inventory management information systems
Poor reconciliation between national and district drug procurement systems and the medicine ordering system makes it challenging to order the correct amount of drugs (Mikkelsen-Lopez 2014). For example, in Tanzania, health facility drug orders are made quarterly, based on the patterns of the previous quarter. However, this does not account for the seasonality of diseases like malaria (Mikkelsen-Lopez 2014).

c. Partnerships
Study authors described how programmes could benefit from strong partnerships, including with local authorities who could provide training for healthcare sta and district managers; with local telecommunications companies who could o er good rates for SMS transfer; with technical system providers who could support the development of the system; and with nongovernmental organizations (NGOs) who could support the rollout and training of the intervention (WHO 2013; very low-certainty evidence; Finding 3; Summary of findings 3).

Programmatic Support a. Data use, supervision, and management
Study authors suggested that the availability and use of data on stock levels at all levels of the health system allowed healthcare o icials to respond to anticipated shortages (lowcertainty evidence; Finding 4; Summary of findings 3). The use of digital data on stock availability was one of the key features of all the interventions identified by this review. Barron and colleagues suggested that data visibility and use are important at all levels of the health system, from the district to the national level (Barron 2016). Quick sharing of data across health workers and facilities was found useful by healthcare workers in Ghana and Malawi (Stanton 2016).
Shieshia and colleagues reported that in addition to data dashboards, comprising of information on stock levels, being available online in a digital format, stock reports were printed routinely at the district level. These stock reports and health facility performance and challenges were discussed at districtlevel meetings, allowing health workers across levels of the health system to become aware of the stock management procedures (Shieshia 2014). While most studies found the availability of stock data across levels of the health system to be useful, authors of one study highlighted a risk of making data accessible in real time across multiple levels of management. For instance, authors of a study in Uganda reported that district health o icials are typically used to having greater control over the data that they report to the national level. With a digital system to track and report stock levels, data becomes simultaneously available to the district and nationallevel stakeholders, and takes the opportunity away from the district o icials to contextualise the data or explain shortcomings (Asiimwe 2011).
Authors emphasised the role of supportive supervision and responsive management for e ective adoption of a digital system (moderate-certainty evidence; Finding 5; Summary of findings 3). Adequate supervision of the stock notification systems and associated data were identified as vital to successful implementation (Barrington 2010; Negandhi 2016; Shieshia 2014). For example, supervision of district level sta was needed to ensure that the data on stock levels were used, and appropriate corrective action was taken in a timely manner (Shieshia 2014). Routine visits and meetings of supervisors and healthcare workers can facilitate problem-solving, and ensure timely communication to discuss any resupply procedures (Chandani 2017). In structuring the management, programmes should consider that health workers must be motivated to report the data, and supervisors must be motivated to use the data (Chandani 2017). Some study authors highlighted the importance of having well defined roles and responsibilities for the management sta (Asiimwe 2011; Barrington 2010; Shieshia 2014), and strict timelines for the roll-out of the intervention, to further accountability (Barrington 2010). In some areas in Uganda, district health o icials, who were expected to monitor stock levels and respond to them, only became involved in an ad-hoc manner. The authors reported that this could be circumvented by having clearer roles defined by the Ministry of Health (Asiimwe 2011).

b. Knowledge, skills, and training
Some of these factors associated with implementation were tied to healthcare workers' and sta members' knowledge and skills, including the extent to which they were familiar with smartphones, comfortable using mobile data services (Stanton 2016), and the extent to which they were given adequate training in using

c. Human resources and incentives
Authors identified that to support successful implementation of a digital intervention, it was important to have technical programming expertise available to develop and install the digital programme, and to maintain the system on an ongoing basis (Asiimwe 2011; USAID 2010). Asiimwe 2011 suggested that locally available expertise in so ware programming was important to responsively develop and test the mobile applications. Having ongoing technology support was important to address so ware bugs and other problems once the system was piloted (USAID 2010; low-certainty evidence; Finding 7; Summary of findings 3).

Cochrane Database of Systematic Reviews
As discussed earlier, several studies o ered performance-based incentives of mobile phone credit to health workers, for timely reporting. While the value of such performance-based incentives was not formally assessed, one study author reported that they found that incentives in the form of airtime credit to healthcare workers was helpful in encouraging timely SMS reporting of stock levels (Barrington 2010; low-certainty evidence; Finding 8; Summary of findings 3).

Technology inputs a. Design of the digital systems
Several factors a ecting implementation were tied to the design of the digital system, including the extent to which the systems were user-friendly, with easy-to-use interfaces, and built with user participation (Namisango 2016; Negandhi 2016; Shieshia 2014), and the extent to which they were aligned with the country's existing health information reporting systems (Shieshia 2014). Two study authors emphasised the importance of iteratively designing the platform with user feedback and input to improve the acceptability and adoption of the digital intervention (Namisango 2016; Shieshia 2014; moderate-certainty evidence; Finding 9; Summary of findings 3).

b. Digital hardware and so ware
Study authors considered the use of basic mobile phones or personal phones by health workers to reduce challenges with data coverage and expense, and support easier adoption of the intervention due to familiarity with the phones (low-certainty evidence; Finding 10; Summary of findings 3). One study author suggested that programmes might consider using basic phones in lieu of android phones, as data network coverage is limited in remote locations, and data packages are prohibitively expensive (Stanton 2016). Another author suggested that having health workers use their personal mobile phones mitigates problems with phone maintenance, familiarity, and issues of ownership (Barrington 2010). Managers might be provided with a Blackberry or a similar device, so they can access dashboards, especially in places where desktop computer access is limited (Barrington 2010).
Study authors highlighted so ware features, such as ability to capture images, map geographic features, support two-way communication, toll-free text messaging, and interoperability (lowcertainty evidence; Finding 11; Summary of findings 3). So ware that had multiple features, such as the ability to capture images and map geographical locations, was amenable to programming, and could be used for di erent programmes was preferable (Negandhi 2016). Having a function for two-way communication with the healthcare workers, either to confirm receipt of their stock reports, or to send them updates on stock availability, helped them to take necessary action, and supported morale (Shieshia 2014). One author emphasised the value of a toll-free number for text messaging, so that health workers were not deterred by anticipated costs in sending text message on stock-level updates (Barrington 2010). Negandhi and colleagues identified interoperability of the stock management systems as important for success (Negandhi 2016). The authors suggested that logistics management systems should be linked to health management systems, so that linkages could be made between supply and demand, which should in turn, could reduce waste.

c. Data visualisation
Several factors that influenced implementation were tied to the design of the dashboards, and data visualisation. Authors emphasised that healthcare mangers should have access to data in an easy-to-use format (Shieshia 2014), with an e ective display of data using factsheets, and graphical and tabular illustrations (Negandhi 2016; moderate-certainty evidence; Finding 12; Summary of findings 3).
In order to accommodate this, Shieshia 2014 reported redesigning the dashboards several months a er the system was set up, so that the users had a better understanding of the metrics and visuals, and could incorporate their experiences interacting with the system into the redesign. The management of data should be detail-oriented, with regular reviews of the database (USAID 2010). To facilitate visualisation of data, healthcare personnel at other levels of the healthcare system also need access to functioning smart phones, laptops, or desktop computers (Biemba 2017).

Summary of main results
Our review provides limited evidence on the primary objective, to assess the e ect of tracking health commodity inventory and notifying stock levels via mobile devices on improvements in availability of commodities, quality of data about stock management, timeliness of stock-level reporting, and provider acceptability. We identified one study, conducted in Malawi, that used a before-a er study design to answer these questions (Shieshia 2014). However, we are uncertain of the e ect of these interventions on the outcomes of interest, because we assessed the certainty of this evidence as very low.
For the secondary objectives, we included 16 studies that described a total of eleven interventions. All studies were conducted in Africa (Tanzania, Kenya, Malawi, Ghana, Ethiopia, Cameroon, Zambia, Liberia, Uganda, South Africa, and Rwanda) and Asia (Pakistan and India). Most of the interventions aimed to make data about stock levels and potential stockouts visible to managers, who could then take corrective action to address them. We identified several factors that may influence the implementation of stock notification and tracking via mobile device. These included challenges tied to infrastructural issues, such as poor access to electricity or internet, and broader health systems issues, such as drug shortages at the national level, which could not be mitigated by interventions at the primary healthcare level. Several factors were identified as important, including strong partnerships with local authorities, telecommunication companies, technical system providers, and non-governmental organizations; availability of stock-level data at all levels of the health system; the role of supportive supervision and responsive management; familiarity and training of health workers in the use of the digital devices; availability of technical programming expertise for initial development and ongoing maintenance; incentives, such as phone credit, to support regular use of the system; easy-to-use systems built with user participation; use of basic or personal mobile phones to support easier adoption; consideration for so ware features, such as twoway communication; and data availability in an easy-to-use format, such as an interactive dashboard.

Overall completeness and applicability of evidence
We identified insu icient high-quality studies to address the primary objective of the review. Evidence was insu icient to recommend the use of mobile tools to track health commodity inventory and stock notification. We did not identify any data on use of resources for such interventions, or unintended consequences. Despite the proliferation of large scale, mobile-based interventions to support stock notification and management, we did not identify any ongoing studies to address questions on the e ectiveness of such interventions.
For the secondary study objective, the studies that described mobile interventions targeted at stock management had some common features, involving timely collection of stock data, visibility of stock data at di erent levels of the health system, and use and responsiveness to these data. Several implementation challenges that were identified by this review are consistent with the global evidence that points to general considerations for the implementation of digital interventions, including problems with network connectivity, access to electricity, device usability, and access to health worker training. Several 'best practices' were identified, based on the experiences of the study authors in implementing the interventions. Given the lack of empirical data from which these conclusions were drawn, and the high level of contextual and infrastructural variability within which such interventions might be implemented, these findings have limited external validity, and should be cautiously interpreted.

Quality of the evidence
We used the GRADE methodology to assess the quality of evidence for the primary objective, and GRADE-CERQual to assess the quality of evidence for the secondary objective. The quality of evidence relating to all five outcomes under the primary objective were downgraded two levels, due to very serious risk of bias concerns, and one level for imprecision, due to a small sample size. Outcomes of quality and timeliness of stock management, and satisfaction and acceptability of the intervention by providers, were downgraded to very low-quality evidence, as these were based on data from the intervention group only.
All but one study included for the secondary objective had significant methodological limitations -they did not include empirical data, had unclear descriptions of the source of the information, resulting in limited evidence to support the findings (Shieshia 2014). These studies described the interventions, and the conclusions were typically drawn from authors' experiences in implementing the intervention. Our confidence in the evidence for the secondary objective was typically downgraded due to methodological limitations of the studies, and adequacy of the findings, owing to a small number of studies contributing to specific findings.

Potential biases in the review process
We do not believe that the potential for bias in the review process for the primary objective was high. The authors meticulously followed the protocol. Where necessary, we attempted to contact the study authors to request missing relevant information.
For the secondary objective, while we followed the study protocol, the inclusion criteria were broadly defined. This could have resulted in the omission of certain articles and reports, especially if these were not published on any of the search engines that we outlined in our approach. In some cases, where reporting on the details of the intervention or factors a ecting its implementation was unclear or incompletely reported, the review authors attempted to infer relevant findings from the study authors' presented opinions.
The review team represents diverse professional backgrounds, which could have influence our input in conducting this review. Three of the review authors (SA, TT, GM) have been closely involved with the development and deployment of digital interventions in low-and middle-income countries, and have experience in conducting primary research to evaluate digital health interventions. One review author (SA) is a co-author on the study included in the primary objective. While these experiences provided us with a platform for understanding the complexities and nuances of evaluating such interventions, they may also have influenced our analyses of the studies included in this review. We tried to moderate this influence by working closely with other review authors. SA questioned the weight she attributed to certain data that resonated with her experiences, and ensured that all data were equally weighted in the final set of findings. Other members of the review team were called upon to verify the findings and ensure that they were supported by the data. As is standard practice in qualitative research, two authors conducted the GRADE-CERQual assessment.

Agreements and disagreements with other studies or reviews
To our knowledge, this is the first systematic review of mobile phone-based interventions for tracking health commodity inventory and stock notification, and trying to understand factors that a ect implementation of interventions targeted at improving stock availability. One literature review assessed the potential impact of mobile-based interventions on drug supply chain and stock management as one of several outcomes (Aranda-Jan 2014). The review narratively summarised results from two studies that were excluded from the primary objective, as they did not meet the study design inclusion criteria (Barrington 2010; Githinji 2013). It concluded that evidence was insu icient to assess the impact of mobile devices on drug stock management.

Implications for practice
Below are a set of questions that may help health system or programme managers when implementing or planning tracking health commodity inventory and notifying stock levels via mobile devices.

Have you considered the availability of necessary infrastructure?
• Do health workers have reliable access to electricity and internet connectivity? • Where network access is a challenge, are there systems in place so that sta can work o line until connectivity is restored? • Have you considered whether health workers might prefer to use basic or simple mobile phones, or their own personal phones, rather than smart phones, for instance because their own phones might be cheaper and easier to use? Cochrane Database of Systematic Reviews • Do you have reliable access to the medicines and supplies that local health facilities need? Are systems in place for regular procurement of medicines at the national and sub-national level, so that supplies can be made available when requested through digital notification systems?
2. Have you taken the needs and view of users into account when developing, planning, and implementing the use of mobile devices for stock notification and tracking?
Have you considered the type and format of data and information that should be presented on a dashboard?
• Will you involve users in an iterative design process, with the system evolving as the needs of users and the health system become clear? • Have you selected technology that is appropriate for your setting's data network coverage, data needs, and local capacity for maintenance? • Have you put in place mechanisms to select so ware that aligns with programme needs for specific functions, such as capturing images, mapping geographical locations, and twoway communication?

Have you considered how to work with key partners and how to share data?
• Have you considered partnering with local authorities to facilitate e ective implementation? This may include partnerships with the government, local telecommunication companies, technical systems providers, or non-governmental organisations (NGO). • Have you ensured that data are available at all levels of the health system? Would developing digital dashboards help in making data available to managers at district, regional, and national levels? If developing dashboards is not feasible, is it possible to develop a paper-based system for sharing stock availability reports with district and national levels?

Is there a plan for addressing training and support needs?
• Have you ensured that health facility sta have adequate training in the use of the digital system, and where necessary, in the use of any equipment, such as smart phones? • Do health facility sta have su icient mobile phone credit to support timely reporting of stock data and adoption of the system? • Do you have the technical programming expertise that is needed to develop, install, and maintain the system on an ongoing basis? • Have you ensured that health facility sta have access to supportive supervision and responsive management structures?
These questions were drawn, based on this review. They also align with similar implications for practice identified in a linked review on health workers' perceptions and experiences of using mHealth technologies to deliver primary healthcare services (Odendaal 2020).

Implications for research
1. Further, well conducted, comparative evaluations are needed to robustly establish the e ects of stock notification and commodity management via mobile devices on improved availability, improve timeliness of stock availability (average time between sending an order request and receiving health products), and reduced stockout of commodities at the point of care. Given the practical challenges in randomising such systems-level interventions, researchers may consider alternate study designs, such as controlled before-a er studies with at least two intervention and control sites, and interrupted time series studies with at least three data points before and a er the intervention.

Interventions targeted at improvements in stock management
have a large amount of variability in core intervention components. Therefore, it is important that research studies describe interventions in su icient detail that readers can discern the core components. 3. Currently, there is no standardisation of outcomes related to measurement of stockouts and other outcomes of interest in this review. In the studies included in this review, stockout outcomes were reported in three di erent ways: the proportion of health workers reporting stockout of specific drugs on the day of the interview; the proportion of health workers reporting stockout of specific drugs in the last 30 days; and the proportion of women of reproductive ages who reported a stockout (of preferred contraceptive) at the health facility (over an unspecified time period). Consistency in measurement of outcomes, and use of standardised metrics, where possible, can help facilitate comparability, pooling, and meta-analysis of the research findings. 4. Comparative evaluations should be accompanied by process evaluations to enhance understanding of the mechanisms and contexts within which di erent mobile-based stock notification and commodity management interventions work well, and the views and experiences of those using these systems. Understanding the conditions under which such systems adequately operate is valuable. For example, the review shows that misalignment of national stock ordering systems and local needs limits the e ectiveness of such digital systems. These process evaluations need to be well conducted, and should report their methods clearly. 5. Studies are needed of how di erent mobile-based stock notification and commodity management systems can be sustainably adopted and used. This review suggests that the use of incentives, such as mobile phone airtime credit, may be considered, and it would be valuable to test empirically the e ects of this and other incentives on the adoption and longterm use of such systems. 6. The cost-e ectiveness of di erent mobile-based stock notification and commodity management systems, compared to paper-based stock-management systems, should be assessed. 7. While there are certain common implementation considerations for mobile-based stock notification and commodity management systems, factors influencing implementation may vary by the exact type of intervention. Research studies should identify specific factors influencing implementation by intervention characteristics.

Tran 2015
Tran DN, Bero LA. Barriers and facilitators to the quality use of essential medicines for maternal health in low-resource countries: an Ishikawa framework. Journal of Global Health June 5 2015;5(1):010406.

WHO 2011
World Health Organization (WHO). The world medicines situation 2011 -access to essential medicines as part of the right to health. Available from apps.who.int/medicinedocs/en/ d/Js18772en/ (accessed 1 October 2017).

WHO 2015
World Health Organization (WHO). Technical consultation on preventing and managing global stock outs of medicines. Available at www.who.int/medicines/areas/access/ Medicines_Shortages.pdf?ua=1 (accessed 13 August 2020).

WHO 2016
World Health Organization (WHO

WHO 2019
World Health Organization. WHO Guideline: recommendations on digital interventions for health system strengthening. Available at www.who.int/reproductivehealth/publications/ digital-interventions-health-system-strengthening/en/ 2019.

Study characteristics
Methods

Study characteristics
Methods Secondary objective: case study using programmatic data Participants Facility-based health workers Context: 87 public health facilities in 5 Kenyan districts

Interventions
Health workers sent information on stock counts of artemether-lumefantrine (AL) and rapid diagnostic tests (RDT) using SMS messages through their mobile phones to a web-based system accessed by district managers.
Outcomes N/A Notes The program described here, SMS for Life, is the same intervention as the one described by Barrington 2010; Mikkelsen-Lopez 2014; WHO 2013, with differences in the products about which stock data were reported.

Study characteristics
Methods Secondary objective: case study describing program implementation Participants Not reported

Context: Ethiopia and Pakistan
Interventions In Pakistan, the use of global standards-based bar codes for inventory management of contraceptive supplies. In Ethiopia, a smart phone application was used to scan the bar codes using the mobile phone camera. These data were collated in a central inventory management system.

Study characteristics
Methods Secondary objective: case study using programmatic data

Participants Health facility workers
Context: 5000 public health facilities in Tanzania   Interventions Facility-based health workers used mobile phones to send information on stock counts of four dosage packs of artemether-lumefantrine (AL), using SMS messages within 27 hours of receiving a reminder, on a weekly basis. These data were made available as summary reports to the District Medical Officer and District Pharmacist.

Mikkelsen-Lopez 2014
Tracking health commodity inventory and notifying stock levels via mobile devices: a mixed methods systematic review ( (Continued)

Study characteristics
Methods Cochrane Database of Systematic Reviews were sampled at baseline and 81 at follow-up. Of the 253 HSAs assigned to the second intervention (cStock + efficient product transport), 44 were sampled at baseline, and 78 at follow-up.
Setting: Health facilities in 10 districts in Malawi Interventions

Intervention group A (cStock + enhanced management (EM))
cStock, a mHealth tool for community-level reporting of stock on hand data, and resupply of 19 health products managed by Health Surveillance Assistants (HSAs). cStock is an SMS and web-based reporting and resupply system that is used by HSAs to report stock data, via SMS through their personal mobile phones. cStock calculates HSA resupply quantities, and sends this information to Health Facility (HF) sta to use to pick and pack products for HSAs, and notify them of a collection time.
The EM intervention addressed challenges related to data availability and visibility, and low motivation among HSAs. The additional component of the EM intervention was District Product Availability Teams (DPATs). These are multilevel quality improvement teams that use the data supplied by cStock to monitor performance of the supply chain, and make informed supply chain decisions.

Intervention group B (cStock + efficient product transport (EPT))
cStock, a mHealth tool for community-level reporting of stock on hand data, and resupply of 19 health products managed by Health Surveillance Assistants (HSAs). cStock is an SMS and web-based reporting and resupply system that is used by HSAs to report stock data, via SMS through their personal mobile phones. cStock calculates HSA resupply quantities, and sends this information to Health Facility (HF) sta to use to pick and pack products for HSAs, and notify them of a collection time.
The EPT intervention addressed challenges of transport, plus data visibility. The additional components of the EPT intervention was training all HSAs on bicycle maintenance, providing a basic tool kit, and using a continuous review inventory control system.

Control group: No intervention
Outcomes Feasibility: the feasibility of cStock was evaluated by looking at sta capacity to use cStock, practicability, and relevance Acceptability: the acceptability of cStock was evaluated by looking at its level of routine use, its effect on users' daily work, and perceived benefits identified by the user.
Effectiveness: findings on effectiveness were presented as a comparison of supply chain performance between the EM + cStock and EPT + cStock groups, according to these four indicators: reporting, complete reporting, lead time, and stockout rates over time, using data from cStock dashboard reports.

Notes
No feasibility or acceptability data were available for the control groups. No effectiveness data were available that compared cStock to no intervention.

Source of funding: Bill and Melinda Gates Foundation
Shieshia 2014 (Continued)

Study characteristics
Methods 8. Incentives, such as receiving phone talk-time credit, to improve adoption and use of the digital system are valuable.

Barrington 2010
Serious concerns, because 1 study had serious methodological limitations (insufficient evidence to support findings) No, or very minor concerns about coherence Concerns about adequacy, as only 1 study contributed to the finding No, or very minor concerns about relevance Low confidence Due to methodological limitations and concerns about adequacy, as conclusions are based on few studies. 9. User-friendly systems, built with user participation with easyto-use interfaces were considered important to implementation.

47
ings), and one study had minor methodologic limitations 10. The use of basic mobile phones or personal phones by health workers reduced challenges with data coverage and expense, and supported easier adoption of the intervention due to familiarity with the phones. Serious concerns, because 2 studies had serious methodological limitations (insufficient evidence to support findings), and one study had minor methodologic limitations No, or very minor concerns about coherence Minor concerns about adequacy, due to few studies and the relevant data are sparse.
No, or very minor concerns about relevance Low confidence Due to concerns about methodological limitations, and concerns about adequacy, as conclusions are based on few studies. 12. Dashboard design and data visualisation played important roles in effective implementation. Managers should have access to data in an easy-to-use format, such as an interactive dashboard.   (Continued) Cochrane Library Trusted evidence. Informed decisions. Better health.