Liberia: Interim Approach Assessment Report

Publication date: 2015

Liberia: Interim Approach Assessment Report AUGUST 2015 This publication was produced for review by the U.S. Agency for International Development. It was prepared by the USAID | DELIVER PROJECT, Task Order 7. Liberia: Interim Approach Assessment Report The authors' views expressed in this publication do not necessarily reflect the views of the U.S. Agency for International Development or the United States Government. USAID | DELIVER PROJECT, Task Order 7 This document was prepared by staff of the USAID | DELIVER PROJECT, Task Order 7, which is funded by the U.S. Agency for International Development (USAID) under contract number GPO-I-00-06-0007-00, order number AID-OAA-TO-11-00012, beginning on March 28, 2011. Task Order 7 is implemented by John Snow, Inc., in collaboration with 3i Infotech, Inc.; Crown Agents USA, Inc.; FHI 360; Foundation for Innovative New Diagnostics; Logenix International, LLC; The Manoff Group, Inc.; MEBS Global Reach, LC; PATH; PHD International (a division of the RTT Group); Population Services International; Social Sectors Development Strategies, Inc.; UPS Supply Chain Solutions, Inc.; and VillageReach. Task Order 7 supports USAID's goal of reducing the malaria burden in sub-Saharan Africa by procuring and delivering safe, effective, and high-quality malaria commodities; by providing technical assistance and on-the-ground logistics expertise to strengthen in-country supply systems and build capacity for managing commodities; and by improving the global supply and long-term availability of malaria commodities. Recommended Citation Perry, Steven, Ariella Bock, Marie Tien. 2015. Liberia: Interim Approach Assessment Report. Arlington, Va.: USAID | DELIVER PROJECT, Task Order 7. Abstract In June 2015, the Ministry of Health and Social Welfare (MOHSW), with technical assistance from the USAID | DELIVER PROJECT, Task Order 7, conducted an assessment of the performance of the Interim Approach (IA) logistics management and supply chain systems for selected health care commodities in Liberia. The assessment looked at the performance of the IA against 24 benchmark/indicators over the life (November 2013 – June 2015) of the IA. This report, presented to the MOHSW, includes the findings of the assessment, as well as the short- and long-term recommendations to improve the commodity logistics systems in Liberia. Cover photo: A health worker in his store room in Liberia, 2015. USAID | DELIVER PROJECT John Snow, Inc. 1616 Fort Myer Drive, 16th Floor Arlington, VA 22209 USA Phone: 703-528-7474 Fax: 703-528-7480 Email: askdeliver@jsi.com Internet: deliver.jsi.com Contents Acronyms. vii Acknowledgments . ix Executive Summary . xi Introduction . 1 Context. 1 Purpose and Objectives of the Assessment . 2 Methodology. 3 Data Collection. 4 Assessment Limitations . 5 IA Performance: Availability and Accountability . 7 Product Management at Health Facilities. 8 Stock Availability at Health Facilities.10 Stockout Rates .10 Stocked According to Plan (Prior to Distribution) .11 Accountability of Supply Chain from County to Facility Level .13 Delivery Schedule and Coverage Rates.13 Order Fill Rates from County Depot to Facility.14 Accountability of Supply Chain from NDS to County.16 Summary: IA Performance on Availability and Accountability.19 Recommendations .20 Interim Approach Costs .21 Costing Methodology .21 Cost Results.21 Recommendations .27 NDS Performance against Benchmarks.29 NDS Strengthening Benchmarks.29 Summary of NDS Performance.32 Recommendations .32 IA Field Findings .33 Recommendations .36 Key Supply Chain Strategies/Objectives.36 Supply Chain Design Exercise.37 Recommendations .38 iii Appendices A. Performance Indicators.39 B. Key Informant Meetings .59 Figures 1. Average Stockout Rate for Malaria Products across All Counties by Round.11 2. Average Stockout Rate for Family Planning Products across All Counties by Round .11 3. Health Facility Stock Levels for Malaria Products by Round by Funder.12 4. Health Facility Stock Levels for Family Planning Products by Round by Funder .12 5. Delivery Schedules from Counties to Facilities.13 6. Health Facility Coverage Rate by Round .13 7. Health Facility Stock Levels for Malaria Products Following Distribution by Round by Funder .16 8. Health Facility Stock Levels for Family Planning Products Following Distribution by Round by Funder .16 9. NDS Order Fill Rates within ± 5 Percent of Order Across Rounds.18 10: Order Fill Rates for Malaria Products within ±5 Percent of Order Across by Rounds.18 11. Order Fill Rates for Family Planning Products within ±5 Percent of Order Across by Rounds.19 12. Supply Chain Costs by Function .22 13. Supply Chain Costs by Partner (US$) .23 14. IA distribution costs by USAID and Global Fund by round (U.S.$) .23 15. Transport Costs by Round.24 16. IA Distribution Expenses by Cost Type.25 17. IA Distribution Costs by Funding Partner (U.S. $).25 18. Estimate Annual Cost of the Previous System and IA.26 19. Average Stockout Rate for Malaria Products across Global Fund Supported Counties by Round .43 20. Average Stockout Rate for Malaria Products across USAID Supported Counties (excluding Montserrado) by Round .43 21. Average Stock-out Rate for Malaria Products in Montserrado by Round.44 22. Average Stock-out Rate for Family Planning Products across Global Fund Supported Counties by Round .44 23. Average Stock-out Rate for Family Planning Products across USAID Supported Counties (excluding Montserrado) by Round .45 24. Average Stock-out Rate for Family Planning Products in Montserrado by Round.45 25. Distribution of Percent Difference in Quantity Requested and Received in Global Fund Supported Counties of all Orders-Round 3 .52 26. Distribution of Percent Difference in Quantity Requested and Received in Global Fund Supported Counties-Round 4.52 27. Distribution of Percent Difference in Quantity Requested and Received in USAID Supported Counties-Round 3.53 28. Distribution of Percent Difference in Quantity Requested and Received in USAID Supported Counties-Round 4.53 iv 29. Distribution of Percent Difference in Quantity Requested and Received in USAID Supported Counties-Round 5 .54 Tables 1. IA Assessment Benchmarks and Indicators. 3 2. IA Tracer Commodities. 7 3. Number of Products Distributed in Counties by Program Type by Round .8 4. Number of Health Facilities Managing IA Tracer Products by Round by Funder.9 5. Percent of Orders Filled Within ±5 Percent of Requested Amount.15 6. Cost per IA Round (Round 5 includes only USAID costs) .22 7. Cost Against Supply Chain Performance .26 8. Number of Health Facilities Managing IA Tracer Products by Round by County .39 9: Stockout Rates by Product by Round and County .45 10. Health Facility Stock Levels for Tracer Products by Round by Funder .48 11. Health Facility Stock Levels Post Distribution by Round and Funder .55 v vi Acronyms AIDS Acquired immunodeficiency syndrome AMC Average monthly consumption ARVs Antiretrovirals AS/AQ Artesunate/amodiaquine CCM Country Coordination Mechanism CHAI Clinton Health Access Initiative CMO Chief Medical Officer of Liberia DA Depot Assistant DD District Depot DDFP District Drug Focal Person DHO District Health Officer DHT District Health Team DSA Daily subsistence allowance FHD Family Health Division GF Global Fund to Fight AIDS, Tuberculosis and Malaria HF Health Facility HIV Human immunodeficiency virus IA Interim Approach LMHRA Liberia Medicine and Health Products Regulatory Authority LMIS Logistics Management Information System MOF Ministry of Finance MOHSW Ministry of Health and Social Welfare M&E Unit Monitoring and Evaluation Unit MOS Months of stock NACP National Aids Control Program NDS National Drug Service NGO Non-governmental organization NMCP National Malaria Control Program vii NTLCP National Tuberculosis and Leprosy Control Program OFM Office of Financial Management PBL Pharmacy Board of Liberia PD Pharmacy Division of Liberia PR Principal Recipient PU Procurement Unit-MOHSW SBRR Stock balance reporting and requisition SLA Service Level Agreement SCMP Supply Chain Master Plan SCMS Supply Chain Management System SCMU Supply Chain Management Unit SOP Standard operating procedure SP Sulfadoxine-pyrimethamine TB Tuberculosis UNDP United Nations Development Program WHO World Health Organization UNAIDS United Nations Programme on HIV/AIDS UNFPA United Nations Population Fund UNICEF United Nations International Children's Emergency Fund USAID United States Agency for International Development viii Acknowledgments The authors would like to acknowledge the hard work and valuable contribution of local stakeholders who participated in developing the purpose and scope of this assessment, or as formal members of the data collection team, or as key informants, or as helpful hands pointing us in the right direction when we were unsure. This list features almost all of the prime drivers of the health supply chain efforts in Liberia and representing the following organizations and institutions: • MOHSW programs including RH, HIV/AIDS, Malaria and TB • MOHSW/SCMU, Pharmacy Division and NDS • MOHSW county pharmacists and county teams, as well as colleagues from the district level • USAID, Global Fund, UNFPA and WHO • CHAI and USAID | DELIVER PROJECT It should be noted that the assessment relied on data collected by these stakeholders and presented in the distribution round reports, data bases, plans reports and guidelines related to the distribution of health products in Liberia. ix x Executive Summary In June 2015, a team led by technical advisors from the USAID | DELIVER PROJECT carried out an assessment of the Interim Approach supply chain (IA). The team, which included MOHSW and USAID | DELIVER PROJECT/Liberia staff, conducted key informant interviews with central and county level stakeholders and a field exercise covering 42 facilities (36 service delivery points and six county depots) in six counties to assess the performance of the IA across 24 selected benchmarks. Performance data was extracted from five rounds of distribution in USAID supported counties and three rounds in GF supported counties. The purpose of the assessment was to review the quality and effectiveness of the IA through time, and, to the degree that data was available, compare cost elements of the IA to the MOHSW system that preceded it. The assessment engaged national expertise and the collaboration of national counterparts and partners, through a participatory approach, to help understand the degree to which the IA met its goals. The results are intended to provide stakeholders with a foundation for informed decision making regarding the future of public health supply chain in Liberia. This assessment is one component of the broader initiative to strengthen the distribution of health supplies and will be followed by exercises to update the Supply Chain Master Plan and a supply chain design workshop. Key Findings Commodity Availability and Accountability: Availability of products has improved during the IA, from approximately 28 percent at the start of the IA to over 70 percent during the last round. Stockouts, however, are still relatively common, with rates varying between 20-50 percent for malaria related products and 10-30 percent for family planning commodities during the last two rounds. Additionally, at the start of almost every distribution round, 40-60 percent of facilities were stocked out of or had less than two months of malaria products while 20-60 percent of facilities were stocked out of or had less than two month of family planning products (rates varied by individual products and counties). Despite this, health facility personnel across all counties consistently feel that the program products have become much more available over the past year and a half and that stockouts, when they occur, do not last as long as they did before the IA. While successfully covering more than 90 percent of targeted facilities each round, the IA has not consistently been delivering products per schedule. Since Round 2, timing of delivery has occurred closer to once every four months rather than every three months, as designed. In Global Fund supported counties the delivery delays were more pronounced. In addition, most county delivery teams across rounds failed to follow order fulfillment guidelines across all products and consistently over or under supplied facilities with products by more than five percent of the required amounts. Going forward a more unitary mechanism, coordinated by the SCMU, will be required to ensure that the deliveries to all counties go forward in the time frame needed to prevent stockouts. This will require that the funds for each distribution round are in place at least one month in advance of the scheduled distribution. xi Most health facility staff interviewed had limited knowledge of the IA system. Therefore, staff was unable to hold the delivery teams or counties accountable for how much product facilities received or when it would arrive. However, the adoption of the IA data form beginning in Round 3 helped to improve accountability. Health facilities were able to produce signed copies of the IA data form (i.e. the delivery round waybill) showing signatures of the delivery team, the facility OIC, as well as a community representative confirming receipt of the products. CHTs also had limited insight into NDS deliveries and limited ability to hold the higher level accountable. At the time of the assessment, central level teams from the USAID | DELIVER PROJECT and SCMU completed the requisitions, so most CHTs did not have visibility into the final requisition or program-approved order until products arrive at the county depot. Nevertheless, based on requisition from the e-SBRRs/SBRRs and signed waybills, NDS filled program approved orders within +/- 5 percent across all rounds for most products. Finally, although the IA’s overall performance in terms of stock availability is mixed, it is important to note that in-depth quantitative analysis was successfully carried out and is evidence of the IA’s goal of increasing data visibility of the system. While the availability and quality of data varied by round and by county, a plethora of data is now generated from the IA data form, SBRRs, e-SBRRs, waybills and invoices, which strengthens accountability at each level of the system Costs: Costs of the IA are generally declining as the stockout rate continued to decrease over time. Additionally, the order fill rate from NDS to the county also improved, indicating accountability is also increasing. In summary it has cost approximately $2 million to run the IA in all 15 counties since the beginning of the IA to June 2015. This includes health facility staff time, distribution costs— including vehicle rental, and DSA, SCMU, and NDS operational costs. Over the life of the IA, transport costs have declined while logistics performance has improved, suggesting there have been gains in efficiency and effectiveness of the IA system to deliver products. It costs an estimated $0.15 to deliver one dollar’s worth of IA commodities. Over time this metric can be tracked to measure the cost of the supply chain as a percentage of commodity costs. NDS: Overall, the achievement of NDS against the benchmarks set for the Interim Approach is very strong. IA commodities are well organized, managed and secured within the NDS compound, if not co-located within the same building. The records, both manual and electronic, indicate strong inventory control overall. The specific steps to strengthen NDS under the IA initiative have been successfully undertaken. In particular, the deployment of the USAID | DELIVER PROJECT Warehouse Advisor has had a positive impact on NDS’ role in the IA. However, it should be noted that external support may also mask capacity challenges in the NDS. The most obvious challenge to the NDS’ role going forward is the limited space at its current compound vis a vis the needs of the IA and for the MOHSW mission overall. The second challenge—managerial capacity—is less well captured in this assessment as much of the central level management of IA commodities is driven by external support. Stakeholder Perceptions of Supply Chain Priorities: Central, county and facility level staff, as well as partner participants at the IAA workshop, were fairly homogenous in their supply chain vision. The preference for the new supply chain was one that will support essential drugs as well as the current program commodities, in a pull system in which lower levels determine needs and place orders to higher levels based on the data derived from the current LMIS. Further, NDS and county capacities for warehousing would be used and by and large the deliveries would be quarterly from central to county and monthly from county to HFs. Transport (private vs public) was the area in which the xii opinions of the groups differed most. In sum, participants at the workshop favored a future supply chain design which was largely the same as the pre-IA supply chain for the health sector in Liberia. xiii xiv Introduction Context Prior to the implementation of the Interim Approach supply chain (IA), assessments of the public health supply chain systems in Liberia highlighted significant challenges regarding the proper management of, and accountability for, the medicines and medical supplies within the public health supply chain. In June 2013, the U.S. Agency for International Development (USAID) informed the Ministry of Health and Social Welfare (MOHSW) that the products it had donated could not be distributed unless appropriate control measures were put into place. In response to these concerns, the MOHSW implemented the IA, which included a series of interventions aimed at improving commodity availability as well as accountability through the implementation of a modified top-up delivery model. The MOHSW’s IA plan also outlines measures to improve storage and inventory control at the central level, transport distribution within the counties and strengthen monitoring and supervision throughout the supply chain. USAID and the Global Fund set the execution of the IA as a pre-requisite for the continued disbursement of donated health commodities. The IA initiative is coordinated by the Supply Chain Management Unit (SCMU) of the MOHSW in close collaboration with the National Drug Service (NDS), under the oversight of the Supply Chain Task Force, a body that convenes the key drug supply and technical assistance stakeholders. Financing to implement IA distribution activities is being provided by USAID through the USAID | DELIVER PROJECT in five of Liberia’s 15 counties, and by the Global Fund through the MOHSW in 10 counties. A key objective of the IA is to ensure increased visibility in the system through enhanced quality and availability of logistics data from both the county and facility levels. Rigorous supervision and documentation associated with each round of distribution is available, yielding the following key indicators: • Stock status (e.g. under-stock, over-stock and stockout rates) • Coverage rates • Transaction accountability (e.g. what was issued, what was received) • Data management capacity The information derived from these data forms the basis for subsequent drug distribution activities and support decisionmaking to improve the supply chain processes. Given past and continuing challenges within the model, the transition from the interim approach to a longer term distribution model(s) needs to be guided by evidence, the context of the country, and resources available. The IA and whatever supply chain solution is selected going forward need to be considered in the context of the Liberia Supply Chain Master Plan (SCMP), established in 2010. The SCMP is the reference for the design of the national medical supply chain, though it has not been fully implemented and contains provisions that are no longer realistic given both performance and resource challenges. The SCMP will be reviewed and revised beginning with a review workshop in 1 September 2015. The key findings of this IA assessment will feed into the review and update of the SCMP and inform a detailed supply chain design process. It should be noted that this assessment was initially planned to occur in August 2014, but the outbreak of Ebola delayed implementation. Although the IA was conceived as a temporary “band­ aid” it has now been stretched out across six full (quarterly) distribution rounds in USAID supported counties and four full rounds in GF supported counties Purpose and Objectives of the Assessment The purpose of the assessment was to review the quality and effectiveness of the IA through time, to assess the degree of data availability, and compare cost elements of the IA to the MOHSW system that preceded it. The assessment involved a collaboration of national counterparts and partners, through a participatory approach, to help understand the degree to which the IA met its goals. The results are intended to provide stakeholders with a foundation for informed decision making regarding the future of public health supply chain in Liberia. This assessment is one component of the broader initiative to strengthen the distribution of health supplies and will be followed by an exercise to update the Supply Chain Master Plan and a supply chain design workshop. The objectives of the IA assessment were to— • assess the extent to which the IA implementation influenced and improved accountability and visibility for products along the supply chain and in comparison to the previous system • determine the extent to which the IA increased availability of health supplies at the facility level and in comparison to the previous system. • determine the extent to which the IA increased the visibility and use of LMIS data • review the operational costs associated with the IA top-up system, and identify potential gains realized or lost during implementation of the IA (e.g., transport efficiency, reduction of wastage and leakage of products along the supply chain, availability of products at counties and facilities etc.) • disseminate findings to stakeholders and organize a consensus building workshop for relevant partners to enable a decision around next steps (short and long-term) for strengthening distribution and future revision of the SCMP and development of a distribution system suitable to the Liberian context. Regarding the costing of both the traditional distribution system and the IA, it should be noted that analysis was limited by availability of information on the previous MOHSW supply chain and by the time parameters of this assessment. The assessment team collected and analyzed the data available on historical throughput, supply chain cost, and staff time spent on supply chain activities and performance data to make a comparison between the traditional distribution system and the IA supply chain. The scope of the assessment was to provide insight into the performance of the IA in both USAID and Global Fund supported counties. The assessment relied on IA performance data across five rounds of implementation in USAID supported counties and four rounds in Global Fund supported counties, as well as additional primary qualitative data collected through a field exercise in six counties and existing documentation regarding performance of the previous MOHSW supply chain. The team sought to answer the following questions: 2 1. To what extent has the IA achieved its goals? 2. What contextual factors impact the IA supply chain performance? 3. What has been the effect of the IA on staffing, management and supervision capacities within the supply chain system in Liberia? 4. How is the IA perceived by staff at each level and by the key stakeholders? Methodology The methodology-along with the background, purpose, and objectives—was described in detail in the IA Assessment Plan (June 5, 2015), which was developed with in-country stakeholders through an iterative process. It was developed specifically to address the objectives listed above in providing stakeholders with objective measures of IA performance, as well as contextual information to help undrstand why the system did, or did not, achieve the hoped for results. An assessment framework based on the Improving Commodity Security through Improved Accountability and Controls: An Interim Approach (July 2013) (commonly refered as the IA Concept Note), was developed to determine the extent to which the IA increased accountability and availability in the supply chain. Twenty benchmarks and indicators were identified from the IA Concept Note or the standard documentation associated with each round of distribution (ie Round Reports). Additionally, four standard indicators for costing supply chains were added to the framework. Table 1 below presents these 24 benchmarks and indicators. Table 1. IA Assessment Benchmarks and Indicators IA Concept Note Reference Benchmarks/indicators 1 IA 1 Program products are co-located, clearly organized by batch number with bin/stock cards 2 IA 4 ERP data, physical count and stock bin cards match +/-5% 3 IA 2 Products are secured, locks and guard service in place 4 IA 3 Incoming shipping documents, proof of delivery and receipts reconcile 5 IA 7 Reconciliation between issues from NDS, quantities on signed waybills, and any returned stock to NDS matches +/- 5% 6 Operations manager hired 7 IA 5 Warehouse Advisors (2) hired 8 IA 10 Pharmaceutical supervision team identified with clear TORs 9 IA 8 Reconciliation between depot requisition, issues from NDS, quantities on signed waybills match +/- 5% 10 IA 6 Roles and reporting between CHTs and NDS clarified 11 IA12 LMHRA inspections of facilities undertaken 12 IA 9 Reconciliation between facility requisition, issues from county depots, quantities on signed waybills match +/- 5% 13 Adherence to delivery schedules from NDS and from counties to facilities 14 Delivery coverage rates 15 Stockout rates 3 IA Concept Note Reference Benchmarks/indicators 16 % of tracer commodities understocked (<2 mos) 17 % of tracer commodities overstocked (>4 mos) 18 % of HFs cannot calculate AMC 19 # of HFs managing IA tracer products (by county) 20 # of products being distributed by HFs by program type by county 21 Total supply chain costs 22 Cost per $ of annual throughput 23 Cost per m3 of annual throughput 24 Transport costs per $1,000 of commodities This report is organized to present the observed performance of the Interim Approach against these 24 benchmarks. Data Collection In June 2015, a team, led by technical advisors from the USAID | DELIVER PROJECT, carried out the qualitative component of the assessment. The team, which included MOHSW and USAID | DELIVER PROJECT/Liberia staff, conducted key informant interviews with central and county level stakeholders and a field exercise covering 42 facilities (36 service delivery points and 6 county depots) in six counties to collect primary qualitative data. The key informants were purposely selected based on positions and roles in the IA and national supply chain system. At the central level, informants included government program managers from SCMU, NDS, NMCP, and Office of Financial Management (OFM) as well as other IA stakeholders including USAID, Global Fund, Pool Fund, USAID | DELIVER PROJECT and CHAI. At the county and facility level, informants included members of county health teams (CMO, DDFP, county Depot Assistant, etc.), facility OICs and store managers. The interviews focused on awareness of the IA’s work, contextual factors that helped or hindered the success of the IA, and general views of the IA’s key accomplishments and challenges in terms of accountability, visibility, and availability of commodities. Gbarpolu, Grand Kru, Lofa, Monserrado, Nimba, and Sinoe counties were selected for the field exercise as they provided a crosscut representation of Global Fund and USAID supported counties with both good and challenging communications. Facilities visited within each of the countries were identified by the county health teams. The team also reviewed performance and financial data from each of the rounds of implementation of the IA, as well as limited financial documentation from the previous system. Data sources for the performance based indicators included: health facility stock-levels, consumption and resupply data taken from the “IA Aggregated Tool”; county-level resupply data abstracted from the Stock Balance Reporting and Requisition forms (SBRRs), e-SBRRs, NDS waybills and invoices as well the USAID | DELIVER PROJECT Distribution Round Reports (Rounds 1-5) and MOHSW/SCMU Integrated Distribution Round Reports (Rounds 1-3); and key informant interviews. Analysis focused specifically 4 on accountability and availability of eight tracer products and associated data that are used to track performance in the distribution round reports.1 The costing of the IA top-up system was based on the USAID | DELIVER PROJECT supply chain costing methodology which captures costs of the supply chain according to distribution, storage, and management functions.2 Data came primarily from actual expenditure information provided by the Global Fund fiscal manager, NDS, Office of Financial Management (OFM), and the USAID | DELIVER PROJECT from the start of the IA (June 2013) until time of the assessment (June 2015). The data was supplemented with key informant interviews at the central level and county depot and health facility staff during the field visits. Product value information was provided by the NDS Warehouse Operations Advisor. Assessment Limitations • Availability and reliability of data was not uniform for all benchmarks through time. As such, analysis and ability to disaggregate results vary. • Availability of NDS and county level data (eg. e-SBRRs, invoices, waysbills) in electronic format was limited and varied by round as well as between countries. • Similarly, data and resource limitations initially resulted in unusable data sets for measuring NDS benchmarks 4, 5 and 9. This data was recollected again in August (post in-country work) and reflects NDS performance in 2015, rather than throughout all rounds. Hence, these results say more about how the systems function now than about how those capacities have evolved since June 2013. • Availability of health facility data varied by round as well as between USAID and Global Fund counties. For example, while stockout rates at health facilities were available for all five USAID rounds in all five counties, rates in all 10 Global Fund counties were only available for Rounds 3 and 4. • Quality of data in the IA Aggregator Tool varied. As a result, data was cleaned using a set of agreed upon assumptions. Consequently, as some values were modified or dropped from the final performance based analyses, some results differ slightly from those previously published in the round reports. • Additionally, the IA Data Form, which is the primary source for the health facility level data, was only introduced into the system at the start of Round 3. Therefore facility level results published in Rounds 1 and 2 round reports could not be independently verified. • NDS costs attributable to the IA were estimated by the NDS using percentages to allocate costs specific to the IA versus management and storage of all other products at the NDS. • No recent financial audit report of the NDS is available to validate the data given on NDS expenditures. 1 The eight tracer products are: Sulfadoxine 500mg + Pry 25mg Tablets, AS/AQ ( 1-5years) 50/135mg, AS/AQ ( 2-11months) 25/67.5mg, AS/AQ ( 6-13yrs) 100/270mg, AS/AQ ( Adults) 100/270mg, Depo-provera, Male condoms and Microgynon 2 McCord, Joseph, Marie Tien, and David Sarley. 2013. Guide to Public Health Supply Chain Costing: A Basic Methodology. Arlington, Va.: USAID | DELIVER PROJECT, Task Order 4. 5 • It was not possible to estimate volume distributed to date through the IA since this data is not tracked by NDS. Therefore it was not possible to calculate the cost per m3 of annual throughput, (benchmark 23). • Potential for response bias as members of the CHTs leading the IA were also members of the assessment teams. 6 IA Performance: Availability and Accountability The following section presents findings on the overall performance of the IA system across the rounds in terms of availability and accountability. Six indicators (numbers 15-20) relate to the overall objective of availability while three (numbers 12-15) relate to issues of accountability. Two of the benchmarks (numbers 13 and 18) address the quality of the management of the IA supplies. Most of the benchmarks measure performance of the IA for the eight tracer commodities, presented in Table 2. (Note: These tracer commodities are the same as those used for reporting following each round of distribution.) Table 2. IA Tracer Commodities Malaria Family Planning AS/AQ ( 1-5years) 50/135mg Depo-provera AS/AQ ( 2-11months) 25/67.5mg Male Condom AS/AQ ( 6-13yrs) 100/270mg Microgynon AS/AQ ( Adults) 100/270mg Sulfadoxine 500mg + Pry 25mg (SP) A key objective of the IA, in addition to strengthening internal controls and accountability for inventory, is to ensure increased visibility into the system through enhanced quality and availability of logistics data from both the county and facility levels. The following results are derived directly from records that have been generated from each round of distribution, specifically: the IA Aggregated Tool (which collates the IA Data Form into an excel workbook), SBRRs, e-SBRRs, and NDS waybills and invoices. Availability of data varied by round and by county, particularly between the counties supported by the Global Fund and those supported by the USAID | DELIVER PROJECT. In addition, the IA Data Form, which serves as the primary source for the health facility level data, was only introduced into the system at the start of the third round. However, the quality of data collected in the IA Data Form (or when entered into the IA Aggregated Tool) varied, resulting in some records or values being dropped or modified in the final analysis. In addition to common errors, such as inconsistent spelling of locations or facility names, data quality issues included: missing values for opening balances or stock available on day of visit (physical count); negative values for days since last delivery or values outside of any reasonable range; and, inconsistent values for number of days stocked out (compared with days since last delivery). As a result, some figures differ slightly from results previously published in the round reports. 7 Product Management at Health Facilities The numbers of products distributed through a system is important to understand as it can have an impact on the overall effectiveness and efficiency of the system, as well as the ability to provide quality data for visibility. While designed to specifically support program commodities, a wide range of products are currently being distributed through the IA system. As shown in Table 3, the total number of products distributed to facilities varies significantly by county as well as by round. For example, in Bong, 50 products were distributed in Round 1, 85 products in Round 2, 115 in Round 3 but only 29 products in Rounds 4 and 5. (Note: data was not available for counties funded by Global Fund) Table 3. Number of Products Distributed in Counties by Program Type by Round County Products Groups Round 1 Round 2 Round 3 Round 4 & 5 # Total # Total # Total # Total Bong Malaria 10 50 12 85 10 115 8 7 8 3 3 ND 29 FP/RH 8 12 7 Essential meds* 22 39 81 HIV & AIDS 10 14 16 TB & Leprosy ND 4 1 Other ND 4 ND Lofa Malaria 10 36 12 84 11 88 8 8 24 10 2 2 52 FP/RH 10 10 10 Essential meds* 3 35 57 HIV & AIDS 13 16 10 TB & Leprosy ND ND 0 Other ND 11 0 Margibi Malaria 9 21 10 20 10 49 12 9 73 16 24 114 FP/RH 10 10 10 Essential meds* 2 ND 15 HIV & AIDS ND ND 12 TB & Leprosy ND ND 2 Nimba Malaria 10 17 14 48 11 41 9 7 10 3 3 33 FP/RH 7 9 10 Essential meds* ND 9 16 HIV & AIDS ND 15 3 Other 1 1 Montserrado Malaria ND 11 124 11 143 ND ND ND ND ND FP/RH ND 11 10 Essential medicines ND 51 99 HIV & AIDS ND 19 19 8 County Products Groups Round 1 Round 2 Round 3 Round 4 & 5 # Total # Total # Total # Total TB & Leprosy ND ND 4 ND NDOther ND 32 *Essential medicines figures also includes medical devices ND= no data provided Top-up supply systems such as the IA are not well suited to, or typically used for, the full commodity needs of a health system. It should be noted that decisions by local stakeholders to leverage the success of the IA to move a plethora of other needed products are understandable, but that the system, which was designed for less than 40 commodities, is straining now to handle a much broader range of essential drugs and supplies. When there are twice as many products in the system., the overall quality of data suffers, as the delivery teams make more mistakes in the physical count and in filling out records. When looking closely at the eight IA tracer products and management, most facilities receiving deliveries manage all eight products. As shown in Table 4, most facilities across the rounds manage the four presentations of AS/AQ and the majority manage SP; fewer manage family planning products. The slight fluctuation with the overall numbers between rounds can be explained by the number of facilities reached in that round (See Appendix A, Table 8 for results presented by county). Table 4. Number of Health Facilities Managing IA Tracer Products by Round by Funder Round 3 Round 4 Round 5 # of HF Managing # of HF Managing # of HF Managing Product Product Product Global Fund Counties Total Number of Facilities receiving Product 223 234 n/a AS/AQ (2-11months) 223 233 AS/AQ (1-5years) 223 234 AS/AQ (6-13yrs) 223 234 AS/AQ (Adults) 223 234 SP 196 225 Depo-provera 207 218 Male Condom 187 205 Microgynon 199 210 USAID-Funded Counties Total Number of Facilities Receiving Product 187 180 192 AS/AQ (2-11months) 184 179 191 AS/AQ (1-5years) 184 178 190 AS/AQ (6-13yrs) 186 180 192 AS/AQ (Adults) 187 180 192 9 Round 3 Round 4 Round 5 # of HF Managing Product # of HF Managing Product # of HF Managing Product SP 177 180 188 Depo-provera 184 167 174 Male Condom 180 167 175 Microgynon 183 167 172 Montserrado Total Number of Facilities receiving Product 193 152 200 AS/AQ (2-11months) 189 150 199 AS/AQ (1-5years) 190 151 199 AS/AQ (6-13yrs) 190 150 199 AS/AQ (Adults) 193 152 200 SP 188 148 197 Depo-provera 121 76 84 Male Condom 123 101 138 Microgynon 122 102 137 *Note: These numbers differ slightly from the results published in previous round reports on account of data cleaning. Certain facilities previously determined to be managing product were found not to, in fact, manage the product. Stock Availability at Health Facilities Another primary objective of the IA was to ensure commodity availability at facilities. To this extent, four of the performance indicators capture issues related to availability and stock levels: stockout rates, percent of tracer commodities understocked (<2 months), percent of tracer commodities overstocked (>4 months), and percent of health facilities that cannot calculate average monthly consumption (AMC) (due to missing or incomplete data). Stockout Rates As shown below in Figures 1 and 2, the average stockout rates across all eight tracer products have declined since the first round. Across all counties, following the second round of distributions, stockout rates vary between 20-55 percent for malaria related products and 10-30 percent for family planning commodities. When Montserrado, which constitutes almost a third of the facilities in the country, is excluded from the analysis, rates are slightly lower: 16-41 percent for malaria products and 9-32 percent for family planning commodities. The variance in the stockout rates are more closely related with the individual products then with rounds, indicating potentially on­ going/systematic challenges with those specific products. Stockout rates also vary slightly by funder. For example, in Round 4, stockout rates varied between 33-55 percent for malaria products in Global Fund supported counties, which covers approximately 230 facilities; compared with 10-27 percent in USAID supported counties, which (excluding Montserrado) covers approximately 190 facilities. In Montserrado, which includes approximately 200 facilities, stockout rates for malaria products varied between 28-55 percent in Round 4. (See Appendix A, Figures 19-24and Table 9 for results presented by funder and county). 10 Figure 1. Average Stockout Rate for Malaria Products across All Counties by Round 0% 20% 40% 60% 80% 100% 1 2* 3 4 5* Av er ag e St oc k- ou t R at e Round AS/AQ (2-11mo) AS/AQ (1-5years) AS/AQ (6-13yrs) AS/AQ (Adults) SP *Round 2 and 5 are for USAID counties only Figure 2. Average Stockout Rate for Family Planning Products across All Counties by Round 100% Av er ag e St oc k- ou t R at e 1 2* 3 4 5* 80% 60% 40% 20% 0% Round Depo-provera Male Condoms Microgynon *Round 2 and 5 are for USAID counties only Stocked According to Plan (Prior to Distribution) According to guidelines established at the start of the IA, health facilities should be maintaining between two to four months of stock (MOS) at all times. As shown in Figures 3 and 4, facilities across rounds do not maintain proper stock levels. Excluding Montserrado, over 20 percent of facilities in the Global Fund and USAID supported counties have more than four MOS of malaria product available at the time of distribution, while over 30 percent have more than four MOS of family planning products. At the same time a significant proportion of facilities are stocked out or 11 below the appropriate stock level. (See Appendix A, Table 10 for results presented by product and funder for each round). Figure 3. Health Facility Stock Levels for Malaria Products by Round by Funder 100% 80% % o f F ac ili tie s 60% 40% 20% 0% 29% 38% 20% 18% 14% 32% 42% 50% 26% 24% 29% 24% 30% 26% 23% 17% 22% 22% 31% 26% 26% 18% 11% 13% Round 3 Round 4 Round 3 Round 4 Round 5 Round 3 Round 4 Round 5 Global Fund USAID Montserrado stockout understocked (<2 month) overstocked (>4 month) appropriately stocked Missing Data-MOS/AMC cannot be calculated N/A no consumption in previous period Figure 4. Health Facility Stock Levels for Family Planning Products by Round by Funder 100% 80% stockout understocked (<2 month) overstocked (>4 month) appropriately stocked Missing Data-MOS/AMC cannot be calculated N/A no consumption in previous period 17% 28% 5% 10% 13% 19% 29% 43% 24% 21% 17% 16% 31% 40% 27% 18% 34% 32% 57% 41% 30% 18% 15% 12% Round 3 Round 4 Round 3 Round 4 Round 5 Round 3 Round 4 Round 5 Global Fund USAID Montserrado % o f F ac ili tie s 60% 40% 20% 0% Interviews conducted at the facilities corroborate the above results. At health facilities, staff reported feeling that program products, specifically malaria and family planning, had become much more available over the past year and half. Some staff also reported feeling that availability of essential medicines had improved, too, as they would often arrive along with the program products. While stockouts still occurred, staff reported that they would not last as long as before and that the DHOs and DA/DDFP would at times reallocate stock between facilities throughout the quarter to cover shortfalls. Several teams during the field visits observed this reallocation occurring. 12 Accountability of Supply Chain from County to Facility Level A key component of accountability in the system includes ensuring that products are delivered to facilities according to set schedules and set procedures. Three indicators/benchmarks capture accountability of the system from country to facility level: 1) adherence to delivery schedule, 2) delivery coverage rates and 3) the reconciliation between facility requisitions, issues from county depots, and quantities on signed waybills matching within plus or minus 5 percent. The first indicator is based on dates for each round provided by the USAID | DELIVER PROJECT and SCMU staff. The second and third are derived from the IA Aggregated Tool. Delivery Schedule and Coverage Rates Although designed as a quarterly system, the delivery of products to the county depots and to facilities did not occurr regularly every three months. As shown in Figure 5, the timing for delivery rounds was off schedule even before the Ebola emergency was declared in June 2014, occurring closer to every four months. However, Figure 6 shows coverage rates of deliveries for each round are consistently above 90 percent. CHTs reported that products would arrive from the NDS at least several days before the scheduled start of the delivery round, but they would sometimes have to wait for vehicles to arrive from Monrovia, or for the release of funding to pay for fuel (as was the case in Round 4 for Global Fund supported counties when county vehicles were used), which added to the delay. Figure 5. Delivery Schedules from Counties to Facilities 13 99% 99% 94% 95% Figure 6. Health Facility Coverage Rate by Round 97% 97%93% USAID MoHSW/GF 3 4 5 Rounds During facility visits, most staff appeared to be unaware of the quarterly schedule or to have known when to expect the next round of deliveries. Staff reported that they rarely received advanced notice of product delivery. Even when IA delivery teams came to facilities to conduct physical counts, staff claimed that little to no information was given about when the team was scheduled to return with the delivery. Order Fill Rates from County Depot to Facility An important component of building an accountable system is ensuring that the depot or warehouse fulfills orders per request which subsequently lead to facilities feeling that they can rely on the level above to adequately meet their needs. According to IA established guidelines, delivery teams are supposed to calculate and request the quantity of product necessary for the facility to carry four MOS. Since orders at times require rounding up or down to account for pack size during delivery, a plus or minus five percent buffer is permitted. Facility staff reported that they had no insight into the quantities of products that they would expect to receive until the delivery team arrived with the actual allotment. DHOs and DAs interviewed confirmed that facility staff did not view or receive a copy of the IA Data Form until products were delivered. Nevertheless, staff felt that the procedures set in place requiring signatures from the delivery team, the health facility as well as a community representative on the “IA Data Form” which serves as the facility requisition and waybill had improved overall accountability. Facility staff are more aware of the quantities and types of products being delivered while the community representative is able to share with community members what products are available at the facility. (Many facilities were able to present copies of the IA Data Form (ie. the waybills) upon request during the field visits so that data collectors could confirm signatures.) As shown in Table 5, aside from Round 4 in Montserrado, most county delivery teams do not follow order fulfillment guidelines (quantities delivered are within ±5 percent of those requested) across all products. Rates appear to vary more between rounds rather than between products indicating potential challenges with delivery teams being trained properly in the guidelines. For example, in Global Fund supported counties, during Round 3 the order-fill rates for products ranged between approximately 11-23 percent (average of 18.0 percent) while in Round 4 the rates % o f H ea lth F ac ili tie s C ov er ed 99% 99% 100% 80% 60% 40% 20% 0% 1 2 14 ranged between 8-19 percent (average 13.5 percent). Similarly in USAID supported counties (excluding Montserrado), order-fill rates for products in Round 3 ranged between 33-58 percent (average 44 percent) while in Round 4 the order-fill rates decreased to 9-25 percent (average 15.4 percent). Table 5. Percent of Orders Filled Within ±5 Percent of Requested Amount Global Fund USAID Montserrado R3 R4 R3 R4 R5 R3 R4 R5 AS/AQ (1-5years) 22.0 15.4 57.6 12.5 32.0 13.2 99.2 47.7 AS/AQ (2-11months) 11.3 19.2 39.2 9.3 22.0 10.2 96.5 42.0 AS/AQ (6-13yrs) 17.0 13.3 45.9 20 27.5 13.2 88.5 50.9 AS/AQ (Adults) 20.9 15.4 48.5 12.6 33.5 12.3 95.1 52.5 SP 22.5 8.5 33.7 11.0 4.2 2.0 83.5 17.4 Depo-provera 13.1 8.9 38.2 15.4 35.2 4.8 1.4 16.4 Male Condom (3,000) 17.4 9.2 34.5 24.6 27.3 12.8 98.3 31.3 Microgynon 20.5 18.6 57.8 18.1 27.4 16.0 93.9 59.6 Average 18.0 13.6 44.4 15.4 26.1 10.6 82.1 39.7 The use of a five percent (+/-) margin for order fill rates is rather narrow in a distribution system with a 33 percent safety stock. However, further analysis using a 10 percent margin shows that only 20 percent of all orders in Global Fund supported counties and 40 percent of orders in USAID supported counties are on target during Round 3. During Round 4, approximately 40 percent of facilities in Global Fund supported counties received more than 100 percent of the amounts of quantities requested while approximately 35 percent of facilities in USAID supported counties received 100 percent less than the quantities requested. (See Appendix A, Figures7-11 for histograms showing differences by round and funder). As one would expect, the consequences of orders being improperly filled has a direct impact on the facility inventory rules. As shown below in Figures 7 and 8, fewer than 20 percent of facilities in Global Fund supported counties are stocked appropriately to maximum (between 4 to 5 MOS) following distribution.3 The same pattern is seen in USAID-support counties as well as in Montserrado, although to a slightly lesser degree. It should be noted that a small proportion of facilities, who are stocked out of products prior to delivery continue to be stocked out of both malaria and family planning products. (See Appendix A, Table 11 for breakdown by individual products.) Additionally, a significant proportion of facilities are overstocked (i.e. have more than 5 MOS) following distribution. This might help explain why stockout rates are lower than one would expect, given the fact that deliveries have been occurring once every four to five months rather than on a quarterly basis. Conversely, the high stockout rates in Montserrado, especially when compared with the other counties, can also be explained by the fact that more delivery teams appear to be following 3 According to IA guidelines, delivery teams are supposed to provide facilities with sufficient products so that the facility has 4 MOS following the delivery. Per guidelines, products are distributed in specific pack sizes which are not supposed to be split. Therefore, due to rounding quantities for delivery up to match the pack size, it is possible for delivery teams to provide more than 4 MOS. For purposes of this analysis, therefore, it was decided that “stocked appropriate to maximum” would be considered between 4-5 MOS and over 5 months would be considered “overstock”. 15 the resupply guidelines and providing facilities with quantities to reach only the maximum level of stock (ie 4-5 MOS). Thus, with fewer quantities available to serve as a buffer stock while there are delays in the delivery schedule, stockouts will inevitably happen. Figure 7. Health Facility Stock Levels for Malaria Products Following Distribution by Round by Funder 100% % o f F ac ili tie s 80% 60% 40% 20% 0% 53% 50% 19% 18% 40% 18% 26% 29% Round 3 Round 4 Global Fund Round 3 Round 4 USAID Round 5 Round 3 Round 4 Round 5 Montserrado stocked to max (4-5Mo) stockout Not stocked to max (<4 month) overstocked (>5 month) Missing Data-MOS/AMC cannot be calculated N/A no consumption in previous period Figure 8. Health Facility Stock Levels for Family Planning Products Following Distribution by Round by Funder 100% 80% % o f F ac ili tie s 60% 40% 20% 0% 13% 12% 20% 11% 21% 25% 27% 31% Round 3 Round 4 Global Fund Round 3 Round 4 USAID Round 5 Round 3 Round 4 Round 5 Montserrado stocked to max (4-5Mo) stockout Not stocked to max (<4 month) overstocked (>5 month) Missing Data-MOS/AMC cannot be calculated N/A no consumption in previous period Accountability of Supply Chain from NDS to County As with counties, the IA framework established indicators/benchmarks to measure accountability of NDS to the counties. These included: adherence to delivery schedule from NDS to counties and “reconciliation between depot requisitions, issues from NDS, and quantities on signed waybills matching within +/- 5%”. This last benchmark is indicative of both general IA and NDS-specific performance, but is presented here rather than in the NDS chapter below. As mentioned earlier, 16 there was limited data available for orders from Global Fund supported counties in Rounds 1-3 while Round 5 data only existed for USAID counties (excluding Montserrado) only, as Round 5 for Global Fund supported counties had not occurred at the time of the assessment. Interviews with NDS staff and CHTs, made clear that no set schedule (eg. a truck is sent to Nimba the 1st week of the 3rd month and to Bong in the 2nd week of the 3rd month) was ever established for product delivery prior to the start of the distribution rounds. Rather an NDS delivery schedule is developed for each round, with order of delivery to each county varying by round. Both NDS staff and CHTs both reported that only minimal notification is provided by NDS prior to a truck arriving at the county depot. Notice, if given, was by cell phone calls the day before or day of delivery. Given the lack of an established delivery schedule, delays from the NDS could not be actually calculated. CHTs, as well as the USAID | DELIVER PROJECT and SCMU staff, reported that most products would arrive from the NDS at least several days before the scheduled start of the delivery round to the lower levels. According to those interviewed at the central level, delays in shipment from the NDS to the counties were due primarily to issues related to the release of funds from the donors and OFM or the programs’ approval processes from the rather than from any NDS internal procedure. Additionally, CHTs had minimal insight into the quantities of products being ordered until products arrive at the depot. For the first four rounds of distribution, staff from the central level (SCMU and the project) calculated the required number of products and “raised the requisition” for each of the counties, The requisitions were then sent the appropriate program for approval without first being shared with the CHTs. (It should be noted that this practice was in the process of changing; in the fifth (and sixth round) the CHTs in the USAID-support counties had more insight and involvement in raising the requisition.) Figure 9 presents the order fill rate for NDS, which is calculated by comparing the quantities requested on the county SBRR with the quantities received on the signed waybills, issued by NDS.4 Across all rounds, malaria product orders were generally filled at a rate of 80 percent or above, while only 70 percent of family planning products are within plus or minus five percent. Based on anecdotal evidence, it appears that the family planning program was making adjustments to the quantities requested in the SBRR and NDS would fill based on the approved amount; this would be a factor behind the lower percentage of order fill rates for these products. 4 Data on quantities requested and quantities received for USAID counties were copied from the Round reports and crossed checked with hard copies of the SBRRs and signed waybill stored at the NDS or in the USAID | DELIVER PROJECT office. For Global Fund counties, all data was taken directly from the hard copies and entered into an excel spreadsheet by an assessment team member. 17 72.4 Figure 9. NDS Order Fill Rates within ± 5 Percent of Order Across Rounds 100 92.5 90.0 87.5 85.4 Pe rc en t o f O rd er s F ill ed w ith in + /- 5% 80 72.2 72.7 71.0 60 40 20 0 AS/AQ AS/AQ AS/AQ AS/AQ SP Depo-provera Male Condom Microgynon ( 2-11mo) ( 1-5years) ( 6-13yrs) ( Adults) As shown in Figures 10 and 11, order fill rates have fluctuated between rounds for all products. SP, which had the lowest order fill rate of malaria products, has steadily improved to 100 percent in Round 5. For family planning order fill rates, microgynon has seen the most improvement, starting at less than 20 percent in Round 1; in Round 5, the order fill rate was 100 percent. The order fill rate for Depo-Provera started at 65 percent, increased to 100 then went down to 40 percent and has plateaued at 65 percent, which could also be a result of NDS rationing the product following a central-level stockout. (Note: Round 5 includes only orders from USAID-supported counties.) Figure 10. Order Fill Rates for Malaria Products within ±5 Percent of Order Across by Rounds 100 Pe rc en t o f O rd er s F ill ed W ith in + /- 5% 80 AS/AQ AS/AQ ( 2-11mo) 60 ( 1-5years) 40 20 0 AS/AQ ( 6-13yrs) AS/AQ ( Adults) SP 1 2 3 4 5* Rounds 18 Figure 11. Order Fill Rates for Family Planning Products within ±5 Percent of Order Across by Rounds 100 Pe rc en t o f O rd er s F ill ed W ith in + /- 5% 80 60 40 20 0 1 2 3 4 5* Rounds Depo-provera Male Condom Microgynon Summary: IA Performance on Availability and Accountability In general, availability of products has improved during the IA. Stockouts, however, are still relatively common, with rates varying between 20-50 percent for malaria related products and 10-30 percent for family planning commodities during the last two rounds. When Montserrado is excluded, rates are slightly lower—16-41 percent for malaria products and 9-32 percent for family planning commodities. Additionally, at the start of almost every distribution round, 40-60 percent of facilities where stocked out of or had less than two months of malaria products while 20-60 percent of facilities were stocked out of or had less than two month of family planning products. (Rates varied by individual products and counties). Despite this, health facility personnel across all counties consistently feel that the program products have become much more available over the past year and a half; and that stockouts, when they occur, do not last as long as they did before the IA. While successfully covering more than 90 percent targeted facilities each round, the IA has not been delivering products on schedule. Since Round 2, timing of delivery has occurred closer to once every four months in the USAID supported counties and even more time in the Global Fund supported counties, rather than every three months, as designed. In addition, most county delivery teams across rounds failed to follow order fulfillment guidelines across all products and consistently over or under supply facilities with products by more than five percent of the required amounts. Most health facility staff interviewed had limited knowledge of the IA system. Therefore, staff was unable to hold the delivery teams or counties accountable for how much product facilities received or when it would arrive. However, the adoption of the IA Data form beginning in Round 3 helped to improve accountability. Health facilities were able to produce signed copies of the IA Data Form (i.e. the delivery round waybill) showing signatures of the delivery team, the facility OIC as well as a community representative confirming receipt of the products. CHTs also had limited insight into NDS deliveries and limited ability to hold the higher level accountable. At the time of the assessment, central level teams from the USAID| DELIVER PROJECT and SCMU completed the requisitions, so most CHTs did not have visibility into the 19 final requisition or program-approved order until products arrive at the county depot. Nevertheless, based on requisition from the e-SBRRs/SBRRs and signed waybills, NDS filled program approved orders within +/- 5 percent across all rounds for most products. Finally, although the IA’s overall performance in terms of stock availability is mixed, it is important to note that in-depth quantitative analysis was successfully carried out and is evidence of the IA’s goal of increasing data visibility of the system. While the availability and quality of data varied by round and by county, a plethora of data is now generated from the IA Data Form, SBRRs, e-SBRRs, waybills and invoices strengthen accountability at each level of the system. Recommendations 1. Stakeholders at all levels should establish and adhere to a quarterly based distribution system. NDS and county delivery teams should notify the lower level facilities at least one week ahead of scheduled delivery, in order to ensure necessary reception teams can be adequately mobilized and ensure space is available. 2. Facilities should be informed ahead of time of expected amounts of products they are to receive to further strengthen accountability. For county depots, the final quantities of product approved by the programs in the e-SBRR should be shared with the CHTs when the lists are sent to the NDS to be filled. Similarly during the delivery rounds to Option 1 facilities, teams should share the quantities calculated for resupply to max with the OICs and store managers prior to leaving. (Delivery of products to Option 2 facilities happens immediately following the physical count so advance notice is not possible.) 3. Replace the IA Data form with an electronic-based application loaded onto a laptop or mobile device (tablet or smartphone) to collect health facility data. In addition to improving overall quality of data by removing the need for manually transferring data from paper to Excel, programming the application would mean that delivery teams would have fewer data points to be gathered since calculations (eg. AMC, Maximum Stock Quantity MOS, and Quantities required) would be automated. 4. Indicators—and potentially overall operations between the different counties—should be harmonized so that the timing of rounds and information collected are uniform. This would help to ensure consistency and accuracy of data used for decisionmaking. 20 Interim Approach Costs This section looks as the costs of the IA since the beginning of its implementation in October 2013 to the time of the assessment. The overall costs are described as are costs by supply chain function, by round, and partner. This is followed by looking at cost as percentage of total throughput. Total cost of the IA is $1,995,752 through June 2015. The IA is supported by a number of funding sources. The main supporters are the Global Fund, USAID, the NDS, and the Government of Liberia. USAID and Global Fund provide funds for transportation, fuel, daily subsistence allowance (DSA) (this term is used to comprise lunch money and per diem), orientation meetings, data requisition and verification costs, and miscellaneous costs including communication cards, photocopying, and stationery. Furthermore, the Global Fund supports the SCMU staff and some of their operational costs. There are also the health facility staff who are responsible for managing the inventory of commodities at the county and health facility levels. At the time of the assessment five rounds were completed in the USAID supported counties and four rounds in the Global Fund counties. These costs are reflected in the following results. Costing Methodology Data sources came directly from the Global Fund fiscal manager and his team and the USAID | DELIVER PROJECT’s detailed expenditures for each distribution round. NDS also provided financial data and the percentage of their costs which should be applied to the IA. Key informant interviews at the central level including the Office of Financial Management, county depot and health facility level supplemented the financial data. Limited financial data was available on the previous system from the NDS. Product value for 2013-2015 was provided by the NDS Operations Advisor. Because volume data is not tracked it was not possible to assess total volume and calculate Benchmark 23. Costs were categorized into the supply chain functions of distribution, storage, and management which is based on the USAID | DELIVER PROJECT’s supply chain costing methodology. Cost Results Total Supply Chain Costs (Benchmark 21) The total cost of the IA system from October 2013 to June 2015 is $1,995,752. This includes vehicle rental costs, fuel, DSA, orientation and planning meetings, NDS and SCMU operational costs, and health facility staff time cost spent on inventory management. 21 Table 6. Cost per IA Round (Round 5 includes only USAID costs) Total Cost by Round Cost (US$) Round 1 $464,496 Round 2 $435,285 Round 3 $540,368 Round 4 $403,220 Round 5* $152,385 Total $1,995,752 *Includes only USAID costs The percentage of each supply chain function is illustrated in the graph below. Distribution costs make up 40 percent of total costs followed by NDS operational expenses (26 percent), SCMU costs (19 percent), health facility staff (10 percent), and requisition and orientation meetings (5 percent). Figure 12. Supply Chain Costs by Function DISTRIBUTION, 40% REQUISITION & NDS Operational Expenses, 26% SCMU costs, 19% HF Staff Costs, 10% ORIENTATION MEETINGS, 5% 22 The following figure illustrates only Global Fund, USAID, and NDS costs. SCMU staff and operational costs are included as part of Global Fund costs since they support the SCMU. Figure 13. Supply Chain Costs by Partner (US$) Round 5 - 38,608 74,923 Round 4 101,670 95,881 166,815 Round 2 Round 3 73,973 108,181 100,382 116,838 222,076 276,495 NDS GF all USAID Round 1 84,014 111,753 229,876 ­ 50,000 100,000 150,000 US $ 200,000 250,000 300,000 When only distribution costs (including verification and orientation meetings) are included expenditures between Global Fund and USAID are closer (figure 14). Cost in general for each round has been decreasing. At the time of the assessment the Global Fund supported counties had not yet completed round 5. To date total distribution costs are $855,389. Figure 14. IA distribution costs by USAID and Global Fund by round (U.S.$) $84,014 $73,973 $108,181 $101,670 $74,923 $129,353 $117,241 $115,089 $80,947 ­ 20,000 40,000 60,000 80,000 100,000 120,000 140,000 Round 1 Round 2 Round 3 Round 4 Round 5 U S $ USAID GF w/o SCMU costs 23 Transport costs (vehicle rental, DSA, and miscellaneous costs) for each round are summarized below. Round 5 only includes USAID costs (covering 5 counties). In round 4 county vehicles were used instead of rental of vehicles in the Global Fund counties. The Global Fund paid for the fuel. Figure 15. Transport Costs by Round $185,050 $181,173 $186,030 $169,557 $63,282 ­ 20,000 40,000 60,000 80,000 100,000 120,000 140,000 160,000 180,000 200,000 Round 4 Round 5 U .S $ Round 1 Round 2 Round 3 Cost of health facility staff time spent on inventory management activities is estimated to be approximately $194,268. The average time health facility staff spend completing the daily consumption book, daily tally sheet, ledgers, and bin cards is four hours a day. The amount of time staff spend with the data collection and delivery teams during each visit is four to five hours. This includes time for offloading products, doing a physical count, and completing the IA Data Form. The average amount of staff time from the six field assessment counties were used across all of the health facilities in Liberia to estimate total staff time costs. It was assumed each health facility has one OIC and one dispenser. Because inventory management tasks specific to the IA only include keeping bin cards up-to-date, participating in the physical count, and receiving product, it was estimated this was 15 percent of a health facility person’s time and costs were adjusted accordingly using this percentage. The figure below shows costs broken down into the various cost inputs to implement distribution. Vehicle rental comprises the largest cost with 66 percent; followed by per diem, DSA, and lunch at 39 percent; and communication cards, refreshments and stationary at 5 percent. 24 Figure 16. IA Distribution Expenses by Cost Type Comm. cards, refreshments, stationary , 5% Vehicle rental (including fuel), 66% Perdiem, DSA, lunch, 29% When these distribution inputs are broken down by partner USAID funded distribution costs are slightly higher . Figure 17. IA Distribution Costs by Funding Partner (U.S. $) ­ 50,000 100,000 150,000 200,000 250,000 300,000 350,000 400,000 Vehicle rental (including fuel) Perdiem, DSA, lunch Comm. cards, refreshments, stationary U .S $ Global Fund USAID Cost per $ of Annual Throughput and Transport Costs per $1,000 of Commodities The estimated commodity throughput value since the start of the IA to June 2015 (the time of the assessment) is $13,079,090. With an estimated total IA cost of $1,995,752 supply chain costs are 15 25 percent of the value of commodity costs (Benchmark 22). In other words, it costs $0.15 to deliver one dollar’s worth of HIV, malaria, reproductive health, and TB products. Total transport costs were $785,092 or $60 per $1,000 of commodities distributed (Benchmark 24). When volume data is available the cost metric for cost per cubic meter of annual throughput may be calculated (Benchmark 23 - Cost per m3 of annual throughput). The estimated cost of the previous system for one year was provided by NDS. An equivalent time period of the IA of four rounds was used to equate one year since each round was to take place every quarter. The costs of the previous system include NDS distribution and vehicle repair, operating and administration expenses, and salaries; inventory management training and health facility staff costs to manage products. Hidden costs of the previous system not accounted for are the pick up costs by health staff and health staffs’ time away from patients when picking up commodities. Figure 18. Estimate Annual Cost of the Previous System and IA IA, 1,843,367 Old System, 2,009,456 Costs and Performance When costs are paired with performance it can be seen over time performance has improved in terms of availability of stock over the life of the IA approach. The stockout rates are an average of the tracer products within each round. Table 7. Cost Against Supply Chain Performance Total Cost by Round Cost (US$) Stockout Rates Order Fill Rate from NDS to County** Round 1 $464,496 72% 82% Round 2 $435,285 42% 85% Round 3 $540,368 22% 82% Round 4 $403,220 29% 77% 26 Total Cost by Cost (US$) Stockout Rates Order Fill Rate Round from NDS to County** Round 5* $152,385 31% 88% Total $1,995,752 *Includes only USAID costs. ** Rounds 1-3 does not include all orders from Global Fund counties Costs of the IA are generally declining as the stockout rates continued to decrease over time. Additionally, the order fill rate from NDS to the county also improved indicating accountability is also increasing. In summary, it has cost approximately $2 million to run the IA in all 15 counties through five rounds of distributions. This includes health facility staff time, distribution costs— including vehicle rental—and DSA, SCMU, and NDS operational costs. Over the four rounds that have been completed by both partners, transport costs are seeing a downward trend, while logistics performance improved in certain areas, suggesting there have been gains in efficiency and effectiveness of the IA system to deliver products. Using county vehicles rather than renting has helped keep costs down. However, this will need to be balanced with the many competing needs of county vehicles and their availability to deliver IA products in a timely manner. It costs an estimated $0.15 to deliver one dollar’s worth of IA commodities. Over time this metric can be tracked to measure the cost of the supply chain as a percentage of commodity costs. Table 7 shows availability of products and accountability has also improved over time as costs show a slow, downward trend. Recommendations 1. Track value, volume, and weight of all commodities stored and distributed by the NDS. Tracking of NDS products should be in a database that can allow instant visibility into distribution by value, volume and weight to each county and, at a later time, to each district. The database should be able to track and produce reports not only to each county but as needed by product type and time periods. It should accurately keep track of unit costs and corresponding pack sizes to produce commodity value data. 2. Update the ACCPAC accounting system to allow real time and timely access to data that can be easily imported and exported for analysis and ensure it is accessible by all appropriate personnel. Tracking by funding sources with detailed line items is essential to accurately capture expenditures and variances. If financial SOPs do not exist these should be developed for each part of the finance cycle and financial requirements needed to manage funds. 3. Produce an annual audited report of the NDS. 4. With the above recommendations pairing performance data with expenditure data, over time it will be possible to determine cost effectiveness of the NDS. Cost effectiveness can be expanded to the IA or any subsequent supply chain system if detailed and accurate expenditure data on supply chain expenditures are maintained. 5. When feasible, using GOL, NDS and/or county vehicles, rather than renting vehicles may help offset costs and improve sustainability. The use of NDS trucks in Montserrado has reduced distribution costs. 27 28 NDS Performance against Benchmarks The following section presents findings regarding the performance of the NDS against nine indicators that were embedded in the initial design of the IA and included in the IA Concept Paper. Benchmarks 1, 3, 6, 7, 8 and 10 measure whether specific recommendations for improvements to NDS facilities and procedures were successfully undertaken while numbers 2, 4, 5 and 9 are benchmarks of the performance of NDS in inventory management over the life of the IA distribution rounds. Benchmark 11 relates to the Liberia Medicine and Health Products Regulatory Authority (LMHRA) and whether it has undertaken expected inspection of facilities to determine if public sector commodities are appearing in private pharmacies. By design the NDS benchmarks are meant to have a bivariate response: either yes, the target or measure was successfully implemented/achieved, or no, it was not. In practice this may not be the most helpful level of analysis as there are compound benchmarks for which one target was achieved and another was not. The results are presented below to allow for an understanding of the degree to which each component of the compound benchmarks was successfully implemented or the target was achieved. Readers less familiar with the Liberian supply chain context should note that the NDS, for the purpose of the IA, serves as the central warehouse, and to a lesser degree, a source of vehicles for commodity distribution. It does not serve as an over-arching management unit for the supply chain. That role is provided by the SCMU of the MOHSW. The benchmarks enumerated above were included in the IA design to measure the success of planned interventions to strengthen the NDS which was a particular focus for stakeholders in 2013. NDS Strengthening Benchmarks Program products are co-located, clearly organized by batch number with bin cards All four categories (programs) of the IA commodities are located within NDS storage space adjacent to JFK hospital, but they are not located on the same floor or even in the same building. In particular, the reproductive health products are located in a different warehouse on the other side of the compound. Further, NDS utilize several “overflow” shipping containers located on the NDS compound and modified (i.e. security and temperature control) for RH products that do not fit in the main storage space. The bulkiest RH commodities— condoms—are stored on a covered dock without walls and somewhat exposed to the elements. 29 All the IA program commodities are clearly organized by batch numbers and there are accurate and up-to-date bin cards for each batch of each product. ERP data, physical count and stock card counts match +/- 5% Yes, this benchmark of the accuracy of inventory management records was achieved. As part of the assessment physical inventories were conducted for all eight tracer products, across all batches, and then compared with the ERP and stock card counts for those same products on the day of the physical inventory. In all cases the records and physical counts agreed within the +/- 5 percent margin established for this bench mark. Products are secured; locks and guard service in place Yes, this benchmark was fully met. Products for all programs are well secured by heavily strengthened external doors and further secured in internal cages. Locks were used on both external and internal doors. The external doors were secured by two internal key locks and two sheltered padlocks. The IA commodities within NDS have been protected by the PROSECOM Guard service through a contract between the MOHSW and PROSECOM. At the time of the assessment the contract was coming to its end and the management of NDS believed that negotiations were underway for a renewal of the contract. Despite these measures there was a theft in April 2014 of approximately $22,000 of RH drugs (i.e. Depo Provera). The theft was reported by NDS to the police service. The investigation is still underway. Operations Manager Hired Yes, this benchmark was met. NDS hired an Operations Manager, Thomas Wolapaye, in August 2013 and he held the job until his death in March 2015. At the time of the assessment NDS was committed to begin advertising for a replacement OM to begin in August 2015. 2 Warehouse Advisers Hired Yes, supporting partners did hire and deploy two warehouse advisers to support NDS. The first of these, Vincent Kabanda, was hired by the USAID | DELIVER PROJECT and was actively supporting NDS in the management of IA commodities at the time of the assessment. It was noted by several IA stakeholders that this support was a key to the high quality of the performance of NDS in managing IA commodities. The second adviser was provided by the Global Fund in 2014, but left after six weeks due to the outbreak of Ebola. 30 Incoming Shipping Documents, proof of delivery and receipts reconcile Yes, these records do reconcile within 5 percent for each of the eight products, for all shipments within 2015. The incoming shipping document is taken from the packing list, the proof of delivery is the physical inventory is taken on the day of receipt, and the “receipts” are taken from the ERP. The counts were not perfect, but they were well within this tolerance (5 percent). Quantities issued from NDS, quantities on signed waybills and returned stock to NDS matches +/- 5% Yes, these records do reconcile within +/- 5 percent for all eight commodities for all 15 counties for all shipments in 2015. Note that there were 0 returns of any commodity from the county depots to NDS this year. This was explained by the USAID | DELIVER PROJECT warehouse adviser as being a factor of the use of a data collection visit in advance of the delivery visit to the quantities so that excess quantities are no longer delivered. Pharmaceutical supervision team identified with clear terms of reference Yes, this benchmark was met. A pharmaceutical data verification exercise was conducted in 4th quarter 2014 with a team that was identified and recruited from staff of the Pharmacy Division, SCMU, Programs, USAID | DELIVER PROJECT and CHAI. The exercise was conducted in 10 counties receiving IA support from Global Fund and in 5 counties with the project’s support. Further, central level teams IA teams visited each county for each distribution round per the IA standard operating procedures (SOPs). These teams were instrumental in driving the IA process in ensuring that delivery routes were covered, data was collected and per diems were paid. However, differences in the timeliness and quality of implementation of these teams was noted between the project and the Global Fund supported counties. Roles and reporting between County Health Teams (CHTs) and NDS clarified Pharmacists in charge of IA commodities in the six counties visited and at NDS were consistent and clear in their understanding that NDS filled County orders placed on SBRRs following approval and signature by the programs and by SCMU. What was less clear was the depth and breadth of the role of SCMU and USAID | DELIVER PROJECT in generating key elements (i.e. AMC) of the County requisitions (SBRRs). Quantities requisitioned by county depots, issued from NDS to counties and quantities on the signed waybills match +/-5% No, this benchmark was not met. See “Accountability of Supply Chain from NDS to County” (pp 18-20) for a detailed description of performance against this benchmark. LMHRA inspections of facilities undertaken No, this benchmark was not met. Prior to the development of the IA Concept Note, the Liberia Medicines and Health Products Regulatory Authority had completed a single 5-day exercise in Monrovia to inspect private pharmacies for 31 public sector commodities that had not been approved for commercial sector distribution. The exercise was a success in that LMHRA identified and seized a large quantity of public sector health products in the private facilities. The Managing Director of LMHRA estimated they seized a quantity equal to a 20 foot storage container. However, since that exercise in 2013 they have not conducted any further inspections of private facilities because they have not received a budget or vehicles required. Summary of NDS Performance Overall, the achievement of NDS against the benchmarks set for the Interim Approach is very strong. IA commodities are well housed, managed, and secured within the NDS compound, if not co-located within the same building. The records, both manual and electronic, indicate strong inventory control overall. Further, the measures to strengthen NDS management under the IA initiative have been successfully undertaken. In particular, the deployment of the external (USAID | DELIVER PROJECT) warehouse supervisor has had unambiguous and positive impact on the NDS role in the IA. However, it should be noted that external support may also mask capacity challenges in the NDS which will be important when considering the future of the health sector supply chain within Liberia. The failure of the LMHRA to continue with periodic or ad-hoc inspections of private pharmacies attributed to lack of funding and other resources for this role is a challenge going forward to ensuring that health commodities intended for free distribution do not find their way into private pharmacies and sold for profit. The most obvious challenge to the NDS’ role going forward is the limited space available at its current compound vis a vis the needs of the IA and for the MOHSW mission overall. Recommendations 1. NDS and MOHSW should implement a space requirement exercise to determine maximum volumes of IA commodities, as well as of non-IA commodities, that the central level warehouse will need to manage going forward, and what size and design of facility this will require. This analysis should be a cornerstone of the planned initiative to update the Supply Chain Master Plan and to strengthen health commodity management in Liberia. 2. LMHRA’s mission, capacity and budget should be reviewed to determine if it will be able to fulfill its mandate to monitor and minimize leakage of public sector health commodities. 3. Stakeholders in the future of the health supply chain in Liberia should consider current staffing and systems capacities, as well as space, at NDS when developing NDS’ role in that future. 32 IA Field Findings The assessment team travelled to six counties to assess knowledge of, and adherence to, IA SOPs as well as local capacities and challenges for implementing the IA. Of particular interest were county and facility level perceptions of the IA and the type and quality of supervision at each level. Standard questionnaires were used in each county to collect the above information as well as data related to the costing component of the assessment. The following key questions were addressed by the IA assessment: 1. To what extent has the IA achieved its goals? 2. What contextual factors impact the IA supply chain? 3. What has been the effect of the IA on management and supervision capacities within the supply chain system in Liberia? 4. How is the IA perceived by staff at each level? 5. What are stakeholder perceptions of the future health sector supply chain? The IA and NDS performance chapters above provide a benchmark by benchmark answer to the first question “To what extent has the IA achieved its goals?” The identified strength of the IA is that it has improved the availability and accountability of the malaria, reproductive health, tuberculosis (TB) and HIV/AIDS commodities it supports. Further, it has increased the availability and use of logistics data for decision-making and monitoring and evaluation (M&E), and has achieved broad support and participation amongst diverse in-country stakeholders. These achievements were realized in a compressed time frame and continued during the Ebola epidemic. However, performance is uneven and the IA faces many challenges. The IA operates within a broader context of health system realities that impact the performance of the IA. Key contextual factors are presented below. What contextual factors impact the IA supply chain performance? Partner Support The different level of resources between the USAID | DELIVER PROJECT and Global Fund- supported counties is reflected in the quality and consistency in the implementation of the IA across time. Funding challenges related to the GF supported counties has led to long delays in distribution. Under IA SOPS counties are topped up to a maximum of four months of supply per delivery. GF counties waited nine months for a delivery between rounds 2 and 3 and six months between rounds 3 and 4 compromising the IA performance. Further, the USAID | DELIVER PROJECT supported counties benefit from a high and constant level of support in ensuring the machinery of the IA 33 functions according to design and schedule within five counties. There is no similar level of resource or mechanism to support the GF supported counties. LMIS The IA was designed to leverage the existing LMIS for inventory management data. Hence, challenges in the design of the old LMIS impact the IA. Staff interviewed consistently reported that the complexity of the LMIS, and the time it takes to complete the many ledgers and stock records was a serious problem. The challenge is compounded by the requirement that staff complete the all the old LMIS forms in parallel to the delivery teams completing the IA Data Form. In effect, the HFs have not benefitted as much as they might have from a supply chain designed to minimize the burden on staff at the lowest levels. Staff at health facilities reported using three hours per day, on average, to update the consumption and inventory records. The impact is seen in inaccurate inventory records (eg. bin card and physical counts do not match) leading to incorrect understanding of average consumption. Both the IA Data form and the SBRR at the facility-level are paper-based which is time consuming and lends itself to poorer data quality than would an automated system. Storage The IA relies on existing storage space at the HF, County and central levels. That capacity was observed to be a challenge at all levels in the IA supply chain, as it was for the old system. The issue is compounded by the space requirements for [Ebola] infection protection commodities at the country- and HF-level. NDS storage space is also constrained, but the physical security, inventory management, and recordkeeping for the IA commodities has been greatly improved. However, insufficient space remains the problem and the IA commodities are spread across the NDS compound in rooms, storage containers and even in outdoor (covered) platforms. Counties and HFs are similarly constrained by lack of adequate storage space. Transport The coverage and quality of the road network is a limiting factor for commodity distribution in Liberia. This is a particular challenge for the MOHSW which has an obligation to deliver supplies to the furthest counties and hardest to reach villages on roads that are destructive to transport and barely usable during the long rainy season. Keeping an adequate fleet of MOHSW vehicles on the road and ensuring they are used for the intended purpose has proved an ongoing impediment to health care commodity distribution through time. Counties have not been able to ensure adequate distribution transport and although NDS has a fleet of new and used vehicles they do not have adequate operational budgets for fuel and per diem. IA has successfully used contracted (private) vehicles, but stakeholders are understandably concerned by the costs associated with IA transport. What has been the effect of the IA on management and supervision capacities within the supply chain system in Liberia? A key attribute of top-up distribution systems, such as the IA, is low management and supervision capacity building requirements. By design delivery teams from the higher levels determine average consumption, days out of stock and current balance in order to determine the quantity to issue. HF staff are not required to learn these skills. The CHTs, with central level support, does this for the health facilities thus allowing for a lower level of capacity building for HF staff. CHTs however, are intended to be a key management and supervision level for the IA. They must fully understand the LMIS, the inventory control SOPs, timelines and transport arrangements of the system. They are responsible for keeping the HFs fully supplied to the four month maximum stock with each 34 delivery, as well as for supervising HFs in how to manage commodities and maintenance of inventory related records – particularly bin cards. Further, the CHTs are responsible for tracking county-wide consumption and balances and placing orders (SBRRs) each quarter to the central level (SCMU). Health Facility Level: At a minimum HF staff are responsible for managing their IA stocks and maintaining accurate and up-to-date stock cards and/or stock ledgers, as well as supporting the physical counts in conjunction with the county delivery teams and receiving the supplies. Additionally the HFs are responsible for the following tasks related to the IA commodities: • Organize reception teams to receive IA deliveries, conduct a physical count of the commodities and sign off that what they received is equal to what the CHT issued. • Support the IA delivery teams on the (option 1) data collection visit. • Prepare SBRRs that include the IA products. HF staff reported receiving supportive supervision from the CHTs in these tasks, but the quality and regularity of this supervision varies between counties and even between HFs within counties. The supervision is provided most often as on-the-job (OJT) training, but also, in some instances, they report receiving logistics management training outside the IA teams (i.e. SIAPS). In some facilities (i.e. Montserrado), HF staff said they did not receive any supervision or OJT from delivery teams. A consistent finding across facilities in all counties is the lack of IA SOPs. Many said they has the SOPs but very few could produce a copy at the time of visit. In practice HF staff are challenged to keep their records up-to-date and accurate. They consistently reported conducting monthly physical counts but review of stock ledgers indicate that either the counts do not consistently take place, or are not noted in the stock cards which rarely register adjustments made in line with the counts. HF capacities are further hampered by lack of LMIS forms including stock cards. How is the IA perceived by staff at each level? Health Facility Level: HF staff do not perceive an IA per se. They are not familiar with the name interim approach, nor could they relay its purpose and objectives. They did know about the CHT deliveries of the “program commodities” and were uniformly positive about the deliveries and the importance of these commodities. The biggest concern about the CHT deliveries of program commodities is late delivery as witnessed in the GF supported counties. By design the “top-up” IA makes few demands on staff at HFs and provides them with a tangible benefit in terms of commodity availability. HF staff were very positive about the deliveries, but frustrated by the complexity and time requirement to complete the LMIS. The LMIS, which is related to but not a part of, the IA design, and the general shortage of non-IA commodities, are the two key challenges cited by HF staff. County Level: Members of CHTs served both as informants and team members on the IA Assessment and in both cases expressed a high level of satisfaction with the IA. IA has improved the availability of drugs, has led to regular refresher trainings, and has, in at least one county, been seen to be a large relief as it unburdened the CHTs of the struggle to identify vehicles and funds required for distribution. A less often expressed benefit are the regular lunch fees paid to CHT members for both the data collection and commodity delivery components of each delivery round. These are significant earnings vis a vis salary levels and may be the reason such a broad spectrum of county 35 level staff are involved in the IA exercise. CHT managers in at least one instance expressed the need to vary the teams in order to “give everyone a chance.” Recommendations 1. Central level stakeholders should identify options to harmonize the IA support provided to all counties to affect a more consistent quality in implementation. A mechanism is needed to ensure that the funds required for distribution are available at least one month in advance of the scheduled round of deliveries. 2. The MOH/SCMU should lead an exercise to define what essential medicines and commodities can and should be added to the program supplies originally intended for IA distribution. The exercise needs to factor in human, physical and funding resources available for the IA going forward. 3. A logistics system design exercise should be undertaken to map out the strategies and systems required for the “post-IA” MOH supply chain to integrate all commodities into a single vision and single national health product supply chain. 4. An essential component of the future supply chain design exercise will be an in-depth assessment of all available transport capacities, where there will be a gap, and how the gap can be filled. 5. The current LMIS should be re-designed to a) reflect whatever supply chain system the MOHSW selects post-IA, and b) to relieve the exaggerated burden of reporting on HF staff required by the current LMIS. 6. Central level stakeholders should revisit the SCMP “cross-docking and hub” strategy for alleviating the acute storage capacity constraint found in the counties. 7. Core cadres of “IA leaders” for each county should be identified to increase the skills and knowledge base of those driving the implementation within the counties. 8. The SCMU and partners should develop, train and disseminate updated IA SOPs for the counties and the HFs. 9. The SOPs and training should emphasize building capacity in, and monitoring of, HF use of physical counts. What are stakeholder perceptions regarding the health supply chain? The IA Assessment team facilitated a workshop on June 25th to review the key findings of the IA Assessment, and to facilitate a discussion amongst the key stakeholders of what the core parameters of the post-IA supply chain design should be. The participants were asked to prioritize from a list of key strategies and objectives for the future supply chain. Each was given three votes. The results (see below) indicated a high-level of stakeholder buy-in to the core concepts and strategies of the Interim Approach. Product availability and accountability were the top priorities followed by the two measures for enhanced transport. The two surprises were simplicity, which was the next most prioritized objective; and cost sustainability, which was next to last in importance to the group. Key Supply Chain Strategies/Objectives • Decentralization (15) • Maximizing partner staff participation (2) 36 • Simplicity; reducing the burden of reporting on HF staff (21) • Transport availability/reliability (14) • Out-sourcing (transport, storage, etc.) (14) • Cost and sustainability (12) • Accountability (26) • Data visibility (18) • Product availability (35) Supply Chain Design Exercise The participants were organized into four groups and each group was tasked with designing a better supply chain solution for Liberia MOHSW going forward, bearing in mind the findings from the IA Assessment and the strategy priorities they had set (see above). Two groups were organized around designing improved modified top-up distribution systems, one was to design a pull” system to improve on the pre-IA system, and the last group was the “Pie-in-the-Sky” Group – tasked with thinking outside the box and designing a system from scratch. In the design the teams were asked to address the following aspects of supply chain design: • How many levels in the system (i.e. Central – County – HFs)? • Period of delivery (i.e. monthly, quarterly)? • Number of products (i.e. program dugs, essential drugs, etc)? • What LMIS (SBRR, IA, other) and what inventory control strategy (i.e. pull vs push vs top-up)? • MOHSW warehousing/transport vs outsourcing of one or more elements? • How much capacity building will be required at each level? Key Finding The primary finding from this exercise was that the participants, composed of central, county and facility level staff involved in the IA, plus donors, partners and other key stakeholders, were very homogenous in their supply chain vision. In the report out the clear preference for the new supply chain was one that will support essential drugs as well as the current program commodities, in a pull system in which lower levels determine needs and place orders to higher levels based on the data derived from the current LMIS. Further, NDS and county capacities for warehousing would be used and by and large the deliveries would be quarterly from central to county and monthly from county to HFs. Transport (private vs public) was the area in which the opinions of the groups differed most. Overall, the four groups proposed new system designs remarkably similar to the pre-IA supply chain system in Liberia. This result is not in line with participant priorities for product availability and reducing the burden of reporting at the HF level. In sum, the group would like to achieve a new outcome with the old system, which was widely criticized for not ensuring availability and for its LMIS, which is notoriously time consuming for facility staff. 37 Recommendations In the follow-on exercises to up-date the Supply Chain Master Plan and design a supply chain system going forward, designers and stakeholders should consider what the challenges were in the pre-IA supply chain and what led to the implementation of the IA. There is no evidence witnessed by the IA assessment team that the physical and managerial capacities of the NDS and MOHSW would support the operation of a new pull system built on MOHSW structures. 38 Appendix A Performance Indicators Stock Management Table 8. Number of Health Facilities Managing IA Tracer Products by Round by County Rounds 3 4 5 # of HF # of HF # of HF Bomi AS/AQ (1-5years) 24 24 AS/AQ (2-11months) 24 24 AS/AQ ( 6-13yrs) 24 24 AS/AQ (Adults) 24 24 SP 22 20 Depo-provera 15 15 Male Condom 4 4 Microgynon 9 9 Bong AS/AQ (1-5years) 32 34 38 AS/AQ (2-11months) 32 34 39 AS/AQ ( 6-13yrs) 34 35 40 AS/AQ (Adults) 34 35 40 SP 36 27 36 Depo-provera 32 29 36 Male Condom 36 28 36 Microgynon 29 35 37 Gbarpolu AS/AQ (1-5years) 10 14 24 AS/AQ (2-11months) 10 14 24 AS/AQ ( 6-13yrs) 10 14 24 AS/AQ (Adults) 10 14 24 SP 11 14 25 Depo-provera 11 14 25 Male Condom 11 14 25 Microgynon 11 13 24 39 3 # of HF Rounds 4 # of HF 5 # of HF Grand Bassa AS/AQ (1-5years) 27 27 54 AS/AQ (2-11months) 27 27 54 AS/AQ ( 6-13yrs) 27 27 54 AS/AQ (Adults) 27 27 54 SP 27 26 53 Depo-provera 26 26 52 Male Condom 25 25 50 Microgynon 26 26 52 Grand Cape AS/AQ (1-5years) 31 33 64 AS/AQ (2-11months) 31 33 64 AS/AQ ( 6-13yrs) 31 33 64 AS/AQ (Adults) 31 33 64 SP 30 31 61 Depo-provera 31 33 64 Male Condom 31 33 64 Microgynon 31 32 63 Grand Gedeh AS/AQ (1-5years) 20 20 40 AS/AQ (2-11months) 20 20 40 AS/AQ ( 6-13yrs) 20 20 40 AS/AQ (Adults) 20 20 40 SP 20 20 40 Depo-provera 20 20 40 Male Condom 20 20 40 Microgynon 20 20 40 Grand Kru AS/AQ (1-5years) 15 18 33 AS/AQ (2-11months) 15 17 32 AS/AQ ( 6-13yrs) 15 18 33 AS/AQ (Adults) 15 18 33 SP 14 18 32 Depo-provera 15 18 33 Male Condom 15 18 33 Microgynon 14 18 32 40 3 # of HF Rounds 4 # of HF 5 # of HF Lofa AS/AQ (1-5years) 59 58 64 AS/AQ (2-11months) 59 58 64 AS/AQ ( 6-13yrs) 59 58 64 AS/AQ (Adults) 59 58 64 SP 57 58 63 Depo-provera 59 58 64 Male Condom 59 58 64 Microgynon 59 58 62 Margibi AS/AQ (1-5years) 34 30 32 AS/AQ (2-11months) 34 30 32 AS/AQ ( 6-13yrs) 34 30 32 AS/AQ (Adults) 35 30 32 SP 32 30 32 Depo-provera 34 30 31 Male Condom 33 30 31 Microgynon 34 30 31 Maryland AS/AQ (1-5years) 24 25 49 AS/AQ (2-11months) 24 25 49 AS/AQ ( 6-13yrs) 24 25 49 AS/AQ (Adults) 24 25 49 SP 23 23 46 Depo-provera 22 21 43 Male Condom 22 21 43 Microgynon 22 21 43 Montserrado AS/AQ (1-5years) 190 151 199 AS/AQ (2-11months) 189 150 199 AS/AQ ( 6-13yrs) 190 150 199 AS/AQ (Adults) 193 152 200 SP 188 148 197 Depo-provera 121 76 84 Male Condom 123 101 138 Microgynon 122 102 137 41 3 # of HF Rounds 4 # of HF 5 # of HF Nimba AS/AQ (1-5years) 59 56 56 AS/AQ (2-11months) 59 57 56 AS/AQ ( 6-13yrs) 59 57 56 AS/AQ (Adults) 59 57 56 SP 59 57 56 Depo-provera 55 52 43 Male Condom 56 50 44 Microgynon 54 51 43 River Gee AS/AQ (1-5years) 20 20 40 AS/AQ (2-11months) 20 20 40 AS/AQ ( 6-13yrs) 20 20 40 AS/AQ (Adults) 20 20 40 SP 18 20 38 Depo-provera 19 19 38 Male Condom 17 19 36 Microgynon 19 19 38 Rivercess AS/AQ (1-5years) 18 19 37 AS/AQ (2-11months) 18 19 37 AS/AQ ( 6-13yrs) 18 19 37 AS/AQ (Adults) 18 19 37 SP 14 19 33 Depo-provera 15 19 34 Male Condom 12 19 31 Microgynon 14 19 33 Sinoe AS/AQ (1-5years) 34 34 AS/AQ (2-11months) 34 34 AS/AQ ( 6-13yrs) 34 34 AS/AQ (Adults) 34 34 SP 17 34 Depo-provera 33 33 Male Condom 30 32 Microgynon 33 33 42 Stockout Rates by Funder and County Results presented below for Rounds 3, 4 and 5 differ from the results published in previous Round Reports on account of data cleaning. Rounds 1 and 2 used results taken from the Rounds 1 and 2 reports Figure 19. Average Stockout Rate for Malaria Products across Global Fund Supported Counties by Round 100% 80% 60% 40% 20% 0% 1* 2† 3 4 AS/AQ (2-11mo) AS/AQ (1-5years) AS/AQ (6-13yrs) AS/AQ (Adults) SP *Data available from only 5 counties † No stock-out data available Figure 20. Average Stockout Rate for Malaria Products across USAID Supported Counties (excluding Montserrado) by Round 0% 20% 40% 60% 80% 100% 1 2 3 4 5 AS/AQ (2-11mo) AS/AQ (1-5years) AS/AQ (6-13yrs) AS/AQ (Adults) SP 43 Figure 21. Average Stock-out Rate for Malaria Products in Montserrado by Round 0% 20% 40% 60% 80% 100% 1 2 3 4 5 AS/AQ (2-11mo) AS/AQ (1-5years) AS/AQ (6-13yrs) AS/AQ (Adults) SP Figure 22. Average Stock-out Rate for Family Planning Products across Global Fund Supported Counties by Round 100% 80% 60% 40% 20% 0% 1* 2† 3 4 Depo-provera Male Condoms Microgynon *Data available from only 5 counties † No stock-out data available 44 Figure 23. Average Stock-out Rate for Family Planning Products across USAID Supported Counties (excluding Montserrado) by Round 100% 80% 60% Depo-provera Male Condoms 40% Microgynon 20% 0% 1 2 3 4 5 Figure 24. Average Stock-out Rate for Family Planning Products in Montserrado by Round 100% 80% 60% Depo-provera Male Condoms 40% Microgynon 20% 0% 1 2 3 4 5 Table 9. Stockout Rates by Product by Round and County County AS/AQ AS/AQ AS/AQ AS/AQ SP (Adults) (2-11mo) (1-5years) (6-13yrs) Round 1 Bomi 61% 100% 96% 100% 50% Bong 74% 98% 90% 93% 57% Grand Bassa 21% 29% 25% 79% 39% Grand Gedeh 22% 94% 39% 56% 41% Lofa 68% 97% 95% 97% 64% Margibi 73% 88% 91% 88% 93% Montserrado 75% 85% 80% 88% 45% Nimba 81% 97% 86% 98% 88% 45 County AS/AQ (2-11mo) AS/AQ (1-5years) AS/AQ (6-13yrs) AS/AQ (Adults) SP River Gee 63% 94% 50% 56% 59% Round 2 Bong 38% 60% 38% 48% 30% Lofa 29% 11% 9% 8% 13% Margibi 46% 63% 40% 43% 20% Montserrado 83% 88% 86% 89% 89% Nimba 33% 53% 76% 52% 35% Total 46% 55% 50% 48% 37% Round 3 Bomi 29% 63% 25% 33% 50% Bong 25% 44% 15% 21% 31% Gbarpolu 10% 40% 30% 40% 45% Grand Bassa 30% 85% 70% 85% 52% Grand Cape 6% 16% 10% 26% 40% Grand Gedeh 50% 80% 75% 80% 55% Grand Kru 0% 0% 0% 0% 29% Lofa 3% 27% 5% 8% 30% Margibi 18% 21% 9% 20% 13% Maryland 8% 54% 25% 25% 22% Montserrado 22% 38% 31% 41% 30% Nimba 17% 41% 14% 31% 17% River Gee 10% 15% 5% 5% 28% Rivercess 0% 6% 6% 22% 36% Sinoe 3% 6% 3% 9% 18% Round 4 Bomi 29% 63% 25% 33% 55% Bong 0% 26% 6% 6% 29% Gbarpolu 7% 43% 14% 50% 36% Grand Bassa 30% 37% 41% 56% 88% Grand Cape 24% 18% 6% 6% 19% Grand Gedeh 55% 75% 70% 85% 85% Grand Kru 6% 22% 17% 22% 39% Lofa 14% 19% 16% 16% 26% Margibi 7% 13% 7% 13% 27% Maryland 24% 60% 16% 28% 78% Montserrado 28% 44% 43% 55% 55% Nimba 12% 57% 9% 23% 26% River Gee 70% 75% 85% 90% 70% 46 County AS/AQ (2-11mo) AS/AQ (1-5years) AS/AQ (6-13yrs) AS/AQ (Adults) SP Rivercess 0% 47% 32% 37% 53% Sinoe 3% 3% 0% 3% 71% Round 5 Bong 0% 11% 3% 8% 19% Lofa 0% 3% 2% 2% 35% Margibi 9% 16% 13% 22% 16% Montserrado 49% 50% 51% 54% 64% Nimba 7% 59% 9% 27% 27% 47 Stock Levels at Facilities (Prior to Distribution) Results presented below for Rounds 3, 4 and 5 differ from the results published in previous Round Reports on account of data cleaning. Table 10. Health Facility Stock Levels for Tracer Products by Round by Funder Stock-out understocked (<2 month) overstocked (>4 month) appropriately stocked Missing Data- MOS/AMC cannot be calculated N/A no consumption in previous period Global Fund Counties Round 3 AS/AQ (1-5years) 35.9 29.1 18.4 9.9 3.1 3.6 AS/AQ (2-11months) 14.8 24.7 30.9 14.8 3.6 11.2 AS/AQ(6-13yrs) 24.2 29.6 25.1 9.9 3.6 7.6 AS/AQ (Adults) 31.8 31.8 17.5 11.2 3.6 4 SP 37.4 12.8 16.4 5.6 10.3 17.4 Depo-provera 20.3 33.8 22.2 15.5 7.7 0.5 Male Condom 12.3 11.8 49.7 6.4 10.2 9.6 Microgynon 18.1 26.1 31.7 10.6 7.5 6 Round 4 AS/AQ (1-5years) 40.6 34.6 12.4 9.4 3 0 AS/AQ (2-11months) 24.5 18 37.3 11.2 5.2 3.9 AS/AQ(6-13yrs) 27.8 27.4 27.8 10.7 3.8 2.6 AS/AQ(Adults) 35.9 35.5 14.5 10.7 2.6 0.9 SP 59.8 4.5 15.2 6.7 9.8 4 Depo-provera 38.5 27.5 13.3 11 7.8 1.8 Male Condom 21 14.6 46.3 6.3 7.8 3.9 48 Stock-out understocked (<2 month) overstocked (>4 month) appropriately stocked Missing Data- MOS/AMC cannot be calculated N/A no consumption in previous period Microgynon 24.8 19.5 37.1 7.1 9 2.4 USAID Round 3 AS/AQ (1-5years) 33.2 39.7 16.8 9.2 0 1.1 AS/AQ (2-11months) 14.1 21.2 39.1 17.9 1.6 6 AS/AQ(6-13yrs) 10.2 26.9 38.7 19.4 1.6 3.2 AS/AQ (Adults) 19.8 35.8 25.7 15 1.6 2.1 SP 20.5 18.2 32.4 10.8 3.4 14.8 Depo-provera 4.9 17.4 58.7 13.6 3.3 2.2 Male Condom 4.4 17.8 53.3 8.3 3.9 12.2 Microgynon 4.9 17 57.7 11 3.3 6 Round 4 AS/AQ (1-5years) 30.3 29.2 14.6 16.3 9.6 0 AS/AQ (2-11months) 9.5 16.8 38 20.7 11.2 3.9 AS/AQ(6-13yrs) 10 26.7 29.4 20.6 11.7 1.7 AS/AQ (Adults) 15.6 33.3 21.7 18.9 10.6 0 SP 23.5 11.7 24.6 13.4 21.2 5.6 Depo-provera 15.6 20.4 31.7 16.8 13.2 2.4 Male Condom 6.6 8.4 53.3 9 17.4 5.4 Microgynon 6.6 19.8 38.9 11.4 18 5.4 Round 5 AS/AQ (1-5years) 23.2 40 14.2 14.2 3.7 4.7 AS/AQ (2-11months) 3.7 19.9 44 17.3 4.2 11 AS/AQ(6-13yrs) 5.7 29.2 38 19.8 3.6 3.6 AS/AQ (Adults) 13.5 42.2 13.5 21.4 4.2 5.2 49 Stock-out understocked (<2 month) overstocked (>4 month) appropriately stocked Missing Data- MOS/AMC cannot be calculated N/A no consumption in previous period SP 23.9 19.1 22.3 5.9 11.7 17 Depo-provera 17.2 37.4 19 12.1 6.3 8 Male Condom 5.1 23.4 43.4 7.4 5.1 15.4 Microgynon 16.3 31.4 26.7 13.4 4.1 8.1 Montserrado Round 3 AS/AQ (1-5years) 37.4 33.2 16.8 8.9 3.7 0 AS/AQ (2-11months) 21.7 20.6 25.9 15.9 11.6 4.2 AS/AQ(6-13yrs) 30 33.2 18.4 10 6.3 2.1 AS/AQ (Adults) 39.9 35.2 10.9 9.3 3.6 1 SP 29.8 5.3 15.4 14.4 6.4 28.7 Depo-provera 20.7 48.8 11.6 12.4 3.3 3.3 Male Condom 25.2 19.5 25.2 13.8 6.5 9.8 Microgynon 12.3 53.3 18 11.5 4.1 0.8 Round 4 AS/AQ (1-5years) 43 33.1 7.9 7.3 8.6 0 AS/AQ (2-11months) 25.3 15.3 23.3 9.3 22.7 4 AS/AQ(6-13yrs) 40.7 34 8.7 4 11.3 1.3 AS/AQ (Adults) 53.9 30.9 5.3 4.6 5.3 0 SP 45.3 3.4 10.8 5.4 25 10.1 Depo-provera 35.5 21.1 10.5 3.9 21.1 7.9 Male Condom 28.7 16.8 21.8 3 19.8 9.9 Microgynon 24.5 40.2 10.8 6.9 14.7 2.9 50 Stock-out understocked (<2 month) overstocked (>4 month) appropriately stocked Missing Data- MOS/AMC cannot be calculated N/A no consumption in previous period Round 5 AS/AQ (1-5years) 47.7 24.1 11.6 10.1 4.5 2 AS/AQ (2-11months) 46.2 10.6 19.1 6 12.1 6 AS/AQ(6-13yrs) 47.2 20.1 16.1 7.5 7.5 1.5 AS/AQ (Adults) 50.5 26.5 9 8.5 4 1.5 SP 57.4 3.6 8.6 0.5 11.2 18.8 Depo-provera 58.3 16.7 6 4.8 8.3 6 Male Condom 35.5 14.5 15.9 6.5 13 14.5 Microgynon 40.9 21.2 10.9 5.8 13.9 7.3 51 Order Fill Rates and Stock Levels Post Distribution Figure 25. Distribution of Percent Difference in Quantity Requested and Received in Global Fund Supported Counties of all Orders-Round 3 Figure 26. Distribution of Percent Difference in Quantity Requested and Received in Global Fund Supported Counties-Round 4 52 Figure 27. Distribution of Percent Difference in Quantity Requested and Received in USAID Supported Counties-Round 3 Figure 28. Distribution of Percent Difference in Quantity Requested and Received in USAID Supported Counties-Round 4 53 Figure 29. Distribution of Percent Difference in Quantity Requested and Received in USAID Supported Counties-Round 5 54 Table 11. Health Facility Stock Levels Post Distribution by Round and Funder Stockout Not stocked to max (<4 month) overstocked (>5 month) stocked to max (4-5 months) Missing Data- MOS/AMC cannot be calculated N/A no consumption in previous period Global Fund Round 3 AS/AQ ( 1-5years) 0.9 26.0 42.6 22.4 3.1 4.9 AS/AQ ( 2-11months) 0.0 11.2 55.2 17.9 3.6 12.1 AS/AQ ( 6-13yrs) 0.0 18.8 48.4 21.1 3.6 8.1 AS/AQ ( Adults) 0.0 23.3 47.5 20.2 3.6 5.4 SP 6.2 21.5 27.2 11.8 10.8 22.6 Depo-provera 7.2 31.4 33.8 18.8 7.7 1.0 Male Condom 3.2 16.0 52.9 7.5 10.2 10.2 Microgynon 4.5 20.6 49.2 11.6 7.5 6.5 Round 4 AS/AQ ( 1-5years) 0.4 17.5 56.4 20.5 3 2.1 AS/AQ ( 2-11months) 3.4 14.2 54.5 17.6 5.2 5.2 AS/AQ ( 6-13yrs) 3 19.7 52.1 17.9 3.8 3.4 AS/AQ ( Adults) 1.3 23.1 51.3 19.7 2.6 2.1 SP 7.1 22.3 32.6 13.8 9.8 14.3 Depo-provera 2.8 23.9 47.2 15.6 7.8 2.8 Male Condom 12.2 15.1 51.7 9.3 7.8 3.9 Microgynon 3.3 23.3 52.4 9.5 9.0 2.4 USAID Round 3 AS/AQ ( 1-5years) 0.5 27.2 13.6 57.6 0.0 1.1 AS/AQ ( 2-11months) 0.0 17.9 40.2 34.2 1.6 6.0 55 Stockout Not stocked to max (<4 month) overstocked (>5 month) stocked to max (4-5 months) Missing Data- MOS/AMC cannot be calculated N/A no consumption in previous period AS/AQ ( 6-13yrs) 0.0 20.4 34.9 39.8 1.6 3.2 AS/AQ ( Adults) 0.0 24.6 25.1 46.5 1.6 2.1 SP 1.7 26.7 33 18.8 3.4 16.5 Depo-provera 0.5 14.1 58.2 21.7 3.3 2.2 Male Condom 0.6 16.7 50.6 16.1 3.9 12.2 Microgynon 0.5 13.2 56 20.9 3.3 6.0 Round 4 AS/AQ ( 1-5years) 10.7 48.9 15.2 15.7 9.6 0.0 AS/AQ ( 2-11months) 1.7 29.6 35.2 17.9 11.2 4.5 AS/AQ ( 6-13yrs) 1.7 32.8 27.2 25 11.7 1.7 AS/AQ ( Adults) 5.6 43.9 17.2 22.8 10.6 0.0 SP 8.9 30.7 22.9 10.6 21.2 5.6 Depo-provera 8.4 35.3 28.1 12.6 13.2 2.4 Male Condom 1.8 12.6 52.7 10.2 17.4 5.4 Microgynon 1.2 29.9 35.9 9.6 18.0 5.4 Round 5 AS/AQ ( 1-5years) 0.5 50 14.2 26.8 3.7 4.7 AS/AQ ( 2-11months) 0.0 17.8 45.0 22 4.2 11.0 AS/AQ ( 6-13yrs) 0.0 20.3 38.5 33.3 3.6 4.2 AS/AQ ( Adults) 0.0 36.5 17.2 36.5 4.2 5.7 SP 3.2 28.2 28.7 9.6 11.7 18.6 Depo-provera 1.1 36.8 23 23.6 6.3 9.2 Male Condom 0.0 18.3 41.7 18.9 5.1 16 Microgynon 1.2 37.2 27.9 20.9 4.1 8.7 56 Stockout Not stocked to max (<4 month) overstocked (>5 month) stocked to max (4-5 months) Missing Data- MOS/AMC cannot be calculated N/A no consumption in previous period Montserrado Round 3 AS/AQ (1-5years) 0.5 12.6 44.7 37.4 3.7 1.1 AS/AQ ( 2-11months) 1.6 6.9 47.1 27.5 11.6 5.3 AS/AQ ( 6-13yrs) 0 10.5 46.8 33.2 6.3 3.2 AS/AQ ( Adults) 1.6 9.3 43.0 40.9 3.6 1.6 SP 18.6 14.4 26.6 4.3 6.4 29.8 Depo-provera 15.7 51.2 17.4 9.1 3.3 3.3 Male Condom 1.6 18.7 39.8 22 6.5 11.4 Microgynon 0.0 17.2 33.6 42.6 4.1 2.5 Round 4 AS/AQ ( 1-5years) 0.0 4.6 11.3 72.8 8.6 2.6 AS/AQ ( 2-11months) 0.0 4.7 19.3 45.3 22.7 8 AS/AQ ( 6-13yrs) 6.0 8.0 8.7 60.7 11.3 5.3 AS/AQ ( Adults) 0.0 7.2 7.2 75 5.3 5.3 SP 0.0 10.8 18.9 8.8 25 36.5 Depo-provera 35.5 23.7 10.5 1.3 21.1 7.9 Male Condom 1.0 6.9 20.8 33.7 19.8 17.8 Microgynon 0 29.4 9.8 39.2 14.7 6.9 Round 5 AS/AQ ( 1-5years) 0.0 7.0 26.1 58.8 4.5 3.5 AS/AQ ( 2-11months) 0.5 3.0 28.1 44.7 12.1 11.6 AS/AQ ( 6-13yrs) 0.5 5.5 26.1 54.8 7.5 5.5 AS/AQ ( Adults) 0.5 10.0 21.5 60.5 4.0 3.5 SP 1.5 3.6 22.3 31.0 11.2 30.5 57 Stockout Not stocked to max (<4 month) overstocked (>5 month) stocked to max (4-5 months) Missing Data- MOS/AMC cannot be calculated N/A no consumption in previous period Depo-provera 33.3 16.7 6.0 21.4 8.3 14.3 Male Condom 0.7 5.1 23.2 32.6 13 25.4 Microgynon 0.7 16.8 18.2 34.3 13.9 16.1 58 Appendix B Key Informant Meetings Meeting with SCMU, NDS and Pharmacy Division for IA Assessment In-brief 10 June 2015 Name Org Division Title Email John Harris MOHSW SCMU Manager MOHSW johntutuharris@yahoo.com Beyan Johnson MOHSW NDS Managing Director beyankjohnson@yahoo.com Joseph Jimmy MOHSW Pharmacy Division Ass Chief Pharmacist jjosephy1958@yahoo.in Miata MOHSW Pool Fund Pool fund Manager poolfundmanager@gmail.com Yusuf Babaye USAID | DELIVER PROJECT Director ybabaye@lr.jsi.com Tapiwa Mukwashi USAID | DELIVER PROJECT Snr Logistics Advisor tapiwa.jsilib@gmail.com Meeting with GF for IA Assessment In-brief 10 June 2015 Name Org Division Title Email Dr B Dahn MOHSW Minister bdahn59@gmail.com Abrahima Diakite Cardno Local fiscal Agent Team Leader diakite@cardno.com John Harris MOHSW SCMU Manager MOHSW johntutuharris@yahoo.com 59 Meeting at CHAI for IA Assessment In-brief 10 June 2015 Name Org Title Email Simba Nyanyiwa CHAI Programme Manager Snyanyiwa@clintonhealthaccess.org Lawrence Mumbe CHAI Snr Programme Officer lmumbe@clintonhealthaccess.org John Harris MOHSW Manager MOHSW johntutuharris@yahoo.com Meeting with NMCP, NACP, RH and TB for IA Assessment In- brief 10 June 2015 Name Org Division Title Email Cell Julius Janafo MOHSW NMCP Pharmacist/SC Manager jjjanafo63@yahoo.co m 0886517601 Charles Morba MOHSW NACP SC Officer Charles_mor@yahoo .com 0886451089 Moses Toby MOHSW TB Programme Pharmacist 0886584686 Kwabena Labi MOHSW NMCP Technical Advisor klrabi@msh.org 0888392032 Tabadeh Collins MOHSW NMCP Supply Chain Ass Tabadehpeaches_koll ah@yahoo.com 0886845316 Onismus Davis USAID | DELIVER PROJECT Logistics Officer odavis@lr.jsi.com 0886855669 Meeting at Meeting with LMRHA for IA Assessment In brief 11 June 2015 Name Org Title Email Davis Sumo LMRHA Director Menmon Nduna USAID | Malaria Advisor mdunah@lr.jsi.com DELIVER PROJECT 60 Meeting at Meeting with OFM for IA Assessment In brief 12 June 2015 Name Org Division Title Email Toagore T Karson MOHSW Finance (OFM) Finance Director Tkarzon62@yahoo.com John Harris MOHSW SCMU Manager MOHSW johntutuharris@yahoo.co m Tapiwa Mukwashi USAID | DELIVER PROJECT Snr Logistics Advisor tapiwa.jsilib@gmail.com Ariella Bock JSI M & E Advisor abock@jsi.com Marie Tien JSI Technical Advisor mtien@jsi.com Meeting with SCMU for IA Assessment In-brief 11 June 2015 Name Org Division Title email John Harris MOHSW SCMU Manager MOHSW johntutuharris@yahoo.com Kpakama Kromah MOHSW SCMU LMIS Officer kpakama@yahoo.com Meeting with NDS for IA Assessment In-brief 11 June 2015 Name Org Title Email Beyan Johnson NDS Managing Director beyankjohnson@yahoo.com Ambrose Fatorme NDS Manager fatormaambrose@yahoo.com Other Key Informants Name Org Title Email Tolbert Nyentswa MOHSW Assistant Minister Shahid Muhammed WHO Logistics Coordinator shahidm@who.int Kaa Williams USAID Project Management Specialist- Malaria & other infectious kwilliams@usaid.gov 61 Name Org Title Email diseases Sando Dogba IPCA Supply Chain Specialist sando_dogba@lr.jsi.com Tara Milani USAID Health Team Leader tmilani@usaid.gov Ben Zinner USAID Team Leader bzinner@usaid.gov Christine Hersey USAID Consultant chershey@usaid.gov 62 For more information, please visit deliver.jsi.com. USAID | DELIVER PROJECT John Snow, Inc. 1616 Fort Myer Drive, 16th Floor Arlington, VA 22209 USA Phone: 703-528-7474 Fax: 703-528-7480 Email: askdeliver@jsi.com Internet: deliver.jsi.com Key Findings Total Supply Chain Costs (Benchmark 21) Cost per $ of Annual Throughput and Transport Costs per $1,000 of Commodities Costs and Performance Program products are co-located, clearly organized by batch number with bin cards ERP data, physical count and stock card counts match +/- 5% Products are secured; locks and guard service in place Operations Manager Hired 2 Warehouse Advisers Hired Incoming Shipping Documents, proof of delivery and receipts reconcile Quantities issued from NDS, quantities on signed waybills and returned stock to NDS matches +/- 5% Pharmaceutical supervision team identified with clear terms of reference Roles and reporting between County Health Teams (CHTs) and NDS clarified Quantities requisitioned by county depots, issued from NDS to counties and quantities on the signed waybills match +/-5% LMHRA inspections of facilities undertaken What contextual factors impact the IA supply chain performance? Partner Support LMIS Storage Transport What has been the effect of the IA on management and supervision capacities within the supply chain system in Liberia? How is the IA perceived by staff at each level? What are stakeholder perceptions regarding the health supply chain? Key Finding Appendix A Appendix B

View the publication