Tanzania: Integrated Logistics System Pilot-Test Evaluation: Using the Logistics Indicator Assessment Tool

Publication date: 2005

October 2005 This publication was produced for review by the United States Agency for International Development. It was prepared by the DELIVER project. TANZANIA: INTEGRATED LOGISTICS SYSTEM PILOT-TEST EVALUATION USING THE LOGISTICS INDICATOR ASSESSMENT TOOL TANZANIA: INTEGRATED LOGISTICS SYSTEM PILOT-TEST EVALUATION USING THE LOGISTICS INDICATOR ASSESSMENT TOOL The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. DELIVER DELIVER, a six-year, worldwide technical assistance support contract, is funded by the U.S. Agency for International Development (USAID). Implemented by John Snow, Inc. (JSI), (contract no. HRN-C-00-00-00010-00) and subcontractors (Manoff Group, Program for Appropriate Technology in Health [PATH], and Social Sectors Development Strategies, Inc.), DELIVER strengthens the supply chains of health and family planning programs in developing countries to ensure the availability of critical health products for customers. DELIVER also provides technical management of USAID’s central contraceptive management information system. Recommended Citation Amenyah, Johnnie, Barry Chovitz, Erin Hasselberg, Ali Karim, Daniel Mmari, Ssanyu Nyinondi, and Timothy Rosche. 2005. Tanzania: Integrated Logistics System Pilot-Test Evaluation: Using the Logistics Indicator Assessment Tool. Arlington, Va.: DELIVER, for the U.S. Agency for International Development. Abstract The Tanzania Ministry of Health, in response to decentralization, is in the process of transferring responsibility for drug management from the central level, primarily through the kit system, to districts. A new system for drug ordering, called the Integrated Logistics System (ILS), was pilot tested in Dodoma and Iringa regions from April 2005 to September 2005. In October 2005, the Pharmaceuticals and Supplies Unit of the Ministry of Health, which is responsible for implementing the ILS, conducted an evaluation of the ILS using the JSI/DELIVER Logistics Indicator Assessment Tool (LIAT). The results show that the ILS is performing as expected and meets the needs of most facilities. Health care workers overwhelmingly prefer it to the previous system. Stockout rates are about the same or a little better than under the previous system, which is an accomplishment given that the transfer of responsibility to districts has taken place. Proposed recommendations are improvements to the ILS that can be applied as it is rolled out to additional regions. No major changes are proposed. DELIVER 1616 North Fort Myer Drive, 11th Floor Arlington, VA 22209 USA Phone: 703-528-7474 Fax: 703-528-7480 Email: deliver_project@jsi.com Internet: deliver.jsi.com CONTENTS Abbreviations and Acronyms.v Acknowledgments.vii Executive Summary .1 Background .3 Impetus for Integration .3 Reliance on Organizations for Success.4 Development of the Integrated Logistics System .5 Pilot Testing.7 Methodology.8 Findings .11 Stock Status.19 Conclusions and Recommendations .25 Increase the Evaluation Period for the Next Rollout Regions, and Provide Additional Supervision .25 Review the List of Priority Items.25 Increase Availability of Priority Items at MSD .25 Extend the Length of the Training to Five Days.26 Improve Review of Reports and On-Site Supervision.26 Develop a System for Nonreporting or Unable-to-Perform Facilities .27 Reduce Complications in Start-Up .27 Improve NGO Participation.27 Include Vaccines and Tuberculosis/Leprosy in the ILS .28 Improve Monitoring and Evaluation of the ILS.28 Appendix Survey Questionnaire.29 Figures 1. Percentage of Facilities Surveyed, by Order Group .11 2. Distribution of Competency Test Scores, by Score Ranges .13 3. Percentage of Respondents Reporting a Blank Row on the Integrated Logistics System R&R by Reason and Level .16 4. Percentage of Respondents Reporting a Zero Row on the Integrated Logistics System R&R, by Reason.17 5. Percentage of Facilities Stocked Out on the Day of the Visit, by Facility Type and Program.21 6. Percentage of Facilities out of Stock on the Day of the Visit, by Program, 2003 National Baseline and 2005 Integrated Logistics System Pilot-Region Survey .22 7. Weighted Average Months of Stock on Hand, by Facility Type and Level for Integrated Logistics System Pilot Sites.23 Tanzania: Integrated Logistics System Pilot-Test Evaluation iii Tables 1. Ordering Cycle for the Integrated Logistics System.8 2. Products Selected for the Survey .9 3. Survey Teams .10 4. Number of Participants by Region and Percentage Passing .12 5. Respondent Confidence in Their Integrated Logistics System Tasks.14 6. Integrated Logistic System Ordering Formula .15 7. Stores Ledger Availability for Survey Sites and Completeness of Available Ledgers.16 8. Number of Report and Request Form Submitted, by Facility Type .18 9. Formula for Calculating the Quantity Needed in the Integrated Logistics System.19 10. Percentage of Facilities Ordering to the Maximum Level in the Integrated Logistics System, by Facility Type .20 11. Percentage of Facilities Ordering to the Maximum Level in the Integrated Logistics System for Items for Which the Facility Is Not Charged, by Facility Type .21 12. Stock Status at MSD.23 iv Tanzania: Integrated Logistics System Pilot-Test Evaluation ABBREVIATIONS AND ACRONYMS ABC analysis method based on item cost, grouping costs into A, B, and C categories AMC average monthly consumption ART antiretroviral therapy ARV antiretroviral drug CHMT Council Health Management Team DHS Directorate of Hospital Services DPS Directorate of Preventive Services EDP Essential Drugs Program EPI Expanded Program on Immunization FBO faith-based organization FP family planning ILS Integrated Logistics System IUD intrauterine device JICA Japan International Cooperation Agency JSI John Snow, Inc. LSAT Logistics System Assessment Tool MCH maternal and child health MOH Ministry of Health MSD Medical Stores Department MTUHA health management information system (Swahili name) NGO nongovernmental organization NTLP National Tuberculosis and Leprosy Program OJT on-the-job training OPD outpatient department OPV oral polio vaccine ORS oral rehydration solution PMP Pharmaceutical Master Plan PSU Pharmaceuticals and Supplies Unit R&R report and request form Tanzania: Integrated Logistics System Pilot-Test Evaluation v RHMT Regional Health Management Team SP sulfadoxine-pyrimethamine STI sexually transmitted infection TB tuberculosis USAID U.S. Agency for International Development VEN vital, essential, necessary ZTC Zonal Training Center vi Tanzania: Integrated Logistics System Pilot-Test Evaluation ACKNOWLEDGMENTS JSI/DELIVER’s Tanzania office would like to acknowledge the work of those who are at the facilities and who participated in the pilot test of the evaluation. The trainers for the integrated system, from the Zonal Training Centers in Morogoro and Iringa, are to be acknowledged for their work in transferring knowledge to the health care workers. The entire Integrated Logistics System (ILS) owes its success to the work of the Pharmaceuticals and Supplies Unit (PSU) within the Ministry of Health and to the Chief Pharmacist, in particular, whose willingness to pilot test a new drug ordering system was critical. His support and the support of his team helped in convincing regional and district-level staff members that the benefits of the new system would exceed the risks. Tanzania: Integrated Logistics System Pilot-Test Evaluation vii viii Tanzania: Integrated Logistics System Pilot-Test Evaluation EXECUTIVE SUMMARY The Integrated Logistics System (ILS), a system for reporting about use of drugs and related medical supplies and for requesting resupply, was designed to move beyond the current indent system by integrating the drugs and supplies for numerous vertical programs. Each program previously had its own method for resupply, with varying degrees of effectiveness in ensuring appropriate supplies in health facilities. In October 2003, a nationwide stock status survey was completed to serve as a baseline for the implementation of the ILS. The ILS was pilot tested in all facilities in Dodoma and Iringa regions from April to September 2005. This evaluation survey was compared to the baseline survey for some indicators. The evaluation results are based on data collected for selected items in the ILS regions and on a facility-based survey conducted at a minimum of seven sites within each of the 11 districts in the pilot study area. The survey included both quantitative questions about stock status and qualitative questions about facility-based staff attitudes toward the ILS. The findings of the survey indicate that all health workers overwhelmingly prefer the ILS to the kit system and the numerous vertical systems. Only 1 of 78 respondents did not prefer the ILS. Nearly three- quarters of the respondents additionally noted that they felt confident in their ability to implement their ILS duties. One- third responded, however, that they faced some challenges in completing the report and request (R&R) form. Overall performance reflects the feelings of the respondents. Among stores, 82 percent of facilities had stores ledgers, but only 67 percent of the ledgers were up to date. For reporting and ordering, 67 percent of facilities submitted the two expected forms, while 33 percent submitted one or none. Many of the forms contained blank rows or rows where zero was used across all columns. When asked, many of the respondents noted either that they did not manage the item or that they did not think the item was a priority. Because all the items listed (preprinted) on the form are supposed to be priority items, this result suggests that the list should be revisited and that facilities should be encouraged to order all of the items listed. The form has a preprinted formula for ordering to ensure that facilities maintain an appropriate stock level. Thus, evaluators expected that the facilities would order using the formula, and 69 percent did so. That result left 31 percent of facilities that did not order using the formula; surprisingly, even when the item was supplied at no cost to the facilities, they still did not follow the formula when ordering. Overall, the stockout rate on the day of the visit across all items was less than 20 percent. Examination of the stockout rate over a six-month period shows an increased stockout rate, but even so, the stockout rate under the ILS is as good as or better than that at the time of the baseline survey. Given that the ILS transfers responsibility for ordering to the facility (moving from a push to a pull system for at least essential drugs), this result is an excellent achievement in terms of ILS performance. The number of months of stock for the selected items (vaccines and HIV tests excepted) should be between three and seven months of supply. However, for most products, the number of months was at or near the minimum. Because the formula was not used for all products, this finding is not a surprise. Also, because facilities must pay for their drugs from an allocation and the maximum stock level of seven months exceeds the current budget allocations, this result is not surprising. Several more ordering cycles will likely be needed for facilities to build up a sufficient buffer stock for all items. Given that as of January 1, 2005, the stock of the Medical Stores Department (MSD) was low or stocked out for half of the priority items in Tanzania: Integrated Logistics System Pilot-Test Evaluation 1 the ILS, facilities are likely to have low stock levels or to be stocked out for many of the same items. Supervision of the ILS was reported by respondents as relatively weak. Only one-third of facilities received a supervision visit during the pilot period that included at least one element of logistics management (for example, stock review, order review, on-the-job training (OJT) or coaching, and removal of expired stock). Recommendations for the continued rollout of the ILS to additional regions include the following: • Increase the evaluation period for the next rollout regions to a full year, and use the additional six-month period to provide on-site supervision. • Review the list of priority items to develop a form for dispensaries that is separate from the form for health centers instead of continuing to use the current combined form. The review should also revisit the prioritization to determine whether some items can or should be removed or added. • Increase availability of priority items at the MSD so that the trickle-down effect of stockouts or understocking at facilities is minimized or eliminated for those items. • Extend the length of the training from four to five days to give participants more time to practice their ILS skills. • Improve the review of reports and on-site supervision so that incomplete forms can be completed, facilities can receive feedback when their reports are late or not submitted, and supervision activities can include logistics as a stronger component. • Develop a system for nonperforming or unable-to- perform facilities to ensure that an order is placed for the facility. • Reduce complications in start- up by developing a more- complete handout or job aid for making first orders. Respondents noted that their first orders were more complicated because of a lack of data, and it is important to build their confidence from the beginning. MSD will have to prepare itself better to fill first orders, because some facilities were hesitant to place a second order when the first order had not been received. • Improve nongovernmental organization (NGO) participation. NGOs were invited to the ILS training but did not seem to participate in the ILS. Districts will likely have to proactively approach NGOs that are authorized to order through MSD. 2 Tanzania: Integrated Logistics System Pilot-Test Evaluation BACKGROUND Beginning in February 2002, the Ministry of Health (MOH) embarked on an ambitious plan to integrate the logistics systems of many of its vertical programs. Those programs included the following: • Essential Drugs Program (EDP) (the kit or indent system, under the Pharmaceutical and Supplies Unit of the Directorate of Hospital Services [DHS]) • Family Planning Program (including contraceptives and condoms under the Reproductive and Child Health Services of the Directorate of Preventive Services [DPS]) • Sexually Transmitted Infection (STI) Program (under the National AIDS Control Program, which is a directorate-level program) • National Malaria Control Program (under the DPS) • Laboratory and Diagnostics Program (including HIV and syphilis testing, and dental and radiological supplies, under the Laboratory and Diagnostics Unit of the DHS) Additionally, the Chief Medical Officer estimated that more than a dozen vertical programs existed whose drugs and related medical supplies should be considered part of the integrated system. At the time of this initial planning, the Expanded Program on Immunization (EPI) and the National Tuberculosis and Leprosy Program (NTLP) were intentionally excluded from the ILS, under the assumption that they were performing well and had remained vertical systems for a number of years. IMPETUS FOR INTEGRATION Decentralization, as part of ongoing reform activity in the central government’s public health sector, as well as general public service reforms, transfers many formerly centralized responsibilities to the district level. Consequently, in the late 1990s, the Pharmaceutical and Supplies Unit (PSU), with support from Danida, designed the indent system to transfer responsibility for drug ordering from the central level to the district level. The indent system was intended to replace the kit system, in which dispensaries and health centers received uniform kits of drugs whose contents were determined by the PSU with the best data then available on the basis of morbidity patterns. Because the kit system had begun in 1983 as an emergency measure, facility-level staff members “lost their ability to indent,” as noted by the Chief Medical Officer. Follow-up studies of the indent system, where facilities ordered drugs that were previously in the kits, suggested that approximately 17 to 20 percent of the drugs previously shipped in kits were wasted because the uniform nature of the kits meant that no facility was likely to receive exactly what it needed. Additionally, stockouts of commonly used antibiotics in the kit system were frequent, with facilities anecdotally reporting stockouts little more than halfway through each month. The indent system allowed districts to spend drug funds according to the needs of each facility within the district, rather than in a uniform manner, which was an improvement over the kit system. The ILS takes this improvement a step farther by including most or all vertical programs and the EDP in the same system. The ILS introduces routine reporting of data coupled with routine ordering of resupplies, which enhances accountability and provides the central level with data for decision making, particularly forecasting. The routine reporting and ordering system also helps structure district-level supervision of the drug- management system. The impetus for seeking assistance from the DELIVER project of John Snow, Inc. (JSI) in creating the ILS was the impending arrival of drugs and medical supplies for the STI vertical program from Japan International Cooperation Tanzania: Integrated Logistics System Pilot-Test Evaluation 3 Agency (JICA). JICA’s donation was to include HIV tests, syphilis tests, and STI drugs for syndromic management. JICA wished to have assurances from the MOH that its donation would be well used and requested that the MOH develop a forecast of its STI and HIV drug and supply needs, and tools to manage their use, before JICA made its donation. Consequently, the MOH requested its long-term partner for logistics management for contraceptives, JSI/DELIVER, which is funded by the U.S. Agency for International Development (USAID), to assist in developing a forecast for JICA’s donation, as well as tools for the management of STI supplies. (As the former Family Planning Logistics Management Project, DELIVER had assisted the MOH in contraceptive forecasting and logistics management since the early 1990s; so the expansion to STI supplies—while a new category of supplies—was a logical extension of DELIVER’s assistance.) Because many of the drugs used in the syndromic management of STIs are also included in the essential drugs kits, because HIV test kits affect the entire laboratory and diagnostics program, and because an effective STI program should include prevention of new STIs through the use of condoms, then treating the JICA donation as part of a new integrated system, rather than as a new vertical program, was logical. Doing so would result in a single report and order for common drugs (such as cotrimoxazole) and related supplies (such as microscope slides and latex gloves), rather than requiring a separate report and order for each program. A process-mapping exercise for ordering and distributing drugs and related medical supplies for STI programs, as well drugs and supplies for other vertical programs, was conducted by the MOH with DELIVER assistance in February 2002. Process mapping revealed the individual strengths and weaknesses of each program. As a result of the process mapping, rather than an attempt to bring each program to a uniform level of performance, the MOH decided that it would be easier to combine all of the programs into an integrated logistics system, simply called the ILS. As previously noted, the NTLP and the EPI were recognized as well-functioning programs whose logistics systems did not necessarily require adaptation. Consequently, no action would be taken to modify them until the ILS was proven effective. Since that time, JSI/DELIVER has concluded that to ignore or omit those programs would be to reinforce their status as vertical programs. The programs already rely on many of the key features of the ILS (as the pilot attempted to demonstrate), and they can use the ILS without changing the way in which they manage their supplies. The ILS is a comprehensive system for drug ordering and management that is best explained as one in which no drug or medical supply is excluded. At the same time, the ILS acknowledges—through the use of subsystems that are slight modifications of the main ILS system—that not all products can be managed in exactly the same way because of considerations such as the need for the cold chain, short shelf life, and other factors. RELIANCE ON ORGANIZATIONS FOR SUCCESS Although the ILS is a system for reporting and ordering, it is not self-managing; it requires the intervention and cooperation of several different organizations. Among those organizations are the following. MEDICAL STORES DEPARTMENT As with any logistics system, the ILS relies on effective functioning of the national distribution system. The semi- autonomous Medical Stores Department was established by an act of Parliament in 1993 to replace the Central Medical Stores with a parastatal entity whose responsibilities are to procure (and clear), store, and distribute drugs and related medical supplies. Because the creation of the ILS will result in the packaging of facility-specific (that is, customized) drug kits, MSD’s role has been expanded to include the need for a packing-line/conveyor-belt system for packaging the orders. (This role can be compared to MSD’s receiving uniform kits from external sources or even to MSD’s packaging uniform kits within its facilities—the work for preparing customized drug kits is 4 Tanzania: Integrated Logistics System Pilot-Test Evaluation similar, but significantly more complex.) MSD’s current capacity in custom-kit packing is limited to the Dar es Salaam Central Store, and this same packing line is also used for the indent system (which currently serves about half of all facilities—a rapid increase since the 2002 expansion of that system). MSD has plans to implement a second packing line in Mwanza, a third in Iringa, and a fourth in Moshi, largely to reduce the burden on the Dar line. Implementation of those additional packing lines is essential to rollout of the ILS beyond the current pilot regions and is far from complete, although the Mwanza line should be operational soon. PHARMACEUTICALS AND SUPPLIES UNIT In the Pharmaceutical Master Plan (PMP) of 1992–2000, which remains in effect today (a new plan has been drafted but has not yet been published), the responsibilities of the Pharmaceuticals and Supplies Unit (PSU) are largely oversight and coordination. The PMP envisions a PSU with sufficient staffing and authority to carry out those responsibilities. During the early 1990s, much of the MOH’s emphasis and resources were geared toward MSD’s development. Consequently, PSU has remained a unit within DHS that lacks resources and sufficient authority to fill its mandate. In early 2005, PSU expanded from two to seven staff members, and plans exist to elevate the unit to subdirectorate status in the near future. PSU staff members will need additional training in logistics management functions to complement their current pharmaceutical management skills and duties. OTHER MINISTRY OF HEALTH PROGRAMS Currently, the vertical programs have primary responsibility for managing their program’s supplies. Forecasting needs, working with MSD to coordinate distribution, and collecting data from facilities largely remain spread out among the vertical programs. Some vertical programs, such as the antiretroviral therapy program in the Care and Treatment Unit, have dedicated logistics staff members, whereas most others do not. In the case of family planning (FP), the Logistics Officer position has been vacant since July 2004. DEVELOPMENT OF THE INTEGRATED LOGISTICS SYSTEM The ILS was created by taking the best elements from the vertical programs on which it was based, particularly the indent system. The ILS was also based on numerous consultations with the managers of the vertical programs as well as facility-level staff members. The steps in the development of the ILS were as follows. FORMATION OF A LOGISTICS TASK FORCE In May 2002, the PSU, with JSI/DELIVER assistance, organized the Logistics Task Force, to be chaired by the Chief Medical Officer with the purpose of guiding the development of the ILS. The Logistics Task Force met only once—in January 2003—after that initial meeting, but that meeting helped push the ILS forward, and a core group of technical people continued to steer the process. DESIGN OF THE INTEGRATED LOGISTICS SYSTEM/DESIGN WORKSHOP In October 2002, a system design workshop was held in Morogoro. Participants included MSD program managers; and facility- based staff members from hospitals, health centers, and dispensaries. The workshop’s purpose was to design the logistics management information, inventory control, transportation, and supervision systems that would complement current efforts and to provide the minimum and essential data for program management and drug ordering. The resulting design is one in which dispensaries and health centers submit reports of drug and related medical supply logistics data combined with a request for resupply using an R&R. The R&Rs are submitted to the district pharmacist, who reviews the forms and submits them to MSD. MSD next prepares a custom package of drugs for each facility and seals each order in cartons. The cartons are shipped to the district level by MSD. Districts are then responsible for delivery of the sealed cartons to the dispensary or health center. Hospitals are treated in the same manner. R&R submission is staggered so that each facility reports and requests resupply once each quarter and Tanzania: Integrated Logistics System Pilot-Test Evaluation 5 receives a resupply from MSD once each quarter. No bulk supplies are stored at the district level. DRAFT OF THE PROCEDURES MANUAL PSU, with the assistance of DELIVER, drafted a procedures manual for the ILS from June 2003 through December 2003. The manual underwent many revisions—hence, the lengthy period of time to develop it. The version of the manual used for the pilot-test consists of four sections: main text, job aids, forms, and annexes. The main text section would be referred to primarily for an initial reading and for starting up. The job aids section, which is in a document format, forms the heart of the manual by providing step-by- step instructions for each ILS process. The forms section includes copies of all ILS forms, which can be photocopied if necessary. The annexes for the manual are the subsystems of the ILS for special categories of supplies: vaccines, tuberculosis (TB) and leprosy, HIV tests, and antiretroviral drugs (ARVs). Each annex explains why those items are in a special category and how the subsystem is different from the main ILS reporting and ordering system. Only the TB/leprosy forms remain unchanged from their current design; however, the annex attempts to draw a correlation between the information on those forms and the main ILS. At the time of the pilot-test, the vaccine and TB/leprosy annexes had not been fully approved by the program. (No review of the annexes was made during the training; consequently, the systems currently in effect should not have been affected by the inclusion of those materials. It should be reaffirmed that the purpose of the annexes for vaccines and TB/leprosy is not to change those systems but to demonstrate how similar they are to the ILS and to adapt the forms only slightly to achieve the “look and feel” of the ILS.) All of the annexes are designed to resemble the main ILS system in terms of fonts and styling, as well as level of technical detail. PRIORITIZATION OF DRUGS In July 2004, a group of pharmacists and program managers met to determine which drugs and related medical supplies would be priority products for the ILS. The criteria for selection were that the item should be available at the facility at all times, should require replenishment (that is, be consumable), should be used in large volume, and should meet the health needs of patients. The process involved using analytical tools common in prioritization schemes for health supplies, and it drew on the principles involved in VEN and ABC analyses as well as throughput analysis to identify the items to be placed in the priority category for routine reporting and ordering. Items specific to each program were added to the list, making a total of 99 items for dispensaries and health centers and 166 items for hospitals that were selected to be preprinted on the R&Rs. Items that are not preprinted on the forms can still be ordered, and some items are expected to be ordered this way, particularly for hospitals. The preprinting of item names on the R&R is intended to save facilities time, to help reduce errors in order entry and packing at MSD, and to focus attention by facilities and MSD on the most-important items to be ordered routinely. PILOT-REGION SELECTION To pilot the ILS, an appropriate pilot region needed to be selected. Kilimanjaro, Iringa, and Dodoma were selected as candidate regions for many reasons—among them, accessibility from Dar by vehicle (within one day’s drive in order to facilitate supervision of the ILS), association with a Zonal Training Center (ZTC) to provide training, and service by an MSD zonal store that MSD believed could handle the change in workload. Each of the three regions was visited by PSU and DELIVER, and the regional, district, and facility-level staff members conducted a Logistics System Assessment Tool (LSAT) exercise in August 2004. The LSAT, a DELIVER-developed qualitative tool completed as a group exercise, helped point out strengths and weaknesses of the current logistics systems. As a result of completing the LSAT, regional and district managers in all three regions agreed to adopt the ILS if chosen for the pilot. DELIVER, with USAID approval, was able to support two pilot regions. Dodoma and Iringa were subsequently selected to pilot the ILS. 6 Tanzania: Integrated Logistics System Pilot-Test Evaluation A follow-up meeting with the regional and district managers was held for the selected regions to discuss the implications of moving to the ILS. The managers all agreed to implement and support the ILS. USE OF ZONAL TRAINING CENTERS AND TRAINING OF TRAINERS An administrative arrangement was reached with the ZTCs in Iringa and Dodoma, which committed their trainers to the full term of training for the ILS pilot regions. A training-of- trainers exercise was conducted in December 2004 by JSI/DELIVER, and 22 of the 24 trainers passed the two-week, competency-based course and were eligible to serve as trainers for the ILS. JSI/DELIVER also presented the training curriculum that the trainers would use to train the participants in the pilot regions, and the trainers practiced extensively with that material. CREATION OF CURRICULUM The training curriculum was written as a four-day, competency-based course. The focus of the curriculum was on the appropriate use of the procedures using job aids and not strictly memorization. The course included practical, experiential exercises that simulated what facility staff members would encounter in the course of implementing the ILS. TRANSLATION AND PRINTING OF FORMS AND MATERIALS Because Swahili is the national language, the procedures manual, workbook, and all forms in the ILS were translated into colloquial Swahili. The trainers’ curriculum was maintained in English, and the trainers translated the material as they worked. All manuals, workbooks, and forms were printed in January 2005. TRAINING Training in the pilot regions began on January 31 and ended on March 24. In all, more than 50 courses of four days each were held for Dodoma and Iringa regions. Each course was attended by approximately 25 participants, and the courses were led by a team of two or three trainers. Each dispensary was permitted to send two participants, each health center could send three, and each hospital could send up to four. All facilities were asked to send the person or people whose jobs involved ordering drugs and related medical supplies. Both the Council (district) and Regional Health Management Teams (CHMT and RHMT) were invited to attend the course as managers and supervisors of the ILS. In total, 1,181 people were trained in the ILS: 503 in Iringa and 678 in Dodoma. All course participants received an Integrated Logistics System Procedures Manual, an Integrated Logistics System Workbook, and a calculator. Participants also received sufficient copies of all of the forms they would need for one year (with some exceptions noted previously). Throughout the training, the main text and job aids were reviewed in detail for each activity and form in the ILS. Management of vaccines, TB/leprosy drugs, HIV tests, and ARVs—all of which appear in the annexes—were not reviewed during the training. Participants completed numerous practical exercises throughout the training, emphasizing completion of forms and practice with calculations. JSI staff members were present at each training venue. Two- thirds of all courses were observed in part by a member of the JSI/DELIVER technical team. Administration for all courses, including managing per diems and travel, was handled by JSI/DELIVER. PILOT TESTING Following the training and distribution of all forms, facilities in the ILS were asked to begin ordering according to the system design. That design includes reporting on a staggered basis and requesting resupply on a quarterly basis, so that all facilities report and request resupply (using the R&R) once per quarter, with one-third of the facilities (depending on group A, B, or C designation) for a single district submitting an R&R each month. Table 1, suggested by one of the trainers during the training of trainers, best represents this design. Because the training ended in March 2005, the first orders from group A were expected at MSD by April 15, 2005. On April 14 and 15, 2005, DELIVER met with the regional and district staff members in the two regions and discussed their Tanzania: Integrated Logistics System Pilot-Test Evaluation 7 preparations for implementing the ILS. Participants agreed during this meeting that the first orders would be submitted to MSD by April 29, 2005, and that all subsequent submissions would be on time. This meeting was also an opportunity to clear up any confusion after the training. Some difficulty in placing first orders was anticipated. Because the training for the ILS ended in March 2005, the pilot test ran from April to September to allow all facilities to submit two reports and to place two orders. By the time of the pilot- test evaluation survey, all groups should also have received at least one order from MSD. In June 2005, Daniel Mmari (DELIVER) and Alan Malisa (ILS trainer and Regional Pharmacist for Morogoro) visited the pilot regions to check on the status of the ILS. In July 2005, Tim Rosche (DELIVER) visited Njombe to check on the status of the ILS in that district. The current survey was intended to evaluate the results of the ILS pilot-test. The objectives were to determine how well the ILS was functioning and how facility staff members felt about the ILS and their role in this new system. METHODOLOGY SURVEY DESIGN In February 2003, DELIVER conducted a stock status assessment for a sample of drugs in 234 facilities in 13 regions and 26 districts. That evaluation’s purpose was to assess inventory control procedures and logistics management practices (ordering, distribution, supervision, and so on) within the various vertical systems and to collect data on stockout rates and duration, consumption and issue rates, stock on hand, and storage conditions. The assessment was intended to serve as a baseline for assessments of the ILS, such as the current ILS pilot-test evaluation.1 The current pilot-test evaluation survey was based largely on the indicators from the previous survey. Because those vertical systems are now integrated under the ILS, questions about separate programs were collapsed into a single set of questions about procedures and policies. 1 Ronnow, Erika, Carolyn Baer, Barry Chovitz. 2003. Commodity Availability for Selected Health Products: Baseline Survey for Integrated Logistics System. Arlington, Va.: John Snow, Inc./DELIVER, for the U.S. Agency for International Development. Recognizing, too, that the stockout situation might not be entirely indicative of the success of the ILS given stock availability issues nationwide, the survey included qualitative questions about how staff members felt about the strengths and weaknesses of the ILS compared to the vertical systems. Table 1. Ordering Cycle for the Integrated Logistics System Month R&R Status Jan. Feb. Mar. Apr. Table 2 lists the products selected for the current survey (and the overlap with the previous survey is also indicated). One of the goals of the design was to be able to compare some of the indicators between the two surveys. The list was reduced from the previous survey (vitamin A; Enzygnost HIV tests; rapid plasma reagin syphilis tests; Venereal Disease Research Laboratory syphilis tests; and diphtheria, pertussis, tetanus vaccine, as well as pediatric doses of ciprofloxacin and ceftriaxone were eliminated from this survey) and was expanded to include items preprinted on ILS forms that were not drugs (that is, medical supplies such as gloves) and several preprinted priority items for hospitals. May June July Aug. Sep. Oct. Nov. Dec. R&R submitted A B C A B C A B C A B C R&R processed C A B C A B C A B C A B Orders received B C A B C A B C A B C A 8 Tanzania: Integrated Logistics System Pilot-Test Evaluation Table 2. Products Selected for the Survey Survey Product February 2003 Survey Current Survey Microgynon X X Lo-Femenal X X Microval X X Male condom X X Depo-Provera® X X Intrauterine device (IUD) X X Ciprofloxacin 500 mg X X Benzathine penicillin 2.4 mu X X Ceftriaxone 250 mg powder X X Podophylline 10% in H2O X X Doxycycline 100 mg X X Metronidazole 200 mg X X Cotrimoxazole 400 mg/80 mg X X Sulfadoxine-pyrimethamine (SP) 500 mg/50 mg X X Oral rehydration solution (ORS) X X Measles vaccine X X Oral polio vaccine (OPV) X X Nonsterile gloves, size M X 5 ml disposable syringe X Scalp vein set X Outpatient department (OPD) cards X Field stain A X For hospitals only: Chlorpromazine 25 mg X Hyoscine-N-Butylbromide 10 mg X Sodium lactate compound (Hartmann’s) X Film x-ray 30 x 24cm X Incomplete anti-D X For sites offering HIV testing only: Capillus® X X Determine® X X Vironostika® X X Tanzania: Integrated Logistics System Pilot-Test Evaluation 9 SITE SELECTION Because the pilot test was conducted in the Dodoma and Iringa regions, the evaluation exercise was conducted in only those regions. For each of the 5 districts in Dodoma and 6 districts in Iringa (total of 11), 7 sites were selected—for a total of 77 sites. The sites were to be the district hospital, two health centers, three dispensaries, and one NGO (or faith-based organization) facility. The health centers and dispensaries were selected at random from a list of facilities receiving central-level funding allocations from the central ministry for ordering drugs through the ILS. A list of alternate health centers and dispensaries was also chosen at random to substitute in cases where no district hospital existed or where a facility was unable to be surveyed. It was decided in the survey design that if no staff member was available at a facility able to answer questions about the ILS, a substitute facility would be chosen. SURVEY TIMING The survey was conducted over the two-week period from September 26 to October 7, 2005. The timing was chosen specifically with the knowledge that all facilities that had submitted timely reports should have placed two orders and that facilities in resupply group A should also have received two orders. SURVEY TEAMS The survey teams were made up of at least two members, at least one of whom was familiar with the ILS. The teams and the districts they visited are listed in table 3. Additionally, Johnnie Amenyah and Erin Hasselberg, both from JSI, made field visits during the first week to Njombe and Kondoa, respectively. The leader for each team was already familiar with the ILS at the time of the survey. PSU staff members Winna Shango and Kitundu Shambogo were both new to PSU, and the survey exercise represented an excellent opportunity to learn about the ILS and to enlarge their understanding of challenges faced by their counterparts in pharmacies in the field. JSI/DELIVER also engaged three consultants with health backgrounds and survey experience to assist in data collection. SURVEY TEAM TRAINING To prepare the survey teams, a three-day workshop was held in the new offices of the PSU at the Mabibo complex, which also houses EPI and the Tanzania Food and Drug Authority. An overview of the ILS was provided and the survey was reviewed question by question. The training also led to minor Table 3. Survey Teams Team Leader Districta Member Survey dates: September 26–30 1 Sultan Mlandula (PSU) Leah Chenya (Consultant) I-Ludewa 2 Alan Malisa (Trainer) Winna Shango (PSU) I-Njombe 3 Sospeter Magambo (Trainer) Kitundu Shambogo(PSU) I-Makete 4 Daniel Mmari (JSI) Margareth Mrema (Consultant) D-Kondoa 5 Barry Chovitz (JSI) Ssanyu Nyinondi (JSI) D-Mpwapwa 6 Tim Rosche (JSI) Cafleen Magege (Consultant) D-Kongwa Survey dates: October 3–7 1 Sultan Mlandula (PSU) Leah Chenya (Consultant) I-Ludewa 2 Alan Malisa (Trainer) Winna Shango (PSU) I-Iringa Urban Sospeter Magambo (Trainer) Kitundu Shambogo (PSU) I-Iringa Rural 3 4 Daniel Mmari (JSI) Margareth Mrema (Consultant) D-Dodoma Rural 5 Ssanyu Nyinondi (JSI) Cafleen Magege (Consultant) D-Dodoma Urban a I = Iringa; D = Dodoma. 10 Tanzania: Integrated Logistics System Pilot-Test Evaluation adjustments of the survey questionnaire. All survey teams received an annotated version of the questionnaire to help them in understanding the meaning of each question because, although the questionnaire was created in English, the surveyors would conduct their interviews in Swahili. The notes were intended to help them rephrase the questions as necessary. At the conclusion of the first week, all surveyors met in Morogoro to collect the data from the first set of districts. The surveyors also discussed how the evaluation was proceeding and clarified any problems or concerns. A brief review of the surveys from the first week allowed the teams to improve the completeness and quality of their surveys during the second week. FINDINGS SITES Over the two-week period of the survey, the six teams collected 78 survey questionnaires used in the analysis. For Iringa Urban district, only five facilities exist and all five were included in the results. In Njombe and Kongwa, nine and eight surveys, respectively, were completed, and all were included (that is, only seven surveys were needed, but because the teams had been able to survey additional sites, they were included in the results). Because not all districts have hospitals, only 10 of the expected 11 hospitals were surveyed. The total sample included 26 health centers and 44 dispensaries. Because this survey was of the ILS, survey teams were told to ask the district level to select an NGO site to be visited. Many teams were informed that NGOs did not participate in the ILS, even where they had been trained. Consequently, the sample included only four sites not under the MOH, one of which is a parastatal (supported by the Ministry of Livestock and Agriculture rather than the MOH). JSI/DELIVER believes that the inability to identify NGOs participating in the ILS is in itself a finding and believes that the low level of participation by NGOs should be addressed. The randomly chosen sites belonged to all three groups in the ordering cycle for the ILS. As shown in figure 1, 45 percent of the selected sites coincidentally belonged to group Figure 1. Percentage of Facilities Surveyed, by Order Group 23% 19% 13% 45% A B C Don't know Tanzania: Integrated Logistics System Pilot-Test Evaluation 11 A, which should have, at the time of the survey, completed two entire order cycles (that is, submitted two R&Rs and received two orders). An important finding was that 13 percent of the sites did not know which delivery group they belonged to. Because knowing one’s delivery group determines when to submit an R&R, facilities must know to which group they belong. Because all districts confirmed in May 2005 that they had assigned all of the dispensaries and health centers they supervised to groups, this finding was unusual and suggests either that the supervisor did not inform them, or that the person questioned could not remember, or perhaps that the name of the group (A, B, or C) did not have meaning to them. Several hospitals reported that they were not assigned to any group because they were waiting for their allocation of funds before ordering, a situation that should be resolved as allocations to all facilities becomes more routine. PERFORMANCE AND PERCEPTIONS ABOUT THE ILS The purpose of the pilot-test evaluation was to examine not only how the system was functioning, but also how people felt about the ILS. This section addresses both of those areas. In terms of functioning, the ILS relies on the staff at each facility to submit reports and to make requests by completing the appropriate R&R (Form 2A for dispensaries and health centers, Form 2B for hospitals, and Form 2C used by both groups for ordering additional supplies not preprinted on Form 2A or 2B). This process contrasts to the kit system, under which facility- level staff members did not need to complete any forms to receive supplies. Because the ILS training was competency based, estimating facility-level staff members’ ability to appropriately fulfill their functions in the ILS is possible. The scores on the final, competency-based exam are listed in table 4. Interestingly, 20 percent of the participants scored 90 percent or higher on the final exam (14 percent of Dodoma and 28 percent of Iringa participants). Overall, the scores follow a generally normal distribution, as shown in figure 2, with about 30 percent failing and 20 percent doing extremely well. (The trainers for the Dodoma region used 50 percent as the passing level, whereas Iringa trainers used the stricter 70 percent rate suggested by DELIVER.) From these scores, DELIVER concludes that the final exam scores give a fair assessment of participant competency. Two important findings can be derived from these results. First, a group of staff members exists who did not gain competence in the performance of the tasks defined in the ILS; those staff members will need to be carefully supervised. This group includes nearly one-third of the people in facilities where the ILS is implemented. Possibly, and even probably, because 90 percent of those we interviewed attended the course and all but one said they had passed the exam, the tasks for the ILS are carried out by those who did pass. (In other words, because all facilities sent at least two staff members, at least one of those participants is likely to have passed the exam and to be capable of implementing the ILS, even where other members of the staff failed the exam.) Although other factors affect performance (such as availability of resources like transport), the training provided the materials for achieving an acceptable level of competency. Thus, no amount Table 4. Number of Participants by Region and Percentage Passing Region Number Trained Score of 70 Percent and Above (%) Score of 50 Percent and Above (%) Dodoma 678 60 88 Iringaa 503 76 83 Overalla 1,181 67 81 a For 9 percent of Iringa participants (4 percent of overall participants), no score was reported because of misplaced scoring sheets. 12 Tanzania: Integrated Logistics System Pilot-Test Evaluation of additional training will help those staff members who failed, and an alternative ordering method may be needed. Second, anecdotal information from the trainers suggests that the sessions may have run into the evenings and that some participants had difficulty with several of the exercises. Given this difficulty and the nearly 30 percent failure rate, some additional practical exercises would be helpful. Because the exercises mirror the tasks in the ILS, the length of the training should be reconsidered. Additional practice should result in a higher passing rate overall. The respondents in the evaluation survey appear to agree with the examination results. While 50 percent of the respondents felt that the training was “sufficient to allow you to perform your ILS tasks,” 41 percent of the respondents felt that the training was not sufficient (and 9 percent did not attend the training). In general, the results are not too surprising, because staff members often feel that training courses should be longer. This finding further supports the argument that the length and content of the training should be reconsidered. One complaint often heard about facility health staff is that turnover is high. For the survey, therefore, respondents were asked about the number of staff members trained who were still working at the facility. Overall, 8 percent of staff trained had left the facility since the training. Although this number seems relatively low, if the rate of trained staff dropout remains at the current level, at the end of three years, facilities would retain little more than half of staff members trained in the ILS. (This extrapolation assumes that the departing staff members did not transfer to another facility or did not use their ILS skills if they did. Consequently, the assumption that half of the staff members would be gone is an overestimation.) Figure 2. Distribution of Competency Test Scores, by Score Ranges 0% 10% 20% 30% 40% 50% Under 50% 50–70% 70–90% 90% + No score The situation does not seem critical at this point, although it does suggest that refresher training at least every three years would be necessary. The fact that the ZTCs already have experience in training for the ILS makes them a good resource for providing refresher and new training opportunities. Because the evaluation was also an assessment of staff attitudes toward the ILS, those interviewed were asked which system they preferred: the kit system and the vertical programs, or the ILS. Of the ILS Tanzania: Integrated Logistics System Pilot-Test Evaluation 13 district supervisors (the district pharmacists), 100 percent said they preferred the ILS. Of facility-level personnel, 99 percent preferred the ILS. When asked what they liked about the ILS, respondents noted the following: • They like the sense that the facility controls quantity and type of commodities ordered. • They feel that the ILS eliminates drugs that are not needed at the facility. • They believe that the ILS reduces the amount of expired drugs in the facility. • They feel that the ILS allows facilities to order products formerly not allowed under the kit system. Interestingly, none of those responses is or should be entirely true in the ILS. Although the facility does control the quantity it orders, the types of commodities ordered should include all of the preprinted items. Because all of the items previously in the kit also appear in the ILS, “unneeded” drugs were not eliminated; although if the calculations result in no need for additional supplies, no order will be placed for that item. It is probably too early to know whether the ILS reduces expiration of drugs, because the pilot lasted only six months. Facilities were always able to order products not included in kits, but the process was not simple and required the use of other funds. What is important is that these are the types of perceptions that match with the purpose of the ILS—to get the right quantity of the right drug of the right quality to the right place at the right time and for the right cost. Staff members in the pilot regions appear to believe that the ILS does a better job of fulfilling those six “rights” than did the combination of the kit system for essential drugs with the various vertical programs. When asked what they did not like about the ILS, respondents noted the following: • Deliveries were late or had not yet arrived. • Some deliveries were missing items that had been ordered. • No explanation from MSD was received about why some items were not included in the order. • Some deliveries included products that were close to expiring. • The allocation of funds at the district level is not fixed and, therefore, unreliable. • If you do not order an item, you do not receive it. Only the last of those responses can be attributed to the ILS, and it is true—if an order is not placed, drugs will not automatically arrive. This result is the consequence of moving from a push system to a pull system and is exactly what should be expected. As for the other responses, although they may be true, all logistics systems rely on the delivery of a complete, timely, high-quality order to be effective (that is, all six rights must be followed). Performance in the areas noted (that deliveries were late, that some items were missing) is not a consequence of implementing the ILS so much as a general management problem. The survey also probed more deeply into staff feelings and attitudes toward the ILS. Specifically, facility-level staff members were asked how confident they felt in implementing their ILS duties. As shown in table 5, 74 percent of those surveyed felt either confident or very confident. None of those surveyed responded “not at all confident,” Table 5. Respondent Confidence in Their Integrated Logistics System Tasks Trained in the ILS Totals Confidence Level Yes (n = 71) No (n = 7) Very confident/confident 76% 57% 74% Somewhat confident 24% 43% 26% 14 Tanzania: Integrated Logistics System Pilot-Test Evaluation Table 6. Integrated Logistic System Ordering Formula Beginning Balance Received This Period Lost/ Adjusted Ending Balance Estimated Consumption Quantity Needed A + B ± which is somewhat surprising, because nearly 10 percent (7 of 78) had not attended the ILS training. Because the ILS is driven by the use of Form 2, the Report & Request for Drugs and Related Medical Supplies, the survey included a question about whether staff members felt they could complete the form with or without difficulty. Two-thirds of respondents felt they could complete the form without difficulty, and one-third admitted that they had some difficulty in completing the form. When asked what was difficult about completing the form, respondents in some cases noted that the formulas of the form were difficult, but more often they found that completing the R&R for the first order was difficult. The current version of the ILS procedures manual has only a short section on making the first order, and this section is covered only briefly during the training. RECORDS AND REPORTS As noted previously, three versions of the R&R currently exist: one for dispensaries and health centers (Form 2A), one for hospitals (Form 2B), and a blank form (Form 2C). The difference among the forms is that Form 2A includes 91 preprinted items that are all considered priority items for health centers and dispensaries, and Form 2B includes 166 priority items for hospitals— which includes all priority items for health centers and dispensaries as well as additional items. Form 2C has no preprinted items and must be completed by hand. The format of the R&Rs is the same; only two calculations are required to complete the form. The formula for ordering each item is noted in table 6. The source of data for columns A, B, and C is the only other form used at the dispensary or health center level in the ILS— Form 1: Stores Ledger. Because this form is nearly identical to the MTUHA (the health management information system) Book 4: Ledger, the completion of this form should not have been new with the introduction of the ILS. The data for column D, ending balance, come from a physical inventory, which should be completed at the end of each quarter (at a minimum) for each item in the ILS and also entered into Form 1: Stores Ledger. Columns E and F are calculated from the values given in columns A–D. An additional column, G, is where the Quantity Needed (F) is rounded to the nearest unit of issue from MSD—one piece, one tin, or one bottle. AVAILABILITY AND COMPLETENESS OF STORES LEDGERS Because the stores ledger is so critical to completing the R&R (in fact, the R&R cannot be completed without data from the stores ledger), the survey included questions about the availability and completeness of the forms. As noted previously, all facilities were expected to have had significant experience using the forms over the years as part of the MTUHA or other systems. For most products, stores ledgers were available; however, this finding varies widely by program. The products are grouped by program, because— as can be seen from the data— the effectiveness of the ILS is somewhat program-specific, despite the ILS aim of treating all items equally. Table 7 shows the percentage of facilities for which stores ledgers were available and whether or not the ledger was up to date. As can be seen from the data, ledgers for the laboratory program items (field stain A, incomplete anti-D) had the lowest availability and were unlikely (just less than 70 percent) to be up to date where they did exist. Although only two items are in this category, the results are similar to the previous stock status survey that C – D = E F = (E ÷ 3) x 7 – D Tanzania: Integrated Logistics System Pilot-Test Evaluation 15 Figure 3. Percentage of Respondents Reporting a Blank Row on the Integrated Logistics System R&R by Reason and Level 0% 10% 20% 30% 40% 50% 60% 70% 80% No t n ee de d No t m an ag ed No t a pr ior ity Fu nd ing MS D did no t h av e Pe rc en ta ge o f F ac ili tie s R ep or tin g District Hospitals Health Center/Dispensary concluded that the laboratory and diagnostics program has a weak logistics system. The family planning program had high availability of ledgers (90 percent) but the lowest percentage up to date (58 percent). The vaccine program had the highest ledger availability, but ledgers were unlikely (just less than 70 percent) to be up to date. Given the recent lack of full supply of contraceptives, one might conclude that a psychological element is at work here; where a program experiences a national stockout of some items, the entire program’s success is diminished because staff members are less motivated when they do not receive supplies. Because the nurses who complete FP forms are usually the same as those who complete vaccine forms, the data suggest that the differences in records being up to date might be, in part, attributable to difficulties in the national contraceptive stock availability. Table 7. Stores Ledger Availability for Survey Sites and Completeness of Available Ledgers Items by Program Ledger Available? (%) Ledger up to Date?a (%) Family planning 90 58 Essential drugsb REPORT AND REQUEST FORM COMPLETENESS All of the items on Forms 2A and 2B are considered priority 89 72 STI drugsb 60 74 Vaccines 96 69 Consumables 69 64 Laboratory 37 69 HIV test kits 58 91 Overall 82 67 a Only asked where the ledger is available. b Benzathine penicillin, ceftriaxone, podophylline, and ciprofloxacin were grouped as STI drugs because of their use primarily for the STI program, whereas metronidazole, cotrimoxazole, and doxycycline were grouped under essential drugs. 16 Tanzania: Integrated Logistics System Pilot-Test Evaluation items; consequently, all of them should be ordered each quarter. For the survey, therefore, the completeness of R&Rs was examined. Reviews of R&Rs submitted for the first orders received by MSD showed that many facilities had either left some rows partly or completely blank or entered zero for most or all of the elements in a row. During the ILS training, participants were told that if the item was not managed at the facility, a blank row would be acceptable; however, the number of products affected was expected to be few and the exception (for example, a Catholic-supported hospital might not offer IUD insertion). Figure 3 shows the reasons given for a blank row. Figure 4 illustrates that the reason for leaving a blank row of “not needed” occurred at more than half of all facilities surveyed. Although it is entirely possible that no new supplies were needed for an individual item, it is, nevertheless, necessary to report about stock levels for the item and to demonstrate, through the calculations, that new supplies are not needed. In other words, the data suggest that staff members “looked” at the quantity of an item and decided that an order was not necessary, rather than reporting the data, which the central level still needs to have, and proving to themselves and the district level that an order is not needed. Too large a percentage of facilities (40 percent of dispensaries and health centers and about 30 percent of hospitals) noted that they left blank rows because the items were “not managed.” This result leaves unclear whether the facilities did not offer the item, whether perhaps it had been out of stock for so long that they felt it was no longer among available items, or whether the item was related to a clinical skill that the facility staff no longer possessed. For example, all facilities are intended to offer syndromic management of STIs, which would include use of podophylline. However, podophylline has been out of stock for so long at MSD that facility staff members may say that they do not manage this item—whereas, in fact, if facilities offer syndromic management, podophylline should be available. Even if it were available, whether clinical staff members would know how to effectively use it is not clear. This result may be the consequence not only of a failure of the logistics system but also of a lack of staff knowledge in the use of an item. Clearly, nearly one-third of all dispensaries and health centers do not agree with the prioritization of items, because they responded that they left a blank row if they felt the item was not a priority. Some combination of adjustment to the list of priority items and training of staff members in the use of priority items is needed. For example, IUDs, which appear on 0% 10% 20% 30% 40% 50% 60% Entirely stocked out New product for this facility Did not think this information was necessary Other Pe rc en ta ge o f F ac ili tie s Reasons Assigned Figure 4. Percentage of Respondents Reporting a Zero Row on the Integrated Logistics System R&R, by Reason Tanzania: Integrated Logistics System Pilot-Test Evaluation 17 the list of priority items for dispensaries and health centers, clearly are not a managed product for dispensaries because they do not offer this service. Some of them, consequently, may have given not a priority as a response because they cannot offer the service. Figure 4 shows the reasons given by respondents for a zero row. Because the surveyors were asked to look at the most recent R&R (which should have been the second R&R), no facility was expected to experience a stockout for any of the priority items. However, this finding was clearly not the case. For more than 20 percent of the facilities, the facility began and ended the quarter entirely out of at least some items, resulting in a row of all zeroes. Again, products like podophylline, which were entirely stocked out at MSD, are among those for which an all zero row would be expected. More than half of the facilities reported a zero row because the product was new to them. This result is a double-edged outcome—while it is a plus that the ILS has opened the door to ordering new items that the facility would value, it is a negative that by the time of the second order the facility still had none on hand. As with a blank row, more than a quarter of the facilities used zero for products they felt were not a priority, which again suggests the need to reinforce (and perhaps adjust) the concept of priority items. These findings again have a largely psychological component to them—where facility staff felt that the item was not needed, or where they decided that item was not a priority, the R&R was left incomplete. But because the ILS is both a reporting and an ordering system, the need to report information even when an order is not needed must be emphasized. REPORTS SUBMITTED AND THEIR TIMELINESS As has been noted throughout, by the time of the survey in late September or early October, all facilities in all three ordering groups should have completed and submitted two complete orders to the district, where they would be reviewed and submitted to MSD for fulfilling and delivering. Table 8 shows the number of R&Rs that the respondents said they submitted. At the dispensary and health center level, submission of R&Rs is pretty good at just over 70 percent, but since districts do not hold any buffer stock for facilities. This result means that as many as one-third of the facilities would not receive an order at all. Submissions from hospitals are more difficult to characterize. Because hospitals have always ordered drugs on a pull system, they have used their available transport to their advantage and believe they can place orders as needed without using a specific form. Although having extra transport is great, that transport could be used more effectively by placing routine orders for all products at the same time, rather than at different times for different programs or categories. Anecdotally, some hospitals reported that they had not used the R&R because either they were told they were not in the ILS by MSD or they decided because they had not received an ILS allocation of funding that they could not order. (This assumption is incorrect. Hospitals do not need a special ILS allocation in order to use the funds they have on account with MSD.) ILS success depends not only on submission of the R&R but also on the timely submission of those reports. The staggered delivery system and MSD’s lead- time require that R&Rs be submitted on time. More than two-thirds (70 percent) of Table 8. Number of Report and Request Form Submitted, by Facility Type Number of R&Rs Submitted Facility Type District Hospital (N = 8)l Dispensary/Health Center/Other (N = 70) Total (N = 78) 0 or 1 75% 29% 33% 2 25% 71% 67% 18 Tanzania: Integrated Logistics System Pilot-Test Evaluation Table 9. Formula for Calculating the Quantity Needed in the Integrated Logistics System Ending Balance Estimated Consumption Quantity Needed facilities reported that they did not submit their reports on time. The primary reason (37 percent of responses) was that the report itself was not completed on time. In about 10 percent of responses, the person trained in the ILS was not in the facility when the report was due. Only one respondent noted transport as a problem, which was surprising given general anecdotes of transport difficulties. Several respondents forgot the deadline or said they did not know when it was. Again, this finding is largely a psychological one—no particular reason exists as to why the report could not be completed on time by the trained person. An important finding to note here is that not all facilities had received their first orders when it was time to place the second order, and some facilities waited until the first order had been received before placing a second order. This situation was particularly true for group B facilities, whose orders were delivered late because of MSD’s annual stocktaking. It is as important for MSD to fulfill its role in delivering to districts as it is for facilities to order on time. REPORTS REVIEWED AT THE DISTRICT LEVEL As noted previously, R&Rs from dispensaries and health centers are submitted for review at the district level by the district pharmacist. This review represents an opportunity for desk-based supervision (as compared to on-site supervision) and an opportunity to ensure that funding is used appropriately, because the district is responsible for funding decisions. The survey asked respondents about reviews and the reasonableness of those reviews. From the facility level, three- quarters (75.38 percent) responded that their order had been reviewed at the district level with the district pharmacist. The main reasons for not receiving a review were related to time—either the district pharmacist was not available or the facility staff member could not stay because of other commitments. Overall, if the 75 percent level can be maintained, it should help boost the effectiveness of the ILS. Because the ILS is a new system, the length of time needed for reviews was expected to be high, particularly during the pilot phase. At the district level, 6 of the 10 respondents reported that they spent more than one hour reviewing each report. Only 1 of the 10 respondents said that he did not review the reports, and the remaining 3 respondents spent between 10 minutes and an hour on each review. Surprisingly, 5 of the 10 district pharmacists responding said that despite the length of time needed to review the report, the length of time was very reasonable” or reasonable. Two believed that the length of time was not at all reasonable. As the system improves, the length of time needed for reviews likely will decline. The amount of time needed to complete and review first orders is clearly quite high. STOCK STATUS INVENTORY CONTROL PROCEDURES The purpose of the ILS, as with any logistics system, is to ensure that the right goods of the right quantity of the right quality are delivered to the right place at the right time for the right cost—the six rights. Chief among logistics functions, therefore, is inventory control. All items in the ILS should be appropriately stocked at each facility. The worst outcome would be a stockout. An overstock, particularly for items with short shelf lives, is also important to avoid. To help facilities make sure they maintain stocks so that they experience neither a stockout nor an overstock, the ILS includes an inventory control system in its design. The heart of the inventory control system is that all facilities are required to submit a report and place an order every three months: a forced-ordering max-min system. The formula for ordering drugs is built into the formula on the R&R shown in table 9. Using the formula, facilities would place an order for a seven- month maximum. The reasons for ordering seven months of stock include the following: D E F = (E ÷ 3) x 7 – D Tanzania: Integrated Logistics System Pilot-Test Evaluation 19 Table 10. Percentage of Facilities Ordering to the Maximum Level in the Integrated Logistics System, by Facility Type Ordered to Maximum? District Hospital Dispensary/Health Center Total Yes 40% 69.23% 68.57% No • MSD requires five weeks to receive an order, fill the order, and ship it to the district. The district requires up to two weeks to deliver orders to all of its facilities for one ordering group (A, B, or C). This period is the lead-time, and the total of seven weeks is rounded up to two months of supply. • Sufficient stock will be needed for use during the three months of each quarter. • Because facilities are ordering only on the basis of the most recent three months of information, that recent information may not be entirely representative of the facility’s needs. A buffer stock must be maintained to account for increases in consumption and delays in ordering. As a general guideline, at least one- half of the review period (here, three months) should be maintained as a buffer. Rounding up because of uncertain circumstances results in a two-month buffer. As a result, the maximum is, therefore, 2 + 3 + 2, or seven months of supply for the maximum. In reality, no facility will ever have seven months’ supply on hand because the request will take nearly two months to be processed before it is received. Facilities will, therefore, have at most five months of supply in reality. Max-min inventory control can be a difficult concept to understand. Particularly where facilities previously received two kits, one per month, delivered six times per year, the process of ordering to a maximum of seven months may appear to be costly and unmanageable. However, as explained, unless more frequent (and, therefore, costly) orders are placed, seven months is a reasonable stock level when placing quarterly orders. During the training, max-min inventory control is explained briefly to participants. They are encouraged to rely on the formula and should always order to the maximum. In the pilot stage, it seemed likely that facilities might be confused or might choose not to order to the maximum despite the instruction. The survey, therefore, included questions about how facilities implemented the use of max-min inventory control. The results are given in table 10. As with ordering, district hospitals were less likely to follow the procedures in the ILS than dispensaries or health centers. This result may again be caused in part by district hospitals’ believing that because they can make more frequent orders, the maximums need not be followed. The reasons for not ordering to the maximum were not requested during this survey, but it was known that the needed quantities, ordered to the full seven months, might exceed the current budgets. This obstacle will be overcome through successive orders—buffers for each item will be built up over time, rather than through the first orders. To ensure that facilities would have at least the appropriate amount of essential drugs, kits were delivered in advance of first orders to increase the buffers for those items. Surprisingly, however, many facilities did not order to the maximum, even for items for which the facility was not charged, as shown in table 11. Given that the items are delivered at no cost to the facility, no reason existed for not ordering those items to the maximum. However, as the data show, the percentages are similar to those for items for which a charge is made. When asked, respondents noted reasons such as the following: they felt their 60% 30.77% 31.43% 20 Tanzania: Integrated Logistics System Pilot-Test Evaluation previous order was sufficient, they did not know it was important, and the items were slow-moving. Those responses do not make sense because the R&R should be completed in any case. Again, an element of psychology seems to be at play here—facilities are trying to predict needs on the basis of personal experiences. Although personal experience is a valid source of information, the formulas in the ILS help take the guesswork out of making those determinations, particularly where staff turnover is high; staff members are not well trained; and the most-experienced staff members are needed for other, equally important work. (One caveat here is that the district Maternal and Child Health coordinators had stocks of FP items that they wanted to distribute to clear out their storerooms in preparation for moving to the ILS. Orders for FP items on the ILS R&Rs were often supplied by the district, therefore, and were deleted from the ILS R&Rs. This situation explains some, but not all, of the orders not made to maximum, particularly when the item was not charged to the facility.) STOCKOUT RATES As noted previously, stockouts are the most serious negative outcome in a logistics system. Stockout information is simple to collect and was collected at the facilities visited during the survey. A stockout on the day of the visit was defined as not having any available stock on the day that the surveyors arrived. (In a few cases, sealed boxes of supplies were at the facility but unopened. Those supplies were counted as not being available, because it was not clear that the item needed was in the sealed box.) Figure 5 lists the results, with products grouped by program. If the ILS were performing entirely as expected, all items from all programs would have about the same (low) level of stockouts. However, this chart again suggests although reporting and ordering are integrated in the ILS, the performance on a per item basis depends on the program and staff members who support it. (Vaccines were usually not ordered through the ILS but through the routine vaccine system.) As previously discussed, Table 11. Percentage of Facilities Ordering to the Maximum Level in the Integrated Logistics System for Items for Which the Facility Is Not Charged, by Facility Type Ordered to Maximum for No-Charge Items? District Hospital Dispensary/Health Center Total Yes 40% 67.79% 65.71% No 60% 34.39% 34.29% Figure 5. Percentage of Facilities Stocked Out on the Day of the Visit, by Facility Type and Program 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% Family planning Essential drugs Vaccines Consumables Laboratory HIV test kits All programs Program Category Pe rc en ta ge o f f ac ili tie s ou t o f s to ck District Hospital Health Center/Dispensary Tanzania: Integrated Logistics System Pilot-Test Evaluation 21 Figure 6. Percentage of Facilities out of Stock on the Day of the Visit, by Program, 2003 National Baseline and 2005 Integrated Logistics System Pilot-Region Survey stockouts of contraceptives are more attributable to a national shortage of some contraceptives than to ILS ordering. (Figure 7 represents only facilities that manage the product; therefore, for the category laboratory at the dispensary or health center level, it accounts for stockouts of field stain A, but only for the 26 facilities that manage that product.) These results can be compared to the results from the 2003 stock status assessment, as shown in figure 6. The most significant finding of this evaluation, as this figure suggests, is that the ILS is performing at least as well as the previous kit system and vertical programs. Given that this is pilot-test of a new ordering system and is based on a pull system (that is, driven by the facility), this result is a remarkable achievement, particularly because we believe that as long as stocks remain available at the national level, facilities will continue to perform better and better in the ILS over future orders. It demonstrates that facilities are doing a good job overall in ordering the items they need, within budget, as compared to waiting passively for the arrival of uniform kits that will result in waste of unnecessary items or in stockouts of items that are needed. STOCKOUTS IN THE PAST SIX MONTHS Because the ILS pilot-test took place over a six-month period, stockout levels could be assessed over the entire period. Not surprisingly, stockout levels for family planning (37 percent), essential drugs (17 percent), STI drugs (44 percent), vaccines (17 percent), consumables (22 percent), and HIV test kits (16 percent) all increased over the longer period. Stockouts would be expected to be more likely in the earlier phase of the ILS. (Labs are not included here because—although the stockout rate was 0 percent over six months—only few facilities managed the only product listed, field stain A.) MONTHS OF STOCK As a final check of inventory control, the survey collected data to calculate the number of months of supply on hand on the day of the visit, using estimated consumption over the past six months. Estimated consumption was calculated using the quantities issued from the stores ledgers and were adjusted for stockouts by dividing by the number of weeks that the stock was available. The result was used to determine the average monthly consumption, or AMC. Finally, the current stock on hand was divided by the AMC to calculate the months of supply. Figure 7 indicates the range of stock on hand. The FP, essential drug, and STI programs seem to be well stocked at the dispensary and health center level. The maximum stock level for immunizations, not indicated, is 1.5 months, and the maximum 9 15 7 13 22 7 9 8 0 5 10 15 20 25 30 Pe rc en ta ge Baseline (national) ILS (pilot) Family planning Essential drugs Vaccines HIV test kits 22 Tanzania: Integrated Logistics System Pilot-Test Evaluation Tanzania: Integrated Logistics System Pilot-Test Evaluation 23 vailability t ), o tire ILS. ing ity- Figure 7. Weighted Average Months of Stock on Hand, by Facility Type and Level for Integrated Logistics System Pilot Sites 0 1 2 3 4 5 6 7 8 Fa mi ly pla nn ing Es se nti al dru gs ST I Im mu niz ati on Co ns um ab le La bo rat ory Te st kit s To tal M on th s of s to ck District Hospital HC/Dispensary stock level for HIV tests is 3 months. The vaccine program, by this measure, is slightly overstocked, and the HIV test kit program slightly understocked. The lab program is understocked for all levels. District hospitals remain understocked, which, as noted from other findings, is likely indicative of their ability to get resupplied more frequently—an idea not promoted by the ILS, but a reality for some districts. By this performance measure, the ILS STOCK STATUS AT MEDICAL STORES DEPARTMENT Key to the ILS is the a of stock at the central MSD. A stockout at MSD will result in a subsequent trickle-down effect a facilities, making reduced stock levels or stockouts much more likely. Although a stockout at the central level is less serious than a stockout in a facility (because customers can get served even when MSD is out of stock as long as the facilities have stock the long lead-times for MSD to procure new supplies suggest that MSD stockouts are likely t have serious, negative consequences for the en Table 12 shows stock status at MSD of the 166 priority items for hospitals (which include all dispensary and health center items) as of January 1, 2005. The results were calculated us sales (issues) from MSD for the previous 12 months. Use of those data was necessary because MSD does not yet have facil level data on use. (One appears to be doing well. Table 12. Stock Status at MSD Supply Status Percentage (%) Products (n) Percentage of products stocked out 11 18 Percentage of products with less than 9 months of stock 40 67 Percentage of products with 9–24 months of stock 28 47 Percentage of products with greater than 24 months of stock 20 33 Missing product (client card) 1 1 Total 100 166 component of the ILS wo m data on use, and such a database is currently being implemented at MSD and is expected to be collecting data for the ILS regions by the end of 2005 or early in 2006.) Sales from MSD are used as a proxy for this information. The 11 percent stocked out is unacceptable uld be a ethod to collect facility-level because those 18 or n r ard, ly a two- at ger in getting As with any management n is an the . The that a row of the form, if desired, for terms ta the the ansport at ot ts, ere done: 0 accuracy, as well as the entire form to get a sense of what is happening at the facility in of consumption of drugs. The data included in the R&R give the supervisor a relatively clear picture of the movement of products are all priority items f facilities. Products with less tha nine months of stock would be in urgent need of replenishment, because the procurement cycle for the MOH—while shorter fo in-country suppliers—can actually be longer than one year for the entire tendering, aw production, and receipt process. Items with greater than two years’ supply (more than 24 months) are at risk for expiration, because most essential drugs have on year shelf life. It is true th contraceptives and consumable goods such as gauze have lon shelf lives, but if the items are paid for by the MOH, this stock level represents funds tied up in inventory rather than appropriately spent. The missing product was a problem data from MSD’s database and is not significant. SUPERVISION system, supervisio important element to keep ILS working appropriately ILS includes the following elements to help keep supervision going strong: The R&R was designed so supervisor can review each supplies (that is, no essential da are missing from the form). The form should be delivered to the district level for review by the facility that is working with the supervisor. As can be seen from the findings, those elements are working well in the ILS. On-site supervision is where supervisor visits the facility to assess its functioning. For ILS, only about one-third of all facilities had received an ILS- specific supervision visit in the past six months. Of those, most had had only one visit. The ILS procedures manual recommends visits at least once per quarter and provides a checklist for supervision. Many more supervision visits are conducted in the country because an established supervision tr plan exists. However, when respondents were asked wh activities were carried out during the visit, the activities were n what would be needed to constitute an effective supervisory visit for the ILS. For most of the additional visi none of the following w check inventory, verify ledger entries, remove expired stock, review R&R, or provide OJT or coaching. Consequently, only 3 percent of facilities received an ILS visit that was effective in managing the logistics system. 24 Tanzania: Integrated Logistics System Pilot-Test Evaluation CONCLUSIONS AND RECOMMENDATIONS Based on the observations in the field during the pilot test; the feedback from regional and district-level managers, MSD staff members, and ILS trainers; and, primarily, the results of this evaluation survey, DELIVER recommends the following actions. INCREASE THE EVALUATION PERIOD FOR THE NEXT ROLLOUT REGIONS, AND PROVIDE ADDITIONAL SUPERVISION This evaluation exercise demonstrated that completing only two order cycles was not long enough to formally determine the success of the ILS. Moreover, visits to the field for this evaluation demonstrated the need for an increased emphasis on supervision during the start- up period for any region. During the rollout of the ILS to additional regions, rather than focus on evaluating the ILS, a similar exercise should be held whose purpose is to provide on- site supervision of facilities, particularly those facilities whose orders were late or not received. For more-effective evaluation of the functioning of the ILS in a region, a one-year period should be allowed before assessing performance. The purpose of this action is to improve start-up of the ILS in new regions and to ensure that future evaluations allow enough time for the region to perform as would routinely be expected. REVIEW THE LIST OF PRIORITY ITEMS Currently, 99 items are on the list of priority items for dispensaries and health centers; 166 items are on the list of priority items for hospitals. This survey included many examples of items that facilities said they do not manage. The preprinted priority lists should be reserved for those items for which a facility should never be out of stock. Items such as ballpoint pens, OPD cards, field stain A, and podophylline might not be among this group because they may be ordered infrequently or in small quantity, or they might not be expected to be found in all facilities. This recommendation may be carried out by reconvening the group (or a similar group) that has sufficient authority to review the list of priority drugs and related medical supplies or by adopting another mechanism that will achieve the broad consensus needed to determine the products that should be included in the priority lists. Separate lists would very likely be needed for dispensaries and health centers. Items such as IUDs, for example, would not appear on the dispensary form because dispensaries are not expected to maintain IUDs in stock on the basis of the training and skills of the service providers at that level. This differentiation between levels would also reduce the ordering burden because all of the preprinted items should most likely be ordered every quarter. Other items are probably not maintained at the majority of health centers, such as field stain A, which is used only at facilities with a laboratory. By reducing the number of items, the idea will be reinforced that all preprinted items should be reported about each quarter (that is, columns A–E of the R&R completed), even if an order for that item is not needed. This change will reduce the number of blank or zero rows. The purpose of this recommendation is to focus both MSD and facilities on priority items. INCREASE AVAILABILITY OF PRIORITY ITEMS AT MSD As was noted in the findings, the unavailability of stock at MSD will result in stockouts and confusion for facilities placing Tanzania: Integrated Logistics System Pilot-Test Evaluation 25 orders. MSD must have at least 9 months of stock of all priority items on hand and generally no more than 24 months of supply. This requirement could mean an enormous effort in terms of time evening hours (for some courses). Simply extending the length of a training course will not necessarily result in a better outcome; however, the curriculum appears to have and funding for quantifying and procuring up to 166 products. rs to s ability it and iate stock y he til the overestimated participants’ ability to absorb the material s ricts t of d e their scores on implications for the cost, as does the PSU proposal to add the topic of rational drug use to the course, and those implications should be considered when extending the course. IMPROVE REVIEW t s. ility-level off-site (Although a meeting with supervisors was held at the training for the t a ch MSD should understand the importance of priority items and should focus its attention on those most important items. Currently, the complete MSD catalog contains several hundred items, and many more are in the database system. MSD management should work with the ILS manager and other managerial staff membe achieve a full supply of the items in the ILS. Ultimately, a loss in confidence about MSD’ to deliver priority items could erode confidence in the ILS, so is important to achieve maintain an appropr status. The purpose of this action is to ensure that priority items are appropriately stocked at MSD. EXTEND THE LENGTH OF THE TRAINING TO FIVE DAYS Not uncommonly, participants feel that training courses should be longer. As was noted previously, 41.03 percent of the survey respondents said that the course was not sufficient to allow them to complete their ILS duties. They noted specificall that they needed more reinforcement for completing t R&R (72.5 percent) and for dealing with mathematics and calculations (47.5 percent). The trainers, anecdotally, reported that their courses went un quickly. (In fact, the original plan was for a three-day course, which was extended to four day after the exercises needed for skills development were added.) As is generally well known, a significant number of participants were medical attendants from rural dist that are geographically difficul to access. Invariably, those medical attendants have only a one-year orientation in general clinical support services following their primary-level education. This cadre forms the bulk of the health work force in all rural health facilities. Because most would not be proficient in even basic mathematics, they would need additional time and exercises in completing each column of the R&R. More time would be needed to work on additional examples and exercises to bring them to a higher level of competency, particularly in beginning with the use of Form 1, Stores Ledger, and Form 2, R&R. A course five full days, therefore, is recommended to allow for additional practice exercises and to give more time for review of the existing exercises. The purpose of this action is to improve participants’ comfort with the activities in the ILS an to help improv the final exam. This recommendation does have OF REPORTS AND ON-SITE SUPERVISION The findings show that nearly one-third of all facilities did no place two orders; those that did, did not place those orders on time. Consequently, they are likely to stockout of some item For facilities to understand the importance of routine ordering, districts will need to provide a higher level of follow-up with the facilities they supervise. The training course does not address supervisors in sufficient detail. Therefore, at the conclusion of all fac courses, which district pharmacists should attend, a separate course of at least two days should be held for DHMT and RHMT staff members. They should discuss how to handle nonreporting facilities, late reporting by facilities, review of orders, and effective ILS supervision. The course should include both on- and supervision. The course can also be used to address start-up issues. conclusion of the pilot regions, it was no training course. This recommendation is to create su a course.) The purpose of this action is to improve the quality and quantity of reports received by MSD and 26 Tanzania: Integrated Logistics System Pilot-Test Evaluation to promote effective ILS supervision. DEVELOP A SYSTEM FOR NONREPORTING OR UNABLE-TO- PERFORM FACILITIES Nearly one-third of participan (30 percent) failed the final competency exam. Even if th training is extended to five d (as recommended), a core grou of staff members will likely remain who simply cannot master the materials in t sufficiently to complete their IL tasks. No matter how many follow-up activities, supportiv supervision visits, OJT sessions, or refresher trainings are conducted, the performance of some staff members will not improve to an acceptable level. For facilities where no staff member ts e ays p he ILS S e can complete the ILS forms, the PSU should consider could visit the facility during the ush left to eeds . receive a elated e the he s. The visits should also include al e on of NGOs should the following possibilities: District-level staff members appropriate time and could complete all forms on the facility’s behalf. Facility-level staff members could bring the Form 1: Stores Ledger booklets to the districts, as well as the results of a physical inventory. The district could then complete the form while working with the staff members. (The district pharmacists were given an Excel® spreadsheet to help facilitate this process.) Districts could complete a default order for any facility whose order is late or not submitted. Although this solution is, in effect, a return to a p system, it would at least ensure that some drugs are received. The development of the default order contents could be the district pharmacist, who would be familiar with the n of similar facilities in the area None of those options is a particularly good choice for handling the facilities that are without appropriately skilled staff members. However, the purpose of this action is to ensure that all facilities timely order of drugs and r medical supplies. REDUCE COMPLICATIONS IN START-UP First orders under the ILS are unique and proved far more complicated than routine orders than was anticipated. This outcome resulted in part because ledgers might not have been started or might not be up to date for all products. Anecdotally, some facilities spent a great deal of effort trying to out-think th ILS and even manipulated formulas in reverse to force t math to work. The current manual has only two pages on first orders. Either a separate, detailed handout or a job aid should be provided during training, or the manual should include an entire chapter for start-up activities. During start-up in new regions, additional follow-up visits from PSU will be necessary to make sure that the rollout is smooth. Using ILS trainers to assist in this activity may be helpful. Facilities and district pharmacists are likely to appreciate this extra assistance, because the maintenance of buffer stocks and the use of an ordering form will be new concepts to the facilitie working with districts on the timely delivery of orders to facilities from the district and should involve MSD at the central level to ensure that initi orders are filled in a timely manner. The purpose of this recommendation is to improve the start-up effectiveness of the ILS rollout, to reduce facility anxiety about placing initial orders, and to develop good order completion habits from the beginning. IMPROVE NGO PARTICIPATION The sample for the survey was to include one NGO facility for each of the 11 districts in the pilot region; yet because many were thought not to be included in the ILS, only three NGOs were surveyed. NGOs that hav MOH permission to purchase supplies through MSD should be included in the ILS because that approach is much more efficient than for NGOs to travel to an MSD zonal store to pick up supplies. The district pharmacist can also help monitor NGO consumption. NGO participation has no cost implications for the district because NGOs are required to pay for all supplies. The inclusi be emphasized in the recommended district-level course. Additionally, CHMTs should be encouraged to reach Tanzania: Integrated Logistics System Pilot-Test Evaluation 27 out to NGOs to ask them t participate mo o re fully in the ILS. e Form h s the ng that ing odify R am ing r o a programs, he support of other rs who support s a ty-level ordering t el icient for s he were used sure el. ble sting and monitoring e, which is Some NGOs might not need to order the full range of products in the ILS. In that case, NGOs should be permitted to us 2C: Blank R&R to order only the items they need, even where some of these items appear on the preprinted forms. The purpose of this action is to improve efficient use of MSD transport for deliveries to bot MOH and NGO facilities, a well as to improve collaboration among all facilities. INCLUDE VACCINES AND TUBERCULOSIS/ LEPROSY IN THE ILS As previously noted, EPI and TB/leprosy program had been purposely excluded from the ILS. Those programs have a lo history of support from multiple donors and many years of experience with their ordering systems. Both programs also have an extensive in-country support network of supervisors and transportation to ensure their program items are not stocked out. The operation of the vaccine ordering system need not be modified in order to include it in the ILS. The current version of the vaccine annex does modify the data collected on the order form (it adds to the data available), but it does not m the inventory control system (1.5-month maximum stock level). The additional data included on the vaccine R& should be helpful to the progr and should assist in collect wastage rates. The tick sheet fo vaccines is unmodified. Likewise, the operation of the TB/leprosy drug ordering system need not be modified in order t include it in the ILS. For the forms, the system already includes an inventory control system and all essential dat items. The form could be modified slightly to give it the ILS look and feel, but this change is not critical. The inclusion of both programs in the ILS should be accomplished through discussion by PSU with both perhaps with t MOH manage further integration. The purpose of this action is to promote the idea that the ILS i single system for ordering all drugs and related medical supplies. IMPROVE MONITORING AND EVALUATION OF THE ILS One of the key purposes in creating the ILS—in addition to improving facili with reduced paperwork—is to create a system for collecting estimated consumption data tha can be used at the central lev for improved forecasting and program management. At present, tools are insuff data management at the central level either at MSD or within PSU. The addition of a database at MSD that complements MSD’s Orion Financial System was envisioned in developing t ILS, but the development and deployment of that system delayed by the need to develop a sufficient scope of work for MSD’s database consultants, Simba Technology. That database is now ready for deployment and should be to its fullest extent by PSU, MSD, and all programs to en that they are aware of what is happening at the facility lev This action will ensure that facility-level data are availa for foreca program performanc not currently possible for most programs. 28 Tanzania: Integrated Logistics System Pilot-Test Evaluation APPENDIX U T-T UES SURVEY Q ILS PILO ESTIONNA EST EVALUATION Q IRE TIONNAIRE 000. Interview ID Questions No. Question Response Go To 001 What is the name of the interv iewer? te? _d _ d _ /___ / ____ / mm / yyyy 002 What is the da 100. F nsacility and Interviewee ID Questio No. Question Response Go To 101 What is your name? [Name of the person being int ___________ erviewed] MO/AMO…………….1 CO……………………2 Nurse Midwife……….3 PHNB……………….4 MCHA……………….5 102 What is your job title? [Choose only one.] Pharmacy Tech…….6 Pharmacist………….7 Other__________….9 Prescriber……………a 103 What is your ILS role? [Choose all approp Dispenser……………b Storekeeper…………c riate answ ers.] Facility In-Charge….d Tanzania: Integrated Logistics System Pilot-Test Evaluation 29 100. Facility and Interviewee ID Questions No. Question Response Go To District Pharmacist….e Hospital Pharm………f DMO………………….g Don’t know……….….h Other__________….z Prescribe drugs…….a Manage drug stores.b D ….c ispense drugs… Comp ledge lete stores r ….d s………….… Complete R&R 1 s….….e 04 What activities do you perform in the ILS? [Choose all appropriate an Manage overall f swers.] acility…………………f Iringa Urban…………1 Iringa Rural/Kilolo….2 Makete……………….3 Ludewa………………4 Mufindi………….….5 Njombe………….……6 Dodoma Urban….….7 Dodoma Rural….……8 M pwapwa…….………9 K ongwa……….…….10 105 What is the district name? K ondoa……….….….11 30 Tanzania: Integrated Logistics System Pilot-Test Evaluation 100. Facility and Interviewee ID Questions No. Question Response Go To Hospital………….……1 Health Center…….2 Dispensary……….….3 106 What is the facility name:__________________ and type? Other__________….9 GOT………………….1 NGO……………….….2 FBO……………….….3 107 What is the facility’s ownership? Other___________….9 A………………………1 B………………………2 C………………………3 Don’t know………….4 108 What is the facility’s delivery group? Other__________….9 Tanzania: Integrated Logistics System Pilot-Test Evaluation 31 200. Training Questions No. Question Response Go To 201 How many people from this facility were trained in the ILS? (Enter number, for example, 04, 10.) 202 this facility? How many people trained from this facility are still working at example, 04, 10.) (Enter number, for Yes………………….1 2 05 203 Were you trained in the ILS? 2 No……………………2 04 Read the manual on my own…………….1 Other trained person me………………….2 (still here) from this facility trained Other trained person (not still here) from this facility trained me………………….3 Supervisor did OJT………………….4 204 How did you primarily learn how to do the activities in the ILS? [Choose only one.] Other_______…….9 Very confident…….1 Confident……….……2 Somewhat confident……….……3 205 How confident do you feel that you can perform your tasks in the ILS? Not at all confident……….……4 32 Tanzania: Integrated Logistics System Pilot-Test Evaluation 200. Training Questions No. Question Response Go To Yes (observed)….….1 206 Do you have a copy of the ILS Procedures Manual? o………………….2 N Yes (observed)….….1 207 [It does NOT have to be the calculator received during training.] .2 Do you have a working calculator? No……………….… Yes…………….…….1 301 No…………….…….2 208 Was the training sufficient to allow you to perform your ILS tasks? Did not attend….….3 Completing stores dgers………….….ale Completing R&Rs…….b Timing of reporting…….c Mathematics training….d Calculator use training………….…e Additional general training…………….… 209 What part of the ILS training needs reinforcing? [Select all appropriate choices.] f Other_________.z Tanzania: Integrated Logistics System Pilot-Test Evaluation 33 300. Records No. Question Response Go To Yes (observed)….1 303 301 ook 4: Ledger? No………………….2 302 Do you have ILS Form 1: Stores Ledger or MTUHA B Yes…………….……1 303 302 s s? No……………….….2 304 Do you have another form for recording receipts and issue of product Yes (observed)….1 303 Is the ledger up to date for most products? No………………….2 Yes (observed)….1 306 304 Do you have Form 2: Report & Request for Priority Drugs and Related Medical Supplies? [Form 2A for health centers and dispensaries, 2B for hospitals] 305 No…………….…….2 Use blank Form 2C for all products…….1 Use blank paper or other format…….….2 CRIN….…………….3 Do not buy from MSD…………….… How do you place orders for priority drugs and m .4 edical supplies? Other________.….9 305 Yes (observed)….1 401 306 Do you have Form 2C: Blank Report & Request for Additional Drugs and Related Medical Supplies? No…………….….….2 307 Use blank paper or other format…….….1 CRIN………….…….2 Do not buy from MSD…………….….3 307 How do you place orders for additional drugs and medical supplies? Other________….9 34 Tanzania: Integrated Logistics System Pilot-Test Evaluation 400. R uantity eport Timing/Q No. Question Response Go To 0 tim es…………….…….1 1 tim e…………….………2 2 tim es…………….…….3 401 How many ILS orders have you placed since April 15, 2005, using ILS Form 2A-C? < 2 times………….…….4 Yes……………….………1 402 Have you placed any non-ILS orders since April 15, 2005? No……………….……….2 404 FP using FP program R&R………………….….a STI drugs/HIV tests using program R&R……….….b 403 What have you ordered with non-ILS order forms? [Select all appropriate choices.] Other__________….….z Yes, with Form 2C…….1 406 Yes 2C… 405 , but not with Form …………….……….2 404 Did you order any additional products? No… 3 406 ………….…………. Did not know this could be don e…………….….…a Do n form ot have the ……………….…….b Nee ava MSD…… ded a product not ilable from ……….……….c 405 Why did you NOT use Form 2C for the additional products? [Select all appropriate choices.] Oth er__________…….z Yes…………………….….1 408 406 day of the month of the end of your group’s quarter? [i.e., by July 10 for Group A, August 10 for Group B, and September 10 for Group C] No…………………….….2 Did you submit your most recent R&R by the 10th No transport………….….1 Report not completed on time .…2 ………………….… Trai available to complete the repo ned staff member not rt………………….….3 Not a priority………….….4 407 Why was your most recent R&R not presented to the district by the 10th day of the month? [Select only one primary choice.] Other___________…….9 Tanzania: Integrated Logistics System Pilot-Test Evaluation 35 400. Report Timing/Quantity No. Question Response Go To Storekee o the per took it t district………………….1 Other facility staff member took it to the district………….2 District representative came to pick it up…………………….….3 413 By post…………….….….4 413 By other nonfacility person going to the district HQ…………………….….5 413 408 How did your most recent R&R reach the district? Other__________…….9 Public transport…….….1 Private transport…….….2 Facility vehicle…….…….3 409 How did the person travel? Foot or bicycle……….….4 Yes……………….….…….1 412 410 When the most recent R&R reached the district, did the person who took it stay for it to be reviewed? No………………….…….2 District pharmacist not available…………….….…1 No time…………….….….2 No allowances……….….3 411 Other__________…….…9 Why did the person NOT stay for a review? 413 Reviewed some of the mathematics……….…….a Reviewed the financial issues……………….…….b Discussed rational ordering………………….c 412 What did the person do during the review? [Select all appropriate choices.] Other__________…….….z 36 Tanzania: Integrated Logistics System Pilot-Test Evaluation 400. Report Timing/Quantity No. Question Response Go To Yes………………….…….1 415 413 Did you use the annex form VAC2/VAC3 to order vaccines? 4 No………………….….2 14 Did not receive the form……………….……….1 Was told not to by DCCO/RCCO/facility in- charge……………….…….2 Prefer the old form….……3 414 Why did you NOT use VAC2/VAC3 to order vaccines? [Select one primary choice.] Other_________……….9 416 Monthly………………….1 Quarterly………………….2 As needed……………….3 415 How frequently do you use VAC2/VAC3 to order? Other………………………9 Yes……………………….1 416 Does this facility offer HIV testing? No………………………….2 501 Yes……………….…….1 419 417 Did you use the annex form HIV2 to order HIV tests? No………………………….2 Did not receive the form……………………….1 501 Was told not to by DACC/RACC/facility that was in charge……….….2 Prefer the old form……….3 418 s? [Select one primary choice.] 9 Why did you NOT use HIV2 to order HIV test Other_________………… Monthly……………………1 Quarterly……………….2 As needed……………….3 419 How frequently do you use HIV2 to order? Other _____…….…/……9 Tanzania: Integrated Logistics System Pilot-Test Evaluation 37 [Ask to see the most recently submitted R&R.] 500. Report Completeness No. Question Response Go To Yes…………….……1 503 No……………….….2 501 Were you able to complete Form 2: R&R without difficulty? .……3 516 I am not the person who filled it in…………. Which part of the form was difficult to complete and why? [Ask about each item on the form. Record “why” information on back of page.] Top section …………….……a Column A: Beginning Balance [should come from previous report] …………….……b Column B: Received This Period [should come from orm 1]F …………….……c Column C: Lost/Adjusted [should come from Form 1, if any] ……………….…d Column D: Ending Balance [should come from both Form 1 and the results of a physical inventory] …………….……e Column E: Estimated Consumption [mathematical formula on form] ……….…………f Column F: Quantity Needed [mathematical formula on form] …………….……g Column G: Quantity Requested [rounding to nearest MSD unit of issue] …………….……h Column I: Cost [mathematical formula on form] …………….……i Total cost this page [addition] ………….………j Cost Summary [copying from previous pages, addition] ………….………k 502 Other …………….……z Yes………………….……1 505 503 Did you order to the maximum (i.e., using the formula for a 7-month maximum) for all products? No………………….…….2 504 Did not have sufficient funds……………….…….1 Was told not to do so by supervisor…….2 504 Why did you NOT order to the maximum of 7 months? Other________….…….9 38 Tanzania: Integrated Logistics System Pilot-Test Evaluation 500. Report Completeness No. Question Response Go To Did you order to the maximum u products for which there ge (i.e., 0 cost for contraceptives)? Yes…… ……1 sing the formula for is no char ………….… 505 No……………….……….2 Why not?__________ Did not need to order the product…………….…….a Product not managed at this facility………….….…b Did not consider it to be a priority product……….…c Did not think there were sufficient funds…….….d Did not think MSD would have the product in stock………………….….e 506 If there are any blank rows, why are they blank? [Select No information was available [e.g., pre-ILS data not kept]….……….….f all appropriate choices.] Other________…….….z Have been entirely stocked out for a long time……………….…….a New product for this facility to order…….b Did not think this important…………….….c information was 507 If the reas [Select all appropriate choices.] Other________…….….z re are any rows with all 0 in cols. A–E, what is the on for this? Own experience…….….1 Ordered same as kit quantity……………….…2 508 If there are any blank columns (i.e., the rows are incomplete for cols. A–E), how was the quantity requested determined? Other_________…….…3 Yes………………………1 5 10 509 Were you aware of the funding limits of the order when it was placed? No………………….…….2 515 Tanzania: Integrated Logistics System Pilot-Test Evaluation 39 500. Report Completeness No. Question Response Go To District informed us before forms………………….…1 we completed District informed us after we submitted orders……………….….2 510 the limits were? How did you know what Other__________….….9 Yes……………….……1 511 Did your initial calculations exceed the allowed amount? 515 No………………….….2 Completed the order anyway, hoping to ask the district for supplemental funds.….1 513 Cut back the order u was below the limit…….2 ntil it 514 512 n exceeded the amount? What did you do when the calculatio Other___________….9 Yes, in full…………….1 515 Yes, partially……….….2 515 513 Did you receive the supplemental funding you requested? No……………….….……3 514 Cut back on nonpriority products first……….….a Cut back on individual products where ro made less sense ( needed was 537, so changed to rounding unding up i.e, down)……………….….b 514 ow did you cut back the order until it was below the limit? [Choose all that apply.] Best judgment……….…c H Other____________….z On own…………….….a With MCH………….….b With STI…………….….c With CO……………….d 515 Did you complete the R&R on your own or working with Other___________.….z others? [Choose all that apply.] 40 Tanzania: Integrated Logistics System Pilot-Test Evaluation 500. Report Completeness No. Question Response Go To ILS………………….…….1 519 516 Which system do you prefer, the ILS, or the kit/vertical programs? 517 Kit/vertical programs…….2 Responsibility shared among several staff members……….….a Less overall cost to b facility……………………… Kits are easier…………….c Facility does not have worry about finances.d to 517 What do you think are the advantages of the vertical [Select all appropriate choices.] .z systems? Other___________.……. Too many people involved in decision making……….a The same product is in many programs……….b Too much paperwork.……c Higher costs than integrated program……….d Many different orders received at different times……………….………e Inefficient use of storage space……………….………f More stockouts…….…….g 518 think are the disadvantages of the vertical systems? [Select all appropriate choices.] ….z What do you Other__________….… 601 Facility controls quantity ordered……………………a Facility controls how funds b are spent…….…. One formula for all systems……………………c Clearer documentation of procedures……………….d Helps me decide how much to order……….…….e Eliminates separate orders for FP, STI, malaria, etc…f 519 What do you think are the advantages of the ILS? [Select all appropriate choices.] Other__________….z Tanzania: Integrated Logistics System Pilot-Test Evaluation 41 500. Report Completeness No. Question Response Go To Too much work for facility staff……………….…….a Time table too rigid……b No buffer stock kept district level…………….c at More costly than ver programs……………….d tical Too much paperwork.…e More stockouts…………f 520 What do you think are the disadvantages of the ILS? [Select all appropriate choices.] Other__________….z 42 Tanzania: Integrated Logistics System Pilot-Test Evaluation 600. Transport/Receipt No. Question Response Go To 601 How many ILS orders have you received from the district/MSD since April 1, 2005? (Enter number, for example, 04, 10.) Yes (observed)………….1 602 Do you have the “sales invoice” for the most recent order? No…………….………2 Yes………………….1 603 Did you receive the orders sealed in cartons? No…………………….2 Lower level picked it up………………….….1 Higher level delivered it………………………2 604 How did the order reach the facility? Other_______….…9 Yes………………….1 701 605 Was a member of the VHC or a witness present when the cartons were opened? No……………….….2 Did not think it was necessary………….…1 Member unavailable……….….2 606 Why was there NOT a member of the VHC or a witness present? Other_____.…….….9 Tanzania: Integrated Logistics System Pilot-Test Evaluation 43 44 Tanzania: Integrated Logistics System Pilot-Test Evaluation 700. On-sit Results e ILS Supervision/Training No. Question Response Go To 0 times………00 801 701 ceived e of following up on drug ordering and reporting issues, i.e., the ILS? [This is not a general supervision visit, but for ILS only.] xample, 04, 10.) How many on-site ILS supervisory visits have you re since April 1, 2005, for the purpos (Enter number, for e 702 Who conducted the most recent supervision visit? Title:___________ Physical count of stock……………….…a Form 1/Leja verified………….…….b Expired stock removed………….….c R&R reviewed/collected.….d OJT/coaching for ILS………………….…e 703 What was done during the most recent supervision visit you received? [Select all appropriate choices.] Other_______….….z Yes……………….……1 801 No………………….….2 705 704 Did you receive a certificate at the end of the training? tend…….…3 801 I did not at Yes………………….1 706 705 Did you receive a certificate after the training from the DMO? No………………….2 801 Nothing, the DMO just gave it to me……….a I received OJT from the district pharmacist……….….b I worked with another staff member from this facility …………….….c 706 What activities did you do to receive the certificate from the DMO? [Select all ap choices.] Other________.d propriate Facility Name Interviewer Name 800. Stock S Forms Revie he Six-Mont eriod April 1, 2005–Sep 0, 2005 tatus and w for t h P tember 3 801a. Stock Status for Sample Priority Products Product Unit of Measure S/out Today? (Y/N) Ledger Available ? Ledger up to Date? S/out Past 6 Months? (Y/N) Total Est. Consum. Number of Months of Data Available SOH SOH Today by Form 1 Today by Physical Count Tot Total Total al Expired Number Duration S/outs S/outs Microgyno n Cycle Lo-Femena l Cycle Microval Cycle Male condom Piece Depo-Provera Vial IUD Piece Cipr 500 oflox mg acin Tablet Benzathi peni mu ne 2.4 cillin Vial Ceft 250 p riaxo mg ne dr Vial Podophylline 10% in H2O 60ml bottle Tanzania: Integrated Logistics System Pilot-Test Evaluation 45 801a. Stock Status for Sample Priority Products Product Unit of Measure S/out Today? (Y/N) Ledger Available ? Ledger up to Date? S/out Past 6 Months? (Y/N) Total Est. Consum. Number of Months of Data Available SOH SOH Today by Form 1 Today Total Total Total by Physical Count Expired Number Duration S/outs S/outs Doxycycline 100 mg Tablet Metronidazole 20 Ta 0 mg blet Cotrimoxazole 400 mg/80 mg Tablet SP 500 mg/50 mg Tablet het ORS Sac Measles vaccine doses Vial [count !] OPV [count s Vial dose !] Nonsterile es, size M Each glov 5-ml disposable syringe Each Scalp vein set Each OPD cards Each Field stain A 25-gm bottle 46 Tanzania: Integrated Logistics System Pilot-Test Evaluation 801b. Stock Status for Sample Prior cts ity Produ Product Unit of Measure S/out Today? (Y/N) Ledger Available ? Ledger up to Date? S/out Past 6 Months? (Y/N) Number SOH Total Est. Consum. of SOH Total Total Duration S/outs Months of Data Available Today by Form 1 Today by Total Number Physical Expired S/outs Count 1 2 3 4 5 6 7 8 9 10 11 12 13 Capillus Test Determine Test Vironostika Test 801c. Stock Status for Sample Prio ty Produ ri cts Product Unit of Measure S/out Today? (Y/N) Ledger Available ? Ledger up to Date? S/out Past 6 Months? (Y/N) Number SOH Total Est. Consum. of SOH Total Total Months of Data Available Today by Form 1 Today by Total Physical Count Expired Number Duration S/outs S/outs 1 2 3 4 5 6 7 8 9 10 11 12 13 Chlorproma- zine 25 mg Tablet Hyoscine-N- Butyl 10 mg bromide Tablet Sodium lactate compound (Hartmann’s) 500-ml bottle Film x-ray 30x24cm Piece Incomplete Anti-D ottle 10-ml b Tanzania: Integrated Logistics System Pilot-Test Evaluation 47 T RANSFER THE DATA FROM THIS TABLE TO TABLE 801 Source o ion f Informat out S e tart Dat S/out EProduct S/ nd Date Duration of Stockout [6-5] Stor er es Ledg Info Re or S/oason f ut rmant Knowledge 1 2 3 4 5 6 7 48 Tanzania: Integrated Logistics System Pilot-Test Evaluation Source of Information Product S/out Start Date S/out End Date Duration of Stockout [6-5] Stores Ledger Reason for S/out Informant Knowledge 1 2 3 4 5 6 7 Reason for stockout: 1 = Higher-level facility did not send enough products 5 = Did not request products at the correct time 2 = Higher-level facility did not send products in time 6 = Insufficient resources (financial, human, or transportation, specify) 3 = Increase in consumption 7 = Othe asons and state the reason in column 10 4 = Did not request the correct amount r re Tanzania: Integrated Logistics System Pilot-Test Evaluation 49 803a. R&R Error Ch ource is the most recently completed R&R] eck [S Product Blank Row (Y/N) “0” Row [cols A–E] (Y/N) [cols A–E] Col. E Math Correct? Col. F Math Col. G Math Col. I Math Correct? Correct? Correct? 1 2 3 4 5 6 7 Microgynon Lo-Femenal Microval Male condom Depo-Provera IUD Ciprofloxacin Benzathine penicillin Ceftriaxone Podophylline 10% in H2O Doxycycline Metronidazole 50 Tanzania: Integrated Logistics System Pilot-Test Evaluation 803a. R&R Error Check [Source is the most recently completed R&R] Product Blank Row “0” Row Col. E Math Col. F Math Col. G Math Col. I Math Correct? (Y/N) (Y/N) [cols A–E] [cols A–E] Correct? Correct? Correct? 1 2 3 4 5 6 7 Cotrimoxazole SP ORS Measles vaccine OPV Nonsterile gloves, M size 5-ml disposable syringe Scalp vein set OPD cards Field stain A Tanzania: Integrated Logistics System Pilot-Test Evaluation 51 803 Error Ch most re te b. R&R eck [Source is the cently comple d R&R.] Only for facilities offering HIV tests. Product Blank Row (Y/N) [cols A–E] . “0” Row (Y/N) [cols A–E] . Col. E Math Cor ct? re Col. F Math Cor ct? re Col. G Math Col. I Math Correct? Cor ct? re 1 2 3 4 5 6 7 Capillus Determine Vironostika 803c. R&R Error Check [Source is the most recently completed R&R.] For hospitals, add the following products. Chlorpromazine Hyoscine-N- Butylbromide Sodium lactate compou (Hartman nd n’s) y 30x24cm Film x-ra Incomplete anti-D GED, PUT A LINE THROUGH THE ROW. IF NOT MANA 52 Tanzania: Integrated Logistics System Pilot-Test Evaluation 804. R&R Review No. Question Response Go To Ye ….1 s……………… 804. OP PART OF T RM COMPLETECORRECTLY? [E.G., MONTH A H] No… ….………2 IS THE T HE FO D BEGINNING ND ENDING MONT ……… Yes……………….….1 805. IS THE DATE S BMITTED BEFORE THE 10TH DAY OF THE H OF THE QUARTER FOR THAT GROUP? No………………….…2 U MONT Tanzania: Integrated Logistics System Pilot-Test Evaluation 53 900. Storage Conditions table 1–13 should be assessed are ready t or dist ents. A table should be filled out for each storage (including refrigerator) housi e or more of the categories of produ se spe ify the types of products being assessed in the storage area (including refrigerator) by circling the category (categories) of product ual inspection of the storage facility, noting any relevant observations in the comments colum eet . Items stored in this area: (circle all appropriate) vaccines contraceptives STI lab essential drugs HIV tests Yes No N/A Comments Items area for all facil ng on ities for products that o be issued cts below. Plea ributed to cli c s below. Place a checkmark in the appropriate column on the basis of vis n. To qualify as “yes,” all products and cartons must m the criteria for each item No. Description 1. s and expiry dates manufacturing dates are visible. Products are arranged so that identification label and/or 2. Products are stored and organized in a manner accessible for first-expiry, first-out (FEFO) counting and general management. 3. Cartons and products are in good condition, not crushed as a result of mishandling. If cartons are open, check that products are not wet or cracked as a result of heat or radiation (fluorescent lights in the case of latex products, e.g. gloves and condoms). 4. The facility makes it a practice to separate damaged and/or expired products from good products and remove them from inventory. 5. Products are protected from direct sunlight at all times of the day and during all seasons. 6. Cartons and products are protected from water and humidity during all seasons. 7. Storage area is visually free from harmful insects and rodents. (Check the storage area for traces of rodents [droppings] or insects.) 8. Storage area is secured with a lock and key but accessible during normal working hours, with access limited to authorized personnel. 9. Products are stored at the appropriate temperature during all seasons according to product temperature specifications. 54 Tanzania: Integrated Logistics System Pilot-Test Evaluation 10. is not accessible to non All hazardous waste (e.g., needles, toxic materials) is properly disposed of and medical personnel. 11. Roof is maintained in good condition to avoid sunlight and water penetration at all times. 12. Storeroom is maintained in good condition (e.g., it is clean, all trash is removed, shelves are sturdy, boxes are organized). 13. reasonable expansion (i.e., receipt of expected pro eliveries for the reseeable future). The current space and organization is sufficient for existing products and duct d fo dditional standards below can be y facility large enough to require stackin of multiple boxes. o. Yes o A COMMENTS The a applied to an g N Description N N/ 14. Products are stacked at least 10 cm off the floor. 15. st 30 cm away from the walls and other stacks. Products are stacked at lea 16. Products are stacked no more than 2.5 meters high. 17. Fire-safety equipment is available and accessible (any item identified as beused to prom ing ote fire safety should be considered). 18. Products are stored separately from insecticides and chemicals. Tanzania: Integrated Logistics System Pilot-Test Evaluation 55 Tanzania: Integrated Logistics System Pilot-Test Evaluation 900. (cont) Storage Conditions No. Question Response Go To 902. CCINES? WHAT IS THE TEMPERATURE OF THE REFRIGERATOR STORING VA [If no thermometer, mark “99”.] 9 “99”.] 03. WHAT IS THE TEMPERATURE OF THE REFRIGERATOR STORING HIV TESTS? [If no thermometer, mark Yes, up to date………….….….1 Yes, not up to date……………….… .2 904. IS THERE AN UPDATED TEMPERATURE CHART FOR THE REFRIGERATOR STORING THE V No……………….……3 ACCINES? 56 Facility Name Interviewer Name 1000. District Level Only No. Question Response Go To District pharmacist……….….1 Hospital pharmacist [not district pharmacist]……….….2 Other pharmacy ….…………3 DNO………………….……….4 DMO………………….……….5 1001 Title of person being interviewed Other_________……….…….9 District pharmacist…….…….1 Hospital pharmacist…………….…….2 DMO………………….……….3 Other_________……….…….9 1002 role o n be d ILS f perso ing interviewe Yes (observed)…………………1 1003 W the dis s divided into s A/B/C using W et 2 No……………………….…….2 ere orkshe trict ? group Yes (observed for all)….…….1 Yes (observed for some)….2 1004 Was the information on each facility collected us s No……………………….…….3 ing Work heet 1? 1005 offe ar t t, regar f being in or ou S How many total facilities ring health services dless oe ther t of th e in e IL he dis ? tric 1006 H to facilities are i the ILS in the GOT Non-GOT ow many tal n district? Tanzania: Integrated Logistics System Pilot-Test Evaluation 57 1000. District Level Only No. Question Response Go To Hospitals Health Centers Dispensaries They are new and not in the GOT list during this year………………1 They offer only limited services (e.g., only FP)…………….….2 Did not think they should be included …………………….3 1007 If the number is 1005 > 1006, why are some facilities not included? Other__________……….….9 Most orders received on or before 10th day………………….…….1 Some orders received late, but within the correct month (i.e., 1010 10th–31st days)………….21008 How timely were the orders that you were supposed to receive? not within the correct 1009 Some orders received late, and month………………….………3 Submitted the orders with the p….1 next grou Told the facility they had to wait for next order………….2 1009 What did you do for facilities whose reports were not received during the correct month? Other___________….….….….9 I did not review most 1014 orders…………………….….….1 < 10 mins……………………….2 10–30 mins…………….….……3 1010 How much time did you take to review the average order? 30–60 mins…………….….……4 > 60 mins……………….….….5 0%……………………………….1 1–25%……………….….………2 26–50%……………….3 51–75%……………….……4 1011 6–100%…………….…………5 What percentage of your reports did you review with a member of the facility staff present? 7 58 Tanzania: Integrated Logistics System Pilot-Test Evaluation 1000. District Level Only No. Question Response Go To Very reasonable….………….…….1 Reasonable…….……….…….2 Somewhat reasonable……………….….…3 1012 ount of time you spHow reasonable is the am end reviewing forms? Not at all reasonable….……………….….4 Fewer than 10 corrections per …1 form……………………….… 10–20 corrections per form……………………….……2 1013 ns of any type I made to [Corrections are for mathematical errors and are not the same as changes made as a result of budget constraints.] The number of correctio the average order is: > 20 corrections per 3 form………………………….… Yes………………………….….1 1016 Did you have timely access to supplemental funds for orders that exceeded their allocation? No…………………………….…2 1014 Nothing. I left the orders as is.…………………………….…1 I reduced the quantities, without consulting the facility…….…….2 I reduced the quantities, when reviewing with the facility….….3 1015 when it was requested, what did you do to change the need for supplemental funds? If you did not give supplemental funding Other __________……….….…9 DMO……………………….…….a CHF…………………….….…….b NHIF……………………….….….c Donor__________……….…….d 1016 What was the source of the supplemental funds? [Select all appropriate choices.] Other__________……….…….e Did not have the form…….….…1 1019 I did not complete the form for any facilities (0%)……………….2 1019 1017 For what percentage of facilities in the district did you need to complete Form 3: Supplemental 1–10%………………….3 Funding? [In other words, what percentage of facilities requested supplemental funding?] Tanzania: Integrated Logistics System Pilot-Test Evaluation 59 1000. District Level Only No. Question Response Go To 11–25%…………………….……4 26–50%………………….…….5 > 50%………………….……….6 I was able requested it everything they to give all those who needed………………….….….1 I divided the amount available equally among the facilities…………………….….2 I divided the amount by population size…………….….3 I used my best judgment………….….…………4 1018 If you gave supplemental funding, how did you decide how much to give to each facility? Other__________……….……9 Yes…………………….…………1 102 1 1019 Did you complete Form 4: Order Compilation for each delivery group (A, B, C)? No……………………….……….2 No facility needed more funds……………………….……1 Did not have the form….….……2 Did not think this was necessary……………….………3 1020 Why did you NOT complete Form 4 for each delivery group? Other__________……….…….9 District paid in cash……….……1 District paid by check…….….2 District paid from its own MSD account……………………….….3 Supplemental funds were not used……………………….…….4 1021 How did the supplemental funds needed reach MSD? Other___________……….……9 They can manage on their own………………………….…1 They can manage with some assistance……………………….2 1022 How do you feel about the ability of the average facility to correctly complete the top part of the R&R? They cannot manage without assistance……………………….3 60 Tanzania: Integrated Logistics System Pilot-Test Evaluation 1000. District Level Only No. Question Response Go To They can manage on their own……………………….………1 They can manage with some assistance……………………….2 1023 How do you feel about the ability of the average facility to correctly complete columns A–E of the R&R? They cannot manage without assistance……………………….3 They can manage on their own……………………….………1 They can manage with some assistance……………………….2 1024 How do you feel about the ability of the average facility to correctly complete columns F and G of the R&R? They cannot manage without assistance……………………….3 Completing Form Ledger………………………….a 1: Stores Completing Form 2: R&R………………………………b Basic mathematics………….….c Storage practices…………….d 1025 For what aspects of the ILS does training need to be reinforced? [Select all appropriate choices.] Other___________…………….z Yes, I have visited most or all of them…………………………….1 Yes, I have visited some of them…………………………….2 1026 Have you made a supervisory visit to the facilities concerning the ILS in the last 90 days? them…………………………….3 1028 No, I have not visited Physical count of stock….….a Form 1/Leja………….…b Expired stock removed….….c R&R reviewed/collected….….d OJT/coaching for ILS………….e 1027 as done during the supervision visit you conducted? [Select all appropriate choices.] What w Other____………………….….z 1028 role as a District Supervisor, do you think the ILS is less work, about the same work, or more .1 In your Less work ………………….… Tanzania: Integrated Logistics System Pilot-Test Evaluation 61 1000. District Level Only No. Question Response Go To About the same amount of work ………………….………2 work for you as the previous kit/vertical systems? More work ……………….………3 ILS…………………………….….1 1032 1029 Which system do you prefer, the ILS, or the kit/vertical programs? 030 Kit/vertical programs……………2 1 Responsibility shared among several staff members……….a Less overall cost to facility……………………….…….b Kits are easier…………….c Facility does not have about finances…….d to worry 1030 What do you think are the advantages of the vertical systems? [Select all appropriate choices.] Other___________…………….z Too many people involved in decision making…………….….a The same product is in many programs…………………………b Too much paperwork……….….c Higher costs than integrated program………………….……….d Many different orders r different times……………….….e eceived at Inefficient use of storage space………………………….f More stockouts…………….…….g 1031 [Select all appropriate choices.] What do you think are the disadvantages of the vertical systems? Other___________………….….z Facility controls quantity ordered………………………….a Facility controls how funds are spent…………………………….b 1032 tages of the ILS? elect all appropriate choices.] One formula for all systems…………………….c What do you think are the advan [S Clearer documentation of procedures……………………….d 62 Tanzania: Integrated Logistics System Pilot-Test Evaluation 1000. District Level Only No. Question Response Go To Helps me decide how much to order……………………….………e Eliminates separate orders for FP, STI, malaria, etc………….….f Other____________……….……z Too much work for facility staff…………………………….….a Time table too rigid…………….b No buffer stock kept at level……………………………….c district More costly than vertical programs…………………………d 1033 What do you think are the disadvantages of th ILS? Too much paperwork………….e More stockouts………………….f e [Select all appropriate choices.] Other__________……………….z Tanzania: Integrated Logistics System Pilot-Test Evaluation 63 1034. Review of Form 4 [for most recent Form 4] ed to report? Number of facilities expect hat reported on time? Number of facilities t that reported late? Number of facilities Numb rting in the wrong group? er of facilities repo 64 Tanzania: Integrated Logistics System Pilot-Test Evaluation For more information, please visit http://www.deliver.jsi.com. DELIVER John Snow, Inc. 1616 North Ft. Myer Drive, 11th Floor Arlington, VA 22209 USA Tel: 703-528-7474 Fax: 703-528-7480 www.deliver.jsi.com IMPETUS FOR INTEGRATION RELIANCE ON ORGANIZATIONS FOR SUCCESS MEDICAL STORES DEPARTMENT PHARMACEUTICALS AND SUPPLIES UNIT OTHER MINISTRY OF HEALTH PROGRAMS DEVELOPMENT OF THE INTEGRATED LOGISTICS SYSTEM FORMATION OF A LOGISTICS TASK FORCE DESIGN OF THE INTEGRATED LOGISTICS SYSTEM/DESIGN WORKSHOP DRAFT OF THE PROCEDURES MANUAL PRIORITIZATION OF DRUGS PILOT-REGION SELECTION USE OF ZONAL TRAINING CENTERS AND TRAINING OF TRAINERS CREATION OF CURRICULUM TRANSLATION AND PRINTING OF FORMS AND MATERIALS TRAINING PILOT TESTING METHODOLOGY SURVEY DESIGN SITE SELECTION SURVEY TIMING SURVEY TEAMS SURVEY TEAM TRAINING FINDINGS SITES PERFORMANCE AND PERCEPTIONS ABOUT THE ILS RECORDS AND REPORTS AVAILABILITY AND COMPLETENESS OF STORES LEDGERS REPORT AND REQUEST FORM COMPLETENESS REPORTS SUBMITTED AND THEIR TIMELINESS REPORTS REVIEWED AT THE DISTRICT LEVEL STOCK STATUS INVENTORY CONTROL PROCEDURES STOCKOUT RATES STOCKOUTS IN THE PAST SIX MONTHS MONTHS OF STOCK STOCK STATUS AT MEDICAL STORES DEPARTMENT SUPERVISION INCREASE THE EVALUATION PERIOD FOR THE NEXT ROLLOUT REGIONS, AND PROVIDE ADDITIONAL SUPERVISION REVIEW THE LIST OF PRIORITY ITEMS INCREASE AVAILABILITY OF PRIORITY ITEMS AT MSD EXTEND THE LENGTH OF THE TRAINING TO FIVE DAYS IMPROVE REVIEW OF REPORTS AND ON-SITE SUPERVISION DEVELOP A SYSTEM FOR NONREPORTING OR UNABLE-TO-PERFORM FACILITIES REDUCE COMPLICATIONS IN START-UP IMPROVE NGO PARTICIPATION INCLUDE VACCINES AND TUBERCULOSIS/ LEPROSY IN THE ILS IMPROVE MONITORING AND EVALUATION OF THE ILS IF NOT MANAGED, PUT A LINE THROUGH THE ROW. IS THE TOP PART OF THE FORM COMPLETED CORRECTLY? [E.G., BEGINNING MONTH AND ENDING MONTH] IS THE DATE SUBMITTED BEFORE THE 10TH DAY OF THE MONTH OF THE QUARTER FOR THAT GROUP? 900. Storage Conditions table Description N/A COMMENTS WHAT IS THE TEMPERATURE OF THE REFRIGERATOR STORING VACCINES? WHAT IS THE TEMPERATURE OF THE REFRIGERATOR STORING HIV TESTS? IS THERE AN UPDATED TEMPERATURE CHART FOR THE REFRIGERATOR STORING THE VACCINES? Y Y

View the publication