WHO Indicators for monitoring national drug policies

Publication date: 1999

Indicators for monitoring national drug policies A practical manual Second edition Pascale Brudon(1) Jean-Daniel Rainhorn(2) Michael R. Reich(3) (1) World Health Organization, Geneva (2) Centre de Recherche et d’Etudes pour le Développement de la Santé, CREDES, Paris (3) Harvard School of Public Health, Harvard University, Boston ACKNOWLEDGEMENTS The manual has been reviewed and commented on at each step of its development by many experts within and outside WHO, including: o For the selection of the indicators: H. Bak Pedersen (EURO/WHO, Copenhagen), J. Baudouy (World Bank, Washington, DC, USA), H. Bangoura (MOH, Guinea), P. Bécu (DRASS, France), K. Besseghir (IUED, Geneva, Switzerland), O. Brasseur (GPA/WHO, Geneva), M. Caraël (GPA/WHO, Geneva), M. Claeson (CDR/WHO, Geneva), V. Diwan (IHCAR, Sweden), M. Everard (EMRO/WHO, Alexandria), G.H. Fernando (University of Colombo, Sri Lanka), N. Gasman (Glas, Nicaragua), P. Hessou (MOH, Benin), C. Hodgkin (HAI, Netherlands), T.B. Ilunga (BAD, Abidjan), D. Jayasuriya (DMP/WHO), K. Kafle (Department of Clinical Pharmacology, IOM, Nepal), J. Kasonde (HRP/WHO, Geneva), Q.L. Kintanar (BFAD, Philippines), Kyaw Win (Directorate of Medical Services, Myanmar), R. Laing (INRUD, Boston, USA), C. Landon (ICO/WHO, Geneva), C. Meillon (Ministère de la Coopération et du Développement, Paris, France), J. Ngaba (MSP, Central African Republic), D. Ofori-Adjei (University of Ghana Medical School, Ghana), E. Ombaka (CMC, Geneva), G. Peters (Institut de Pharmacologie, Lausanne, Switzerland), J. Primrose (DOH, Australia), J. Rankin (MSH, Arlington, USA), S. Sapirie (MEP/WHO, Geneva), P.M. Shah (CHD/WHO, Geneva), A. Sheak (DDA/Ministry of Health, Nepal), K. Shein (SEARO/WHO, New Delhi), N. Shigeno (WPRO/WHO, Manila), P.D. Spivey (School of Pharmacy, Aberdeen, UK), M. Stegeman (Ministry of Foreign Affairs, Netherlands), G. Tomson (IHCAR, Sweden), C. Torel (EPI/WHO, Geneva), R. Vaurs (World Bank, Washington, DC, USA). o For field testing of the indicators in terms of clarity, ease of collection, validity and usefulness: H. Bangoura (MSPAS, Guinea), S. Diakite (MSPAS, Guinea), K. Douno (MSPAS, Guinea), E. Feret (CREDES International, France), K.K. Kafle (Tribhuvan University, Nepal), P.R. Khonje (MOH, Malawi) D. Kilayo (MSP, Central African Republic), Q.L. Kintanar (BFAD, Philippines), P.A. Manuel (DOH, Philippines), J. Ngaba (MSP, Central African Republic), D. Pigot (CREDES International, France), A. Sheak (DDA/MOH, Nepal), A. Toumi (MSP, Tunisia). o Methodology for indicator calculation has been designed with the support of S. Adjei (MOH, Ghana), J. Dumoulin (IREPD, France), A. Fontanet (GPA/WHO, Geneva), M. Garenne (Harvard School of Public Health, USA), S.K. Lwanga (HST/WHO, Geneva), A.N. Malek (DAP/WHO, Geneva), A. Spira (INSERM, France), G. Tomson (IHCAR, Sweden). Methodology for calculating the value of a basket of drugs has been designed with the support of J.M. Fleury (Geode, France). o Comments at each step of development have been provided by members of the staff of the WHO Department of Essential Drugs and Medicines Policy, including: F.S. Antezana (former Director), D. Fresle, V. Habiyambere, M. Helling-Borda (former Director), J. Hetzke, H.V. Hogerzeil, S. Holand, T. Sodogandji, G. Velasquez, K. Weerasuriya. o The new version, updated by P. Brudon (EDM/WHO), has benefited from the experiences and comments of researchers and policy-makers who used the indicators in their own country: E. Carandang (Philippines), N.W. Chitalu, R. Kampamba and M. Msoni (Zambia), M. Fofana (Guinea), E. González Sedano (Colombia), M. Laloge and O. Sidibe (Mali), L. Ngarmadjingaye (Chad), Nguyen Thanh Do (Viet Nam), T. Dimitrova and G. Petrova (Bulgaria), G.N.V. Ramana (India), S. Ratanawijitrasin (Thailand), M. Ndlovu and B. Trap (Zimbabwe) and K. Weerasuriya (Sri Lanka). K. Timmermans (EDM/WHO) also contributed greatly to this revised version. o Secretarial assistance was provided by M. Renevier. ã World Health Organization 1999 This document is not a formal publication of the World Health Organization (WHO), and all rights are reserved by the Organization. The document may, however, be freely reviewed, abstracted, reproduced and translated, in part or in whole, but not for sale nor for use in conjunction with commercial purposes. The views expressed in documents by named authors are solely the responsibility of those authors. able of contents CHAPTER I: Introduction Why indicators of national drug policies are needed. 1 How to use the manual. 2 Who should use the manual. 5 How to use the indicators . 6 How to apply the indicators . 7 How the results can be used . 8 CHAPTER II: Development of the manual Methodology. 11 CHAPTER III: Model lists of indicators Background information. 17 Structural indicators. 19 Process indicators . 23 Outcome indicators . 26 CHAPTER IV: Methodology for indicator calculation Organizing the data collection. 29 Collecting data . 33 Analysis and reporting . 37 Conducting surveys . 44 Calculating the value of a basket of drugs . 58 CHAPTER V: Detailed presentation of indicators Structural indicators. 63 Process indicators . 115 Outcome indicators . 155 ANNEX 1: Data collection forms . 167 ANNEX 2: Glossary . 221 ANNEX 3: Table of random numbers . 225 T hapter I Introduction Why indicators of national drug policies are needed How to use the manual Who should use the manual How to use the indicators How to apply the indicators How the results can be used C Chapter I: Introduction 1 WHY INDICATORS OF NATIONAL DRUG POLICIES ARE NEEDED All countries have a national pharmaceutical policy. Such a policy may exist in the form of a policy document, in which case it is explicit. Alternatively, it may be indicated in certain government procedures and actions, in which case it is implicit. Often these policies reflect the many changes in the pharmaceutical sector which most countries, particularly developing and transitional ones, have experienced in recent years. Overall, the main objectives of these national drug policies are: F to make effective, safe, low-cost drugs available and affordable to meet the needs of the entire population (essential drugs); and F to ensure that drugs are of good quality and used rationally. However, no effective and accepted tools exist at the global or country level to evaluate the performance of pharmaceutical sectors, to monitor progress in the implementation of national drug policies or to assess the effects of changes on drug policy objectives. This manual has been developed to meet this need. It contains a comprehensive set of simple, objective and reliable indicators which can be adapted to fit national contexts. It will allow countries and international agencies to do the following: F assess a country's capacity to implement the various elements of a national drug policy; F monitor the processes by which a national drug policy is implemented and the changes over a period of time; F measure the policy's progress towards the achievement of objectives, allowing decision-makers to adjust strategies accordingly. An effective national drug policy requires: clear and mutually consistent objectives; appropriate strategies (policy measures) adapted to national resources; and the necessary technical means to achieve objectives. Although strategies for drug policy may differ from country to country — depending on the specific political, economic and social situations and the health status — the overall policy objectives and the key strategies are basically the same for all nations. A logical method was therefore sought to identify the major problems faced by countries in achieving the overall objectives of drug policy and selecting key strategies of universal importance for achieving objectives on the basis of which to develop indicators. The method is described in detail in Chapter II. The manual focuses on relatively low-cost and non-complex methods that can be used with limited specialist input and integrated into routine monitoring systems. It derives from the Indicators for monitoring national drug policies 2 belief that in many countries, much of the necessary data exist but are often not organized to address specific policy concerns. This manual therefore suggests ways to organize, analyse and present data for such purposes. It is hoped that these indicators will be used widely for monitoring drug policies and will facilitate the systematic monitoring of national drug policy implementation. In 1996, the indicators were applied in 12 countries using a systematic research-based approach, and the experience acquired in these countries has been incorporated in this new version.1 The manual’s indicators and methods will continue to be reviewed and improved. New experience gained will be incorporated by the Department of Essential Drugs and Medicines Policy in future editions. HOW TO USE THE MANUAL The manual is divided into five chapters, which are designed to be used in different ways. Chapter II provides an explanation of the procedures used to develop the indicators. The core method for indicator development was a Delphi survey among a panel of 54 international experts, which obtained consensus on seven priorities for improving the pharmaceutical situation in the public and the private sectors in developing countries.2 These were: 1. The establishment of appropriate legislation and regulation. 2. The selection of essential drugs and the registration process. 3. The importance of maintaining a significant drug allocation in the health budget and developing relevant financing policy in the public sector. 4. The improvement of procurement procedures in the public sector. 5. The strengthening of drug distribution and logistics in the public sector. 6. The establishment of a drug pricing policy in both public and private sectors. 7. The role of information and continuing education programmes to improve drug use. The indicators described in the manual have therefore been selected to evaluate the pharmaceutical situation in countries and to measure progress in the seven priority areas outlined above and in the four overall objectives of any national drug policy. 1 Comparative analysis of national drug policies: second workshop (WHO/DAP/97.6). 2 Rainhorn JD, Brudon-Jakobowicz P, Reich MR. Priorities for pharmaceutical policies in developing countries: results of a Delphi survey. Bulletin of the World Health Organization, 1994, 72(2):257-264. Chapter I: Introduction 3 Chapter III presents the model lists of the indicators, with a brief description of the use of each list. The indicators are divided into four categories: Background information This first set of indicators is intended to provide data on the demographic, economic, health and pharmaceutical contexts in which drug policy is being implemented in a given country. The information is quantitative data, at a single point in time, which in most countries are readily available at the central level. The manual provides 31 background information indicators. Structural indicators These indicators provide qualitative information to assess the pharmaceutical system's capacity to achieve its policy objectives. They are intended to check whether the key structures/systems/mechanisms necessary to implement a pharmaceutical policy exist in the country, for each of the seven key strategies noted above. These structural indicators are answered "yes" or "no" on the basis of information usually available at the central level. The manual provides 50 structural indicators. Process indicators These indicators provide quantitative information on the processes by which a national drug policy is implemented. They assess the degree to which activities necessary to attain the objectives are carried out and their progress over time. As with the structural indicators, the process indicators monitor the main activities under the same seven key strategies of drug policy. These process indicators are based on information available at the central level and/or obtained through surveys. The manual provides 38 process indicators. Outcome indicators These indicators measure the results achieved and the changes that can be attributed to the implementation of the national drug policy. They have been selected to assess the effects of implementing the policy on the overall NDP objectives: availability and affordability of essential drugs, drug quality and the rational use of drugs. These outcome indicators are based on information available at the central level and/or obtained through surveys. The manual provides 10 outcome indicators. Although a national drug policy is ultimately intended to improve the overall health of a population, health impact indicators are not included in this manual for three main reasons: the multifactorial nature of health status, the consequent complex issues of causality associated with drug policy and health status, and the unresolved methodological difficulties of selecting reliable health indicators directly related to the use of drugs. The manual therefore recommends the use of indicators that measure the effectiveness and efficiency of Indicators for monitoring national drug policies 4 key components of a national drug policy. It is assumed that these will have an impact on the availability and accessibility of essential drugs and on the quality and use of drugs — to the ultimate benefit of the health status of the population. In some circumstances, however, a manager may decide to use a few health status indicators if the data are relatively simple to collect and can be reliably related to drugs. However, such data should be interpreted with great care. Chapter IV describes the various steps to be followed when applying indicators — from the selection of appropriate indicators, the collection of data, the training of staff and the calculation of the indicators, to the presentation of results. It includes a detailed discussion of the procedures for conducting surveys and for calculating the value of a basket of drugs. It also provides a number of suggestions on how to establish a monitoring system. Chapter V presents each indicator on a single page; this chapter is intended to serve as the core reference section of the manual. Each indicator (except for the background indicators) is described with a brief statement that includes: Definition What is the content of the indicator? Use What will this indicator measure? Why is this indicator important? Description What are the definitions of key terms? What is the scope of the indicator? How can the results be interpreted? Sources and methods of data collection and indicator calculation:3 What are the main sources and methods of data collection? How should the indicator be calculated? Limitations What are the main limitations of the indicator? 3 The method of calculating the indicator is provided only for process and outcome indicators as structural indicators require a Yes/No response. Chapter I: Introduction 5 These detailed guidelines should be read carefully before applying the indicators. The information they contain can be adapted at country level, according to national targets and priorities. They can also be used for training data collectors. When new indicators are developed, similar pages should be prepared. Annex 1 provides model data collection forms to be used for entering the information gathered either at central level or during the surveys. These forms can be directly copied or modified according to the country-specific context. Annexes 2 and 3 of the manual provide respectively a glossary and a table of random numbers. WHO SHOULD USE THE MANUAL The manual is designed for use primarily by senior management personnel responsible for NDP implementation. These managers need to obtain information regularly to improve the policy and to guide the allocation of resources. They need to verify whether activities within the main policy areas are carried out as planned, and to detect problems (process monitoring through structural and process indicators). They also need to document whether the progress under the various policy areas is leading to expected results in terms of availability, affordability, quality and use of drugs (outcome monitoring through outcome indicators). This is in order to design new strategies/policy measures to improve the performance of the NDP’s main components. These managers can use the manual to establish or improve a drug monitoring system, or to integrate drug monitoring into existing health information systems. National policy-makers in the ministry of health (and in other ministries) concerned with the overall improvement of health conditions may also find this manual useful in helping them to monitor progress in drug policy. They may select certain indicators for regular reports on the pharmaceutical situation. For example, they may decide to request an annual report on the outcome indicators in order to assess the overall progress of drug policy implementation and policy strategies. National policy-makers concerned with the evaluation of "health sector reforms" may find some indicators described in the manual useful for assessing the effects of the reforms. For instance, indicators on use of drugs (OT7 to OT10) are important indicators of technical efficiency, which is often a main objective of health sector reform. Finally, policy-makers may also find the manual useful in preparing reports on the performance of the pharmaceutical sector, for use with external donors or for presentation to other government agencies (such as the ministry of finance). Managers at intermediate and lower levels will probably find the indicators useful but may need more (or more detailed) information in relation to their field of intervention. More detailed information can of course be collected for any of the issues covered Indicators for monitoring national drug policies 6 by the indicators, but when monitoring an NDP with a wide range of strategies/components, it is important to focus on the main trends. Although collecting more information may be important for micromanagement, it can lead to confusion when monitoring a policy as a whole. International agencies' experts will find the manual useful when developing new projects in the pharmaceutical field and when assessing current projects. It can be a useful tool when organizing the situation analysis for a project, before developing strategies and activities. The background information and structural indicators will provide basic data on the health/pharmaceutical sector and an assessment of the country's capacity to implement the various aspects of a national drug policy. This standardized information can also help reduce duplication of efforts when multiple agencies are working in one country. HOW TO USE THE INDICATORS Indicators contained in the manual can be used as they are; every effort has been made during their development to ensure that they are relevant to most situations and countries and that they provide a comprehensive view of a national drug policy. Their use in 12 countries in 1995 and 1996 confirmed that they are applicable to various socioeconomic contexts and to different pharmaceutical policies. However, it is expected that managers will sometimes need to adapt the indicators to match national circumstances. The indicators can be adapted in several ways. First, countries may have additional objectives, beyond those specified above, such as the development of national pharmaceutical production or the integration of traditional medicine. In this case, supplementary indicators could be developed at country level. This development should be preceded by careful identification of the objectives and strategies, in order to select indicators that will provide useful information about the effectiveness of policy implementation. Second, national managers of drug policy may wish to adapt the normative standards suggested in the indicators to take into account the country's specific priorities. Each country, according to its political structure and level of socioeconomic development, will be at a different stage of formulation and implementation of its national drug policy, and will have its own priorities. This manual proposes provisional standards for process and outcome indicators. The team responsible for applying the indicators should review these standards carefully, to be certain that they correspond to the national context and the drug policy's targets and goals. Third, national managers may wish to have more detailed information on certain aspects of the drug policy; in this case, they can subdivide the indicators. For instance, the indicators that refer to prescribers in general can be made more specific by adapting them to the various groups of prescribers. However, it is important not to forget that the main objective Chapter I: Introduction 7 is to evaluate and monitor progress in implementing the policy and not to follow all the activities in detail. Fourth, because of the differences between countries, some indicators will be of greater relevance than others. Some sections of the manual will thus be best applied selectively to specific problems. Alternatively, national managers may decide that some indicators in this manual are not appropriate for their country or policy. Finally, experience suggests that developed countries may also be able to use this approach for defining indicators.4 In such cases, the list of indicators may be used as a model and adapted to fit the national context. HOW TO APPLY THE INDICATORS This manual is intended for application not only in countries that have adequate monitoring infrastructure for drug policy, but also in those that currently lack this capacity. Where an organizational capacity for monitoring already exists, the manual can be used as a reference tool to review the existing systems for data collection, indicator analysis and report presentation. It may provide ideas for new indicators, sampling methods or report formats, for strengthening the capacity to monitor drug policy implementation. In countries that lack this organizational capacity, the manual is intended to provide the basis for creating an institutionalized system for monitoring the implementation of national drug policy. The goal of such a system is to provide the essential information without the need for ad hoc studies. However, some data will need to be collected through specific surveys, at least during the first years, because existing systems will be inadequate. For these situations, the manual proposes methods that seek to balance the need for collecting accurate data with the need for logistical simplicity. In certain countries, a practical approach could be to begin with what is possible now and gradually increase the monitoring activities as the system develops. The organizational unit in charge of monitoring the implementation of drug policy should be within the ministry of health, either in the pharmaceutical department or the planning department. Even in highly decentralized health systems, some central capacity for monitoring the implementation of national policy is desirable. In general, once the organizational basis for a monitoring system is in place, the process of monitoring requires the following steps: 4 The Government of Australia has adapted the approach contained in this manual to develop its own indicators: Development of a manual of indicators to measure the effect of initiatives under the quality use of medicine arm of the national medicinal drug policy, Department of Human Services and Health, Australia, September 1994. Indicators for monitoring national drug policies 8 F defining the targets within each policy area and objective (this step must be carried out as part of policy formulation and planning); F identifying the indicators and data needed; F collecting the data; F analysing and interpreting the data; F using the results to propose changes, if needed. The technical procedures for collecting and analysing the data are described in detail in Chapter IV and Annex 1. The issue of interpretation of changes in indicators is complex. Knowledge of the specific context in which the indicator has changed is crucial to any such interpretations. Examples of such interpretations are provided in Chapter V and in the report cited in footnote 1. Finally, such an institution-building process will require the commitment of senior policy- makers as well as the support of external organizations. These will help ensure that an effective monitoring system can be created and sustained, and that it is used to improve both the equity and the efficiency of national drug policy. Ultimately, the system should be integrated into the national health information system. HOW THE RESULTS CAN BE USED Indicators and monitoring systems are worthwhile only if they are used. Too often, data are collected but never analysed; or data are analysed but never used to improve or modify existing practices or policy. This manual is intended to produce results that can be used to improve the effectiveness of drug policy in the following ways: First, the indicators derived from the manual can be used to monitor the progress in implementing the various components of a national drug policy. Are the basic structures in place, and are they functioning adequately? Which components are performing well, and which not? If the indicators can be collected regularly over time, then it becomes possible to determine whether particular components have improved or declined in performance. Second, the indicators can be used to assess the priorities for implementation in the drug sector, and also to assess the effectiveness of overall drug policy strategies. If one component of drug policy is performing poorly compared with other components, then it may be desirable to allocate more resources (both human and financial), and thereby revise the relative priorities of different components, in an effort to improve implementation. In some cases, it may be necessary to change the strategy, for example by introducing a programme for the promotion of essential drugs, or by developing new pricing policies in order to encourage private distribution of essential drugs. Chapter I: Introduction 9 Third, the indicators can be used by both national and international agencies to compare drug policy performance across different countries. A comparison of structural indicators would assist in identifying relative weaknesses and strengths in institutional capacity to implement drug policy, and a comparison of process indicators would help to show the relative progress in achieving drug policy targets. Cross-national comparisons can also assist national policy-makers in learning about innovative approaches that may be applicable in their own countries. In addition, the collection of country data by the Department of Essential Drugs and Medicines Policy will facilitate development of an international database on the implementation of national drug policies. This could be disseminated worldwide and could assist national policy-makers in comparing the performance of their country's pharmaceutical sector with that of other countries. Finally, the indicators can be used in negotiations on drug policy among the various interested parties within a country, and also in policy discussions with external donors and international agencies concerning health sector reform. The application of these indicators over time, for example, could help demonstrate the impacts of macro-economic policy changes (such as devaluation) on the health system and on the pharmaceutical system. The indicators can provide data to enable health policy-makers argue more persuasively and coherently, helping, for instance, to ensure that the health sector and the health status of vulnerable groups are not forgotten during times of economic reform. Indicators for monitoring national drug policies 10 hapter II Development of the manual C Chapter II: Development of the manual 11 METHODOLOGY The development and selection of indicators followed 10 logical steps (see box), which are described more fully below. Selection of a conceptual framework for indicator development. Literature review to identify potential key issues and strategies/components of pharmaceutical policy in developing countries. Delphi survey to develop consensus on key issues and strategies/ components of pharmaceutical policy in developing countries. Experts' consultation to review general difficulties in indicator development and to define criteria for selection of indicators. Sets of indicators for monitoring implementation of drug policy in developing countries proposed by the working group (background, structural and process indicators). Field testing of proposed indicators in six countries to assess the clarity, applicability and usefulness of the indicators selected. Review of the first draft manual by experts within and outside WHO to assess the methodology used for indicator development and the categories of indicators. Set of outcome indicators to measure progress towards the overall objectives proposed by the working group. Review of methodology for indicator calculation by epidemiologists and statisticians within and outside WHO to assess the relevance of the proposed methodologies and the appropriateness of the sampling procedures. Finalization of the manual based on a review of all comments received and incorporation of appropriate revisions. Definition of a conceptual framework: Indicators for monitoring national drug policies 12 A working group including people with extensive field experience, academics and a WHO/EDM staff member was set up at the Harvard School of Public Health to support WHO/EDM in developing indicators for NDPs. The first task of the group was to define a logical approach to indicator development which would serve as the conceptual framework for the subsequent activities. The various steps of this logical approach are outlined below (see box). Conceptual framework for indicator development · What are the key issues in the pharmaceutical sector? (diagnosis of problems) · What are the main objectives of NDPs? · What are the main strategies/components which should be developed to achieve those objectives? · What are the main activities which should be implemented under each strategy/component? · What type of indicators should be developed to monitor these activities? · What type of indicators should be developed to assess the impact of the strategies and the activities? Literature review: A literature review to identify the main issues currently faced by developing countries in the pharmaceutical field was carried out by the working group at the Harvard School of Public Health. The review included both published and unpublished documents related to the pharmaceutical sector in more than 50 countries. On the basis of this review, the working group compiled a comprehensive listing of major problems faced by developing countries in the pharmaceutical sector, which were called "key issues". The working group then identified for each key issue those elements of the pharmaceutical system that have a major impact on performance; these were called "key components". Chapter II: Development of the manual 13 Delphi survey: The next step was to achieve general agreement on the ranking of both the key issues and the key components in terms of importance for intervention, as a way to establish priorities. To achieve this general agreement, a Delphi survey was carried out.5 The study was designed by a monitor group set up at the Harvard School of Public Health. The Delphi group consisted of 54 people with substantial expertise in pharmaceutical policy in developing countries. It included people from different types of institutions: multilateral donors, such as the World Bank and the European Union; the UN system, such as WHO and UNICEF; nongovernmental organizations; research and consulting groups; pharmaceutical companies; universities; and individual consultants specializing in drug policy implementation. Half were pharmacists or physicians, and half were economists, managers, policy analysts, anthropologists or statisticians. The group included people from 12 countries on four continents. Through the Delphi technique, a high rate of agreement was obtained on key issues and key components. Seven key components/strategies were mentioned as priorities for action by a large majority of the Delphi respondents (see box). Key components The establishment of appropriate drug legislation and regulation. The selection of essential drugs and the registration process. The importance of maintaining a significant drug allocation in the health budget and developing a relevant financing policy in the public sector. The improvement of drug procurement procedures in the public sector. The strengthening of drug distribution and logistics in the public sector. The establishment of a drug pricing policy in both public and private sectors. The role of information and continuing education programmes to improve 5 The Delphi technique is a method for structuring group communication so that the process allows a group of individuals, as a whole, to deal with a complex problem and reach group consensus. The process involves the use of a series of questionnaires designed by a monitor group and then sent by mail in several rounds to a respondent group of experts who remain anonymous. After each round, the results are summarized by the monitor team and used to develop a questionnaire for the next round. The summary and new questionnaire are then sent to all members who responded. A Delphi survey is considered complete when a convergence of opinion occurs or when a point of diminishing returns is reached. Indicators for monitoring national drug policies 14 drug use. The Delphi technique established these seven key components as particularly important for achieving the objectives of a national drug policy. They were therefore adopted as the basis for selecting indicators to monitor the process of implementation of pharmaceutical policy. Experts' consultation: A major issue in indicator development is the importance of defining criteria. During an informal consultation in Geneva, a set of guiding principles and criteria was discussed and prepared to provide a common approach for indicator development in the field of pharmaceuticals.6 It was agreed that indicators should be developed according to the following principles: F Usefulness for action: The data provided in the indicator should primarily help strengthen national drug policy and programme management, and should secondarily help to promote goals and targets set up at the international level. The indicator should be useful for decision-making and action at the level where the data are collected, which can increase the reliability of data collected. F Clarity: The indicator should express a single idea that is generally agreed to be important. F Ease of generation and measurement: The data should, as far as possible, result from the regular data collection system. If the indicator requires an additional survey, this should be within the capability and responsibility of staff at the level it is performed. F Consistency and validity: The indicator should be proven capable of being recorded throughout the system with an acceptable degree of validity and reliability. F National relevance: The indicator should serve to measure progress towards the goals, objectives and targets stated in national policy. F Ease of comparison: The indicator should, when feasible, provide quantitative data that can be compared with specific norms and objectives. Sets of indicators proposed The working group at the Harvard School of Public Health next proposed structural and process indicators to measure the most important activities in each key component and also background information indicators for the implementation context. The initial list was reviewed with the above six criteria for indicator development, resulting in the elimination of a 6 Development of indicators for monitoring national drug policies, Department of Essential Drugs and Medicines Policy (WHO/DAP/92.6). Chapter II: Development of the manual 15 number of proposed indicators. The work resulted in three provisional lists of indicators: background information, structural and process indicators. Field testing Six countries were selected for field testing the three lists of provisional indicators, with the following objectives: F to validate the selection of indicators in various situations; F to assess for each indicator the clarity, ease of collection, validity and usefulness for action; F to identify other indicators that should be added, and existing indicators that were unnecessary and should be removed; F to assess the usefulness of an indicator-based monitoring system for the implementation of national drug policy. The six countries selected for field testing (Central African Republic, Guinea, Malawi, Nepal, Philippines and Tunisia) provided a range of national contexts and drug policies. The results of the field tests showed that policy-makers and country managers were interested in having available effective and accepted tools for carefully monitoring the implementation of their national drug policy and were eager to implement such an indicator-based system in their own country. Indicators selected were considered to be appropriate. They were easily understood, simple to apply and relatively easy to collect, although some needed special surveys. These results confirmed the relevance of the project and of the approach taken. A preliminary analysis of the field tests took place at a two-day meeting to discuss the relevance of the indicators to each country. The meeting allowed participants to review the full lists of indicators to determine whether any should be removed or revised and to discuss whether new indicators should be added. Additional analysis of the field tests was also done by the working group, which reviewed all comments and incorporated many suggestions from the field tests. Review of the first draft manual by experts within and outside WHO: The draft manual was sent to 60 reviewers within and outside WHO for comments on: F the methodology used for indicator development; F the usefulness of the categories of indicators proposed; F the lists of indicators selected; F the usefulness of the manual at country level. Suggestions for outcome indicators were also requested. More than 30 reviewers sent comments, which were classified into broad categories and systematically analysed by the Indicators for monitoring national drug policies 16 working group along with the results of the field test. There was general agreement on the usefulness of the manual, on the relevance of the methodology used and on the lists of indicators. Most of the comments dealt with specific indicators and with the need for a methodology to collect and calculate the indicators. A few reviewers proposed new indicators, which were added provided that they fitted the six criteria previously defined. Some reviewers commented that the indicators could also be used in developed countries. Set of outcome indicators proposed The working group identified a small number of outcome indicators that could measure the impact of various components on the overall objectives of national drug policy: availability and affordability of essential drugs, good quality of drugs and rational use of drugs. This included the impact of policy on the private as well as the public sector, since national policies affect both. Some indicators previously developed by WHO to assess drug use were incorporated into this list. Review of methodology for indicator calculation In order to assist countries in implementing the indicators, the working group prepared guidelines for the collection and analysis of the data through record reviews, interviews and surveys. These guidelines include a detailed discussion of the procedures for conducting surveys and data collection forms for central and field levels. The guidelines were revised by the working group to ensure the validity of the methodologies and the appropriateness of the sampling procedures, after careful reviews by experts in epidemiology and statistics. Finalization of the first and second editions of the manual The first version of the manual was prepared after further review inside and outside WHO. The indicators were then used in more than 12 countries in 1996, and the current version of the manual has been slightly modified to take into account the experience gained in these countries (footnote 1). In addition, WHO/EDM is preparing a manual on issues related to monitoring systems in the drug field which will complement this one and integrate practical experiences of countries. hapter III Model lists of indicators Background information Structural indicators Process indicators Outcome indicators C Chapter III: Model lists of indicators 17 BACKGROUND INFORMATION These 31 indicators provide background information on the demographic, economic, health and pharmaceutical contexts in which drug policy is implemented in a given country. They are quantitative data, usually available at the central level. The information can be used to help identify major problems in health status, in the health system and in the drug sector. The indicators provide basic data that a policy-maker responsible for the drug sector should know and have available. They should generally be updated annually. This background information can be used by national policy-makers for discussing drug policy within the broader national health context and by international experts for assessing the country situation. Some of the indicators may be useful in cross-national comparisons of drug policy implementation. Some are also used in calculating subsequent indicators. Field tests have shown that the background information can be collected in a few days, if adequate access exists to key personnel in the health sector. Most of these indicators are provided in the annual reports of major international organizations (e.g. the International Monetary Fund, the United Nations Development Programme, the World Bank). However, in a few countries, some information, mainly on financial aspects, may be difficult to collect and not very reliable. It is then important to review each indicator, and to determine its usefulness in the national context and the level of "precision" which is needed. A model data summary form (Summary Form 1) is provided in Annex 1 (when filling it in, do not forget to indicate the year for which the information has been collected and the source). Country information Population data BG1: Total population BG2: Average annual growth of the population BG3: Percentage of the total population living in urban areas BG4: Life expectancy (years) Economic data BG5: GNP per capita BG6: Average annual rate of inflation Indicators for monitoring national drug policies 18 Health information Health status data BG7: Infant mortality rate (per 1,000 live births) BG8: Maternal mortality rate (per 100,000 live births) BG9: Top five causes and rate of infant morbidity BG10: Top five causes and rate of infant mortality BG11: Top five causes and rate of adult morbidity BG12: Top five causes and rate of adult mortality Health system data BG13: Total number of prescribers BG14: Total public health budget7 BG15: Total value of international aid for the health sector BG16: Total health expenditure (public + households + international aid) Drug sector information Economic data8 BG17: Total public drug expenditure7 BG18: Total value of international aid for drugs (cash + kind) BG19: Total drug expenditure (public + households + international aid)9 BG20: Total value of local production (ex-factory price) sold in the country BG21: Total value of drug imports (CIF) BG22: Total value of drugs under generic name (CIF price for imported drugs and ex- factory price for locally produced drugs) sold in the country Human resources BG23: Total number of pharmacists BG24: Total number of pharmacy technicians or other aides/assistants 7 Public finance is understood as general government revenues and compulsory health insurance (sometimes known as social insurance) that is either publicly managed or heavily regulated by governments. Private finance includes out-of-pocket payments and voluntary health insurance. 8 From BG17 to BG22: when data do not distinguish between drugs and other supplies, try to estimate the percentage that corresponds to drugs and record this estimate. 9 If there is a sizeable illegal market, some realistic estimations of the share of this market in drug expenditures should be given. Chapter III: Model lists of indicators 19 Drug sector organization BG25: Total number of drug manufacturing units in the country BG26: Total number of wholesalers in the country BG27: Total number of pharmacies and drug outlets in the public sector (including health facilities and hospitals that dispense drugs) BG28: Total number of pharmacies and drug outlets in the private sector BG29: Total number of private pharmacies and drug outlets in the three major urban areas Number of drugs BG30: Total number of registered drugs (in dosage forms and strengths)10 BG31: Total number of drugs on the national essential drugs list (in INN) STRUCTURAL INDICATORS These 50 indicators provide qualitative information on the basic structures that are considered necessary for implementing a national drug policy. The indicators check whether the basic structures/systems/mechanisms under each key component exist in the country. They do not evaluate the functioning of these structures (for example, these indicators check whether quality control facilities exist, but not whether they work efficiently). The performance of the structures is assessed with the process indicators. The structural indicators monitor the main aspects of the seven key components of drug policy: F legislation and regulation; F essential drug selection and drug registration; F drug allocation in the health budget and public sector financing policy; F public sector procurement procedures; F public sector distribution and logistics; F pricing policy; F information and continuing education on drug use. The structural indicators are answered "Yes" or "No" on the basis of information usually available at the central level. When answered "Yes", they do not provide any more a measure for progress, except to check that things do not deteriorate. Their main significance 10 In some countries the number of registered drugs is considerably higher than the number of drugs currently on the market. In such cases, countries are advised to add an indicator: "Number of drugs currently on the market". Indicators for monitoring national drug policies 20 lies in the negative answers. Many "No" answers suggest that improvements are required in institutional capacity if the drug sector is to make significant progress towards achieving overall policy objectives. For certain indicators, and according to specific country needs, the monitoring unit can decide to collect some written statements to understand the situation better or to quantify these indicators. The structural indicators may be analysed and interpreted one by one; however, structural and process indicators have been designed in such a way that it is more important to analyse all those related to a specific key component together, as shown in the table below. This means that structural indicators are related to one another within each component and should be analysed together, e.g. the results of ST1 to ST11 should be reviewed together to obtain a complete picture of the situation in terms of legislation and regulation. In addition, the structural indicators should be analysed with the process indicators for each component in order to understand what is working and what is not. Key component ST PR Legislation and regulation. ST 1 - ST 11 PR 1 - PR 7 Essential drug selection and drug registration. ST 12 - ST 18 PR 8 - PR 13 Drug allocation in the health budget/public sector financing policy. ST 19 - ST 23 PR 14 - PR 18 Public sector procurement procedures. ST 24 - ST 30 PR 19 - PR 26 Public sector distribution and logistics. ST 31 - ST 36 PR 27 - PR 29 Pricing policy. ST 37 - ST 41 PR 30 - PR 32 Information and continuing education on drug use. ST 42 - ST 50 PR 33 - PR 38 The structural indicators can be used for assisting national and international decision-makers in formulating strategies and designing interventions to improve the pharmaceutical sector. They can be used in comparing the implementation of pharmaceutical policies in different countries. Structural indicators can also be used for advocacy purposes to increase government and donor support to the drug sector. Field experience shows that structural indicators can be collected in a few days if adequate access exists to key persons in the pharmaceutical sector (public and private sectors). A one-page description of each structural indicator is provided in Chapter V and a model data summary form is provided in Annex 1 (Summary Form 2). Legislation and regulation ST1: Is there an official national drug policy document updated in the past 10 years? ST2: Is there drug legislation updated in the past 10 years? ST3: Have regulations based on the drug legislation been issued? Chapter III: Model lists of indicators 21 ST4: Is there a drug regulatory authority whose mandate includes registration and inspection? ST5: Is there a licensing system to regulate the sale of drugs (wholesalers, pharmacists, retailers)? ST6: Are pharmacists legally entitled to substitute generic drugs for brand name products? ST7: Are there legal provisions for penal sanctions? ST8: Is there a checklist for carrying out inspections in different types of pharmaceutical establishments? ST9: Are there any institutions within or outside the country where quality control is carried out? ST10: Is the WHO Certification Scheme on the Quality of Pharmaceutical Products Moving in International Commerce used systematically? ST11: Are there controls on drug promotion based on regulations and consistent with the WHO Ethical Criteria for Medicinal Drug Promotion? Essential drug selection and drug registration ST12: Is there a national essential drugs list (EDL)/formulary using INN officially adopted and distributed countrywide? ST13: Is there an official drug committee whose duties include updating the national essential drugs list (EDL)? ST14: Has the national essential drugs list (EDL)/formulary been updated and distributed countrywide in the past five years? ST15: Do drug donations comply with the national essential drugs list (EDL)? ST16: Are there formal procedures for registering drugs? ST17: Is there a drug registration committee? ST18: Is drug registration renewal required at least every five years? Drug allocation in the health budget/public sector financing policy ST19: Is the public drug budget spent per year more than 20% of the ministry of health operating budget spent per year for the last three years? ST20: Is the public drug budget spent per capita per year more than US$ 1.00 per year for the last three years? ST21: Is the public drug budget spent for national hospitals less than 40% of the total public drug budget spent for the last three years? ST22: Has the public drug budget spent per capita increased in the last three years? ST23: Are there any financing systems in addition to the public drug budget that contribute to the provision of drugs in the public sector? Public sector procurement procedures Indicators for monitoring national drug policies 22 ST24: Are drugs usually procured in the public sector through competitive tender? ST25: Is there a system for monitoring supplier performance? ST26: Is most of the tendering done under international nonproprietary name (INN)? ST27: Does the procurement unit receive foreign currency in less than 60 days (from request to release)? ST28: Is procurement in the public sector limited to drugs on the national essential drugs list (EDL)? ST29: Is the average lead time (from order to receipt at central level) less than eight months? ST30: Is procurement based on a reliable quantification of drug needs? Public sector distribution and logistics ST31: Are good storage practices observed in the central procurement/distribution unit and/or major regional warehouses? ST32: Is the information recorded on the stockcards for a basket of drugs11 the same as the quantity of stock in store? ST33: Are the stocks for a basket of drugs within their expiry dates in the central procurement/distribution unit and/or major regional warehouses? ST34: Have all incoming products been physically inspected for the last three deliveries in the central procurement/distribution unit and/or in major regional warehouses? ST35: Are only drugs which are on the national essential drugs list (EDL) in stock in the central procurement/distribution unit and/or in major regional warehouses? ST36: Are 80% or more of the vehicles of the central procurement/distribution unit and/or major regional warehouses in working condition? Pricing policy ST37: Are drug prices regulated in the private sector? ST38: Is there at least one major incentive for the private sector to sell essential drugs at low cost? ST39: Is the total margin used by wholesalers and retailers less than 35% of the CIF price? ST40: Is there a system for monitoring drug prices? ST41: Are essential drugs under INN or generic name sold in private drug outlets? Information and continuing education on drug use ST42: Is there a national publication (formulary/bulletin/manual, etc.), revised within the past five years, providing objective information on drug use? ST43: Is there a national therapeutic guide with standardized treatments? ST44: Is the concept of essential drugs part of the curricula in the basic training of health personnel? 11 A basket of drugs is provided as an example in Chapter IV (see page 59). Chapter III: Model lists of indicators 23 ST45: Is there an official continuing education system on rational use of drugs for prescribers and dispensers? ST46: Is there a drug information unit/centre? ST47: Does the drug information unit/centre (or another independent body) provide regular information on drugs to prescribers and dispensers? ST48: Are there therapeutic committees in the major hospitals? ST49: Are there public education campaigns on drug use? ST50: Is drug education included in the primary/secondary school curricula? PROCESS INDICATORS These 38 indicators provide quantitative information on the mechanisms and activities by which a national drug policy is implemented. The indicators assess the efficiency and effectiveness of these mechanisms and activities in the seven key components and the progress being achieved over time towards specific targets established at the national level. If all the mechanisms are performing well and the activities are carried out correctly, then it should be possible to make reasonable progress towards the objectives of national drug policy. The process indicators monitor the main aspects of the same seven key components of drug policy considered for the structural indicators: F legislation and regulation; F essential drug selection and drug registration; F drug allocation in the health budget/public sector financing policy; F public sector procurement procedures; F public sector distribution and logistics; F pricing policy; F information and continuing education on drug use. The process indicators are measured by a percentage (percentage of change over time, percentage of coverage, etc.), using information available at the central level and/or obtained through surveys. Percentages allow trends to be monitored over time and to be compared with an ideal value. However, in certain circumstances, the figures used for calculating the percentages can also provide useful information. The process indicators may be analysed and interpreted one by one; however, process and structural indicators have been designed in such a way that it is more important to analyse all those related to a specific key component together as shown in the table on page 20. This Indicators for monitoring national drug policies 24 means that process indicators are related to one another within each component and should be analysed together, e.g. the results of PR19 to PR26 should be reviewed as a whole for obtaining a full measure of how the drug procurement in the public sector is improving or not. In addition, the process indicators should be analysed with the structural indicators for each component. The process indicators can be used for assisting senior management staff and national decision-makers in monitoring progress in the implementation of the national drug policy, by providing quantitative measures of achievement of targets set at the national level. Each year the percentage measured should come closer to the national standard. Field experience shows that data which have to be collected at central level for calculating process indicators can be obtained within two weeks — if the data collection effort is well organized. However, for some indicators, special surveys may be necessary. These indicators are marked with an asterisk (*). Such surveys can be organized to collect data for several process and outcome indicators at the same time (see Chapter IV) and can take up to six weeks. Model sampling procedures, data collection forms for the field work and a summary form are provided in Chapter IV and Annex 1 (Summary Form 3). A one-page description of each process indicator is provided in Chapter V. Legislation and regulation PR1: Number of drug outlets inspected, out of total number of drug outlets in the country. PR2: Number of drug outlets in violation, out of total number of drug outlets inspected. PR3: Number of sanctions and administrative measures implemented, out of total number of violations identified. PR4: Number of samples routinely collected, out of total number of planned collected samples. PR5: Number of samples tested, out of total number of samples collected. PR6: Number of advertisements in violation of regulations on the ethical promotion of drugs, out of total number of advertisements monitored. PR7: Number of sanctions implemented for advertisements in violation of regulations, out of total number of violations identified. Essential drug selection and drug registration PR8: Value of drugs from the national essential drugs list (EDL) procured in the public sector, out of total value of drugs procured in the same sector. PR9: Number of drugs from the national essential drugs list (EDL) prescribed, out of total number of drugs prescribed(*). PR10: Number of drugs from the national essential drugs list (EDL) sold, out of total number of drugs sold(*). Chapter III: Model lists of indicators 25 PR11: Number of locally manufactured drugs sold in the country from the national essential drugs list (EDL), out of total number of drugs from the national essential drugs list (EDL). PR12: Number of combination drugs newly registered, out of total number of newly registered drugs. PR13: Number of registered drugs which are banned in other countries, out of total number of registered drugs. Drug allocation in the health budget/public sector financing policy PR14: Value of public drug budget spent per capita in the last year, out of average value of the same budget during the past three years. PR15: Value of public drug budget spent by major hospitals, out of value of public drug budget spent. PR16: Value of international aid received for drugs, out of value of public drug budget. PR17: Value of revenue generated for drugs through additional financing systems, out of value of public drug budget. PR18: Public drug budget spent, out of public drug budget allocated. Public sector procurement procedures PR19: Value of drugs purchased through competitive tender, out of value of drugs purchased. PR20: Value of drugs purchased from local manufacturers through competitive tender, out of value of drugs purchased through competitive tender. PR21: CIF/ex-factory value of a basket of drugs, out of CIF/ex-factory value of the same basket in the year of reference. PR22: CIF/ex-factory value of a basket of drugs, out of "reference" value on the international market of the same basket. PR23: Average lead time for a sample of orders in the last year, out of average lead time during the past three years. PR24: Average time period of payment for a sample of orders, out of average time period of payment stated in contract. PR25: Number of drugs/batches tested, out of number of drugs/batches procured. PR26: Number of drugs/batches that failed quality control testing, out of number of drugs/batches tested. Public sector distribution and logistics PR27: Average time between order and delivery from central store to remote facilities in the last year, out of average time between order and delivery in the past three years(*). Indicators for monitoring national drug policies 26 PR28: Average stockout duration for a basket of drugs in the central and/or regional stores in the last year, out of average stockout duration for the same basket in the past three years. PR29: Average stockout duration for a basket of drugs in a sample of remote facilities in the last year, out of average stockout duration for the same basket in the past three years(*). Chapter III: Model lists of indicators 27 Pricing policy PR30: Value of a basket of drugs, out of CIF/ex-factory value of the same basket(*). PR31: Average expenditure per prescription, out of average expenditure per prescription in the past three years(*). PR32: Value of a basket of drugs, out of value of the same basket in the year of reference(*). Information and continuing education on drug use PR33: Number of prescribers having direct access to a (national) drug formulary, out of total number of prescribers surveyed(*). PR34: Number of training sessions on drug use for prescribers in the last year, out of average number of training sessions organized in the past three years. PR35: Number of prescribers who have attended at least one training session in the last year, out of total number of prescribers surveyed(*). PR36: Number of issues of independent drug bulletins published in the last year, out of average number of issues of independent drug bulletins published per year in the past three years. PR37: Average number of copies of independent drug bulletins sent to prescribers, out of total number of prescribers. PR38: Amount spent on public education campaigns on drug use, out of total amount spent on public health education campaigns. OUTCOME INDICATORS These 10 indicators provide quantitative information on the achievement of the four objectives of national drug policy: availability of essential drugs, affordability of essential drugs, quality of drugs and rational use of drugs. The indicators measure the degree to which these objectives are being attained. They do not provide information on why results are good and bad; this could partly be obtained through the analysis of the structural and process indicators. It is reasonable to assume that if good results are achieved on the process indicators, then the outcome indicators should also show positive results or improvement over time. If the outcome indicators provide evidence of significant problems, when the structural and process indicators show good results, then decision-makers should undertake a careful analysis of the problems, to identify causal factors and revise strategies accordingly. These outcome indicators are measured by a percentage or a figure based on information available at the central level and/or obtained through surveys. Those indicators which may need surveys are marked with an asterisk(*). The indicators can be used for assisting national and international decision-makers in measuring the Indicators for monitoring national drug policies 28 results of policies and in evaluating drug policies and implementation strategies. They can also be used in comparing the pharmaceutical policies of different countries. Based on field experience, outcome indicators can be collected at the same time as process indicators. Model sampling procedures, data collection forms for field work, and a summary form are provided in Chapter IV and Annex 1 (Summary Form 4). A one-page description of each outcome indicator is provided in Chapter V. Availability of essential drugs OT1: Number of drugs from a basket of drugs available in a sample of remote health facilities, out of total number of drugs in the same basket(*). OT2: Number of drugs at the lowest price from a basket of drugs, out of total number of drugs in the same basket(*). Affordability of essential drugs OT3: Average retail price of standard treatment of pneumonia, out of the average retail price of a basket of food(*). OT4: Value of a basket of drugs, out of the value of the same basket with the cheapest drugs(*). Quality of drugs OT5: Number of drugs/batches that failed quality control testing, out of the total number of drugs/batches surveyed(*). OT6: Number of drugs beyond the expiry date, out of the total number of drugs surveyed(*). Rational use of drugs OT7: Average number of drugs per prescription(*). OT8: Number of prescriptions with at least one injection, out of the total number of prescriptions surveyed(*). OT9: Number of children under five with diarrhoea receiving antidiarrhoeal drugs, out of the total number of children under five with diarrhoea surveyed(*). OT10: Number of drugs from the national essential drugs list among the 50 best selling drugs (EDL), out of the 50 best selling drugs in the private sector. Chapter III: Model lists of indicators 29 hapter IV Methodology for indicator calculation Organizing the data collection Collecting data Analysis and reporting Conducting surveys Calculating the value of a basket of drugs C Chapter IV: Methodology for indicator calculation 29 The objective of this chapter is to provide information on the various steps which should be taken when applying indicators, as part of an indicator exercise (assessment of the pharmaceutical situation at a moment in time) or within a continuous monitoring system. When assessing or monitoring a national drug policy (NDP), identifying appropriate indicators and data sources, determining how to collect the data, organizing the collection and analysing the results should be carefully planned. The choice of the indicators and the quality of the information collected will affect the validity of the results and the appropriateness of the changes proposed. The main steps for ensuring that the results of the monitoring are accurate and reliable are reviewed below. ORGANIZING THE DATA COLLECTION Step 1: Choose monitoring unit members A team at central level, most probably in the ministry of health or national drug authority, should be responsible for the monitoring system, i.e. for data collection, analysis and reporting. This team should include at least three senior professionals: (i) a pharmacist or a medical doctor conversant with the national pharmaceutical policy, (ii) an economist or a manager with strong economic expertise, and (iii) a statistician. Data collectors should be recruited for the field work, when needed. This team can undertake other activities and not work full time on NDP monitoring; however, it is important that its existence be institutionalized and its functions well defined to ensure sustainability. Step 2: Decide on the final set of indicators Deciding on the final set of indicators needed for monitoring the NDP should take place at the beginning of the monitoring process. It is an extremely important step as the selection of indicators will have major implications for the results. This selection should be done once the objectives, strategies and targets of the NDP have been defined (policy formulation and planning). The team should then read the manual carefully and review all the indicators using the model lists of indicators (Chapter III) and the description of the indicators (Chapter IV). It should: F review the purpose of each indicator (what is it supposed to measure? And why is it important to measure this in the national context?); F assess whether the indicator is relevant in terms of the NDP’s objectives and strategies and whether it needs to be modified or adapted; F review whether the necessary data can be obtained, and if so where?; and Indicators for monitoring national drug policies 30 F decide whether additional indicators are needed; if so, which ones? how should the data be obtained? The team may also decide that some indicators in this manual are not appropriate for national circumstances or policy and delete them. When developing supplementary indicators to cover additional policy objectives beyond those specified in this manual, the team should identify carefully the objectives and strategies in order to select indicators that will provide useful information about the effectiveness of policy implementation. It can also decide to apply certain indicators included in the manual to other sectors, for instance the illegal market. Finally, the monitoring unit should also at this stage collect information on any other work to develop health indicators in the country — as examples of what has been done — to assist in deciding on the final list and to avoid duplication of effort. Step 3: Identify data sources The manual was designed in such a way that most of the data needed for the four categories of indicators should come from the national health information system and/or from the national drug management information system and can be collected at central level.12 The monitoring unit should always try to collect existing reliable data rather than new data. The process of collecting new data on a regular basis can have high start-up and operational costs compared with improvements in existing data collection systems. Sources of data vary from one country to another. In general, the following are useful sources: F the ministry of health is the main source of data. The chief pharmacist, the essential drugs programme manager, the head of the central medical stores, the drug regulatory authority, the head of the planning department, etc., will be able to provide information on the main aspects of the pharmaceutical sector and most data for the background information and structural indicators; F drug producers, wholesalers and pharmaceutical associations can provide data on importation, local production, distribution and supply of drugs; F health facilities and pharmacies, whether public or private, will be a good source of specific information on essential drugs’ availability, prices, quality, etc.; F other health programmes may collect data on the drug sector, on population, on morbidity and on health status which may be appropriate for use in the indicator calculations; F nongovernmental organizations, and international and bilateral organizations may also have data on specific projects dealing with drugs at the community level; 12 All background information, structural indicators and 75% of the process indicators can be collected at central level; however, 9 of the 10 outcome indicators will probably require surveys. Chapter IV: Methodology for indicator calculation 31 F other sectors such as the ministry of finance, customs authorities and the central bank, may have information on total government expenditures, total health expenditures, drug expenditures, drug imports, etc.; F international organizations such as the International Monetary Fund, the United Nations Development Programme, the World Bank, the United Nations Children’s Fund and WHO may have data on demographic and economic trends, health status and health systems, and financial data related to drugs. Detailed information on the main sources of data for each indicator is provided in Chapter V. Step 4: Determine methods for data collection Three methods are used in this manual to collect the data: F interview; F record and document review; F surveys. Identifying data which should be collected through interviews and record reviews, and data which will need a survey, is the fourth step in the process. This is specific to each country and depends on the available sources. Once the final list of indicators is established, the monitoring team should make a list of all the data that are needed, including data for the numerators and the denominators of the process and outcome indicators, and identify potential sources for the various data and ways to collect them. Most of the data can be obtained by reviewing records and documents, such as ministry of health reports, evaluations and reports of international and bilateral agencies, inspection reports, stock inventories, sales figures, prescriptions and patient logbooks. All these sources, when reliable, should be used as much as possible, as they are an efficient way of collecting information. Interviews are also an important source of information when the people to be interviewed have been selected carefully. Interviews can be conducted with varying degrees of flexibility, from structured to semi— or unstructured interviews. For collecting indicators data it is recommended that structured interviews should be used. In certain cases, a more open and informal interview style may allow a wider range of information to be explored. Aspects not adequately covered by the indicators may emerge during the course of a conversation. In this case, interviewers should continue to ask questions until they fully understand the situation. Cross-checking the findings obtained from other sources can also be done by this type of interview. Non-structured interviews may also determine whether the respondents are answering truthfully or not (e.g. in the case of the number of samples tested, the head of the laboratory may be inclined to provide figures that are higher than the actual ones). For this kind of approach, much depends on the interviewer's skills. Interviewers should have a Indicators for monitoring national drug policies 32 good understanding of the subject and be able to ask probing questions. The interviews are therefore mainly useful for clarifying and better understanding points which seem strange or contradictory in the information provided through other methods. Alternatively, a list of data to be collected at each source can be developed and data can be sent regularly to the monitoring unit from a number of recognized sources. However, in most countries, data for some indicators will not be routinely recorded and will need to be collected through specific surveys. The selection of samples, the size of the samples, and the sites where data are to be collected, should be decided at this stage as these are important for planning the process and calculating the budget. These issues are discussed in more detail below (see pages 44 to 58). Step 5: Plan the process and calculate the time needed The time needed for collecting, processing and analysing all the information at central and peripheral levels should be carefully calculated. Experience shows that four to six weeks are enough, with the exception of the first year when some preliminary tasks will be necessary. These include organization of the team, planning of the work at central level, budgeting, recruitment and training of staff, and development of data collection forms. It is also important to determine how often the chosen indicators will be measured since monitoring is essentially an ongoing process. Guiding considerations include the ease with which the information can be obtained and the resources available. In some countries, most of the information will flow from the drug management information system. When it has been set up, the process and analysis can take place every year without great difficulty. In others, data will need to be collected through specific studies and should probably take place every two to three years, until the monitoring system is in place. Step 6: Estimate resources and budget A detailed budget should be prepared, based on the total number of staff and the time necessary to collect, process and analyse the data, at central and peripheral levels. The budget should include costs of field work (personnel, transport, supplies) and of the monitoring unit (personnel, supplies, meetings, etc.). Lack of resources is usually a major limiting factor. The budget can be reduced (i) by identifying other health-related institutions or other departments within the ministry of health willing to temporarily allocate personnel to the unit, (ii) by training personnel from the survey sites to collect the data, (iii) by using students and community volunteers, where appropriate, and (iv) by planning for strict control of expenditure. It is important that the ministry of health or the authority responsible for monitoring the NDP secures a budget implementing the monitoring system, including the necessary surveys. Compared with the large drug expenditures in all countries and the potential savings which can be achieved with a good monitoring system, a small budget for such a system is a good investment. This budget allocation should be strongly advocated by drug sector personnel to decision-makers in the health sector and at government levels. Chapter IV: Methodology for indicator calculation 33 COLLECTING DATA Step 7: Develop data collection forms The data collection forms which will be used to collect and record the data must be designed, pretested and finalized before starting the data collection process. Data collection forms are necessary for collecting data both from interviews and record reviews and from surveys. The first are mainly checklists; the monitoring unit can adapt for this purpose the summary forms in Annex 1. The second are more complex to design as the same form should be used to collect data for several indicators; a set of model data collection forms has been developed to assist countries. These forms can be used as they are and therefore directly copied from this manual or adapted to national situations (see Annex 1). When adapting or designing new forms, it is important to consider the following: F the precise objective of the indicators for which data are collected: lack of understanding of the precise question asked in the indicator can lead to the collection of unnecessary data and to delays and additional work later; F the exact data which need to be collected: as a general principle, the minimum amount of data sufficient for monitoring national drug policy should be collected in the simplest possible way; F the design of the form: the text should be easy to understand and enough space should be provided on each form for data collectors to be able to note questions or comments; F facilities for analysis: computer or manual; if computers are used, data processing personnel should be asked for advice on the best way to design the data collection forms and to code the answers. Step 8: Select and train staff A key step in preparing for field work is for the monitoring unit to identify and train data collectors. In this manual, the majority of indicators are collected at central level from interviews and records, and the remaining ones at central and peripheral levels through surveys. It is therefore advisable, as long as a monitoring system is not in place, to set up two groups of data collectors: one to collect information from records and interviews, and the second to carry out the surveys. The first group should be familiar with pharmaceutical terms to be able to extract information reliably from records and to note it accurately during interviews. Two people (for example, one pharmacist and one economist) are normally sufficient, as the monitoring unit should also participate in the data collection at central level. The composition of the second group will depend on the number of sites where surveys are Indicators for monitoring national drug policies 34 to be carried out and their distance from the capital city; each team may include one supervisor (if possible one member of the monitoring unit as well as the first group of data collectors) and two other people: one familiar with surveys but not necessarily a health/pharmaceutical specialist and one familiar with drug issues and drug names. To ensure consistency in results, the monitoring unit should carefully train all data collectors. Data collectors should be thoroughly familiar with the objectives of the work and with the methodology. Therefore, it is recommended that the following be discussed with them: objectives of the work; rationale; data collection tools to be used; sampling procedures; plan for data collection, and analysis. For each indicator, they should know why the information is required and how it will be collected. They should also be taught how to fill in the data collection forms, which part they need to fill in, who will carry out the calculations, etc. Although each data collection form lists the data needed to obtain the indicators, some instruction should be given on how to handle any missing information so that this will be done consistently by different data collectors in the field. They should be requested to note anything which is unusual, as well as comments from staff in the places surveyed and reasons for modifying forms and methods during the field work. Collectors should receive clear instructions on what to do when a facility is not operational or when it is impossible to obtain 30 prescriptions/sales in one day. They should also be taught basic interview and record review techniques so that there is no misunderstanding over how to record answers, observations and data. Finally, explanations should be given concerning how the data collector should introduce himself or herself (a survey is not an inspection), how to explain the objective of the work, how to ask for consent, etc. Step 9: Conduct the pretest A pretest should always be conducted. This serves two purposes: (i) to test the relevance of the data collection forms, and (ii) to train the data collectors. The pretest also enables the monitoring unit to revise the methods and logistics of data collection before starting the actual field work. As a result, a good deal of time, effort and money can be saved in the long run. Pretesting should determine whether the data collection forms allow collection of the information needed, whether the forms are reliable, how much time is required to administer the various checklists and questions, and whether the forms need to be revised (e.g. wording unclear, coding system inadequate, space for answers insufficient, etc.). Involving the data collectors in the pretest will provide them with a practical training and will strengthen their interest in and commitment to the work. It will also provide them with some basic experience on how to organize their work before starting the real data collection. Finally, training and pretesting allow the central team to check the accuracy of the budget and of the estimated time required for collecting data at the various sites. Chapter IV: Methodology for indicator calculation 35 Step 10: Schedule data collection Although the work is usually performed by the ministry of health, it is important that all relevant authorities are adequately notified of the purpose and methods of the exercise. This is particularly true for surveys which mobilize regional administrative staff as well as local authorities. The monitoring unit should inform these authorities of the objectives of the data collectors' visit and the dates proposed, in order to promote their active cooperation. A schedule of visits should therefore be prepared with the dates of every visit, whether at central or peripheral level. Logistical arrangements (transport, accommodation, etc.) should also be made in advance, not forgetting such factors as appropriate seasons, public holidays and vacation periods, and accessibility and availability of the key people to meet. A critical factor for the success of any survey is the cooperation of the owner/person in charge and personnel of the facility surveyed. In the research discussed (see footnote 1), a few countries reported problems in obtaining cooperation at private drug outlets/pharmacies; in most countries, however, this was not the case. Countries where private sector cooperation was readily obtained used one or several of the following strategies: · data collectors had an official letter of introduction addressed to the facility; this letter also stated briefly the purpose of the survey; · an official letter announcing the survey and asking for the collaboration of the facility was sent in advance (but data collectors should also have a copy of the letter with them); · data collectors were well aware that they were not inspectors, and therefore that they had no right to impose themselves, but had to request (and depend on) cooperation; · it was clearly explained that results would be presented in a randomized way and therefore be "anonymous", so that it would be impossible to relate specific data to a particular facility. The same strategies are useful in the public sector. Indicators for monitoring national drug policies 36 Step 11: Collect data Data should be entered on the spot on the relevant data collection form. When data are incorrectly entered the data collector should be told what to do, e.g. to cross them out neatly, sign and enter the correct answer alongside. Additional information should be given if the data collectors are also requested to code the data and to fill in summary forms. This can be done only in countries where the level of knowledge and training of data collectors will be sufficient for them to code indicators reliably, e.g. people who are not familiar with drug names may be confused by different ways of referring to the same product. The possibility of assigning incorrect codes in such situations is greater than if better-trained people assign all codes at a later stage. Step 12: Ensure quality It is extremely important that the data collected are of good quality, i.e. accurate, reliable and valid. To ensure this, supervision and evaluation of the data collection should be done by the monitoring unit in addition to proper selection and training of data collectors and pretesting of data collection forms and procedures. This ongoing supervision of the data collectors — in the field for the surveys and at central level for the other methods — should aim to ensure that the agreed procedures and methods are being followed and the data collected make sense, and are complete and of good quality. It should also assist in solving problems (e.g. incomplete answers, unclear statements, omissions, etc.) and in handling and storing data collection forms before they are processed and analysed. In addition, it is recommended: · to cross-check data whenever possible; · to use data from (official) reports, if possible; · to note down not only the data themselves, but also the source and the period to which they apply; · to ascertain whether the figure given is an actual figure, a forecast or an estimate; · to use data from the same source when comparing one year with another; · to use data from a source that (most likely) will be updating its information regularly, so that next time you can use data from that source again; · in some cases, to consider using data from "international sources", such as the World Bank and/or the IMF, instead of local data. Chapter IV: Methodology for indicator calculation 37 Biased data are also an important problem; bias can be intentional or subconscious. Deliberate biases can be introduced when staff involved in the survey are very motivated to show that their programme or region is performing very well, or when personnel in the facility investigated have the feeling they are being inspected. On the other hand, some people may have an interest in introducing a negative bias. Whilst not a great deal can be done about bias that one is unaware of, some "tips" can help: have a supervisor in the field often, and ask data collectors to note down anything unusual they see, as well as remarks from personnel in the facilities investigated. It should be clear to anyone involved in collecting, analysing or presenting the data that the purpose of the exercise is to measure what is really going on, in order to improve the situation, and not to control anybody. ANALYSIS AND REPORTING Step 13: Calculate indicators Once the data needed for the calculation of each indicator have been collected through interviews, record reviews or surveys, the next step is to process them. The data processing should be carefully designed and resources for processing should be identified before starting data collection. Correct design of data collection forms will facilitate processing. The main part of the processing will be done at central level by the monitoring unit, which will receive all the forms completed during interviews, record reviews and surveys. All the calculations can be done manually, but a computerized spreadsheet can also be used to enter the data, consolidate the results and prepare reports. A computerized programme (Excel) is available on request in the Department of Essential Drugs and Medicines Policy to facilitate the data processing and calculation. For the background information, the monitoring unit may report the figures collected on a summary form, such as the one provided in Annex 1 (Summary Form 1: National drug policy background information). For the structural indicators, the monitoring unit may also record the Yes/No answers on a summary form such as the one provided in Annex 1 (Summary Form 2: National drug policy structural indicators). For the process and outcome indicators, more attention needs to be paid to the data processing, as data for the same indicator (numerator and denominator) are sometimes collected in different places (central/peripheral), with different methods (record review/interview/survey), and are reported on different data collection forms. For most of the indicators, processing the data involves two phases: the first one consists of "ticking" and "counting" on each data collection form (e.g. "ticking" whether a certain drug is on the essential drugs list (EDL), counting the total number of drugs prescribed, etc.). It can be Indicators for monitoring national drug policies 38 done either by the monitoring unit, or by the data collectors under the supervision of the monitoring unit. The second phase consists of summarizing the results of all the individual data forms (one per facility surveyed). A form can be created for each indicator or can cover a number of indicators as in Example 1. This phase is carried out by the monitoring unit, which, at the beginning of the exercise, should be careful to separate data from different types of facilities (e.g. private/public sector) and give each facility a code. Example 1 To calculate the indicators PR 9, OT 7, OT 8, etc. it is first necessary to summarize information collected in all the health centres surveyed and recorded on separate data collection forms. HC code Number of prescriptions Number of drugs Number of drugs from EDL Number of prescriptions with injections HC-1 30 85 75 8 HC-2 30 69 44 12 HC-3 29 68 68 5 Total: 89 222 187 25 For these three health centres: PR 9 (187 : 222) x 100% OT 7 (222 : 89) x 100% OT 8 ( 25 : 89) x 100% = 84% of drugs prescribed are from the EDL = 2.49 drugs per prescription = 28% of prescriptions include at least 1 injection All the process indicators and most of the outcome indicators are answered by a percentage. Once obtained, this percentage can be recorded on a summary form such as the ones provided in Annex 1 (Summary Form 1 and Summary Form 2). The method of calculating the indicator will depend on the type of information needed for the numerator and the denominator. It is important to understand that the purpose of the indicators is to identify trends; therefore, it is not necessary to present the results with two decimals, since this will give a false impression of their accuracy. For most process and outcome indicators, a maximum of one decimal is recommended. Chapter IV: Methodology for indicator calculation 39 Example 2 To calculate the percentage of drug outlets inspected in one year (PR1) for example, it will be necessary to count the number of drug outlets inspected, to divide this number by the total number of drug outlets in the country, and then to multiply the figure obtained by 100. The result will be: % of drug outlets inspected = Number of drug outlets inspected –––––––––––––––––––––––––––– – Total number of drug outlets x 100 For this indicator, the data are relatively easy to retrieve as the numerator and the denominator normally come from the same source at central level. Example 3 Other indicators are more time-consuming to obtain. For instance, to calculate the percentage of drugs from the national essential drugs list (EDL) sold (PR10), it will be necessary to count all the drugs from the EDL recorded on Data Collection Form 1 (see page 167), to divide this number by the total number of drugs sold, and then to multiply the figure obtained by 100. The result will be: % of drugs from EDL sold = Number of drugs from EDL sold –––––––––––––––––––––––––– Total number of drugs sold x 100 For this indicator some work will be needed, most probably from the monitoring unit, to classify the drugs into the two categories: EDL drugs and non-EDL drugs. Data Collection Form 1 has been designed to assist in this task. Example 4 In other cases, information for the numerator and the denominator come from different sources. For instance, to calculate the value difference for a drug basket between retail value and CIF/ex-factory value (PR30), it will be necessary to use data from a survey in a sample of drug outlets for retail value (see Data Collection Form 1, page 167) and data from central level for CIF/ex- factory value. Indicators for monitoring national drug policies 40 Example 5 Finally, when calculating the indicators for the first time, some denominators will not be readily available, e.g. average expenditure per prescription out of average expenditure per prescription in the past three years (PR31). In this case an estimate can be made with the assistance of responsible personnel in the area concerned and from a review of studies, if any. If this is not possible, the indicator will be calculated only from the second year. Further detailed information on the calculation of each indicator is given in Chapter V. Step 14: Perform quality control checks Usually the data have already been checked in the field to ensure that all the information has been properly collected and recorded. However, the information should be checked again during data processing for completeness and internal consistency. If some data collection forms have not been filled in completely, data will be missing for some of the indicators. It may still be possible in most cases to collect the missing information. However, consistency of data collection may be more important than completeness of information, as the information needs to contribute to monitoring activities, to identifying problems and to proposing changes in management and policy. If data seem inconsistent, it may be possible to check with the data collector or to return to the respondent and ask for clarification. If it is not possible to correct information that is clearly inconsistent, it can be decided to exclude this particular part of the data from processing and analysis. However, this decision should be considered carefully, since if too many data are excluded it may affect the validity of the results. When using a computer, quality control checks must also include a verification of how the data have been coded and subsequently entered. Step 15: Analyse and interpret the results After all data have been processed and all indicators calculated, a mass of information will be available; this information should be analysed. It is recommended that a systematic approach is followed which includes the following two steps: · Take just the summary sheets (Annex 1) with the reply/value obtained for the indicators and, where appropriate, the value of the numerators and denominators. Chapter IV: Methodology for indicator calculation 41 · Look at subgroups of indicators, and "evaluate" each subgroup (see Example 6). Example 6: Legislation and regulation In Example 6, the structural indicators related to legislation and regulation show an improvement in 1995; in 1996 the process indicators have improved considerably. This shows that the performance in the field of legislation and regulation has improved; it does not show why. In order to understand why, it is necessary to review all the indicators for key component 1. A possible reason could be that inspection activity has increased; PR 1, PR 3, PR 4 and probably PR 6 and PR 7 will show this. On the other hand, increased adherence to the rules by the facilities inspected could also have caused this improvement; in this case, indicators PR 2 and PR 6 should have decreased. The indicators can thus give an indication on "why there is a change", but in order to know exactly what has happened, a more detailed investigation is often needed. If, in the example above, an increase in inspection activities has been the main cause of the improvement, there are various possible underlying reasons for this increase. To list a few: inspectors may have become more active, they may have started to address some areas that were neglected previously, they could have been given more resources to carry out inspections, or the number of inspectors may have increased. Identifying the questions will help to get information on the reasons for improvements, deterioration or stagnation in implementation of the NDP; this in turn will help to focus attention, resources and efforts, and to identify the inputs that have had a real impact. Step 16: Report results As a summary of the whole process, a report should be prepared by the monitoring unit in order to communicate the information obtained, the results and conclusions to the 0% 25% 50% 75% 100% 1994 1995 1996 ST PR Indicators for monitoring national drug policies 42 department which has requested the information (usually the department in charge of national drug policy at the ministry of health). This report should be the basis for decisions on drug policy and strategies. It can be a persuasive and influential tool if it is well prepared, and if results are interpreted and discussed in a comprehensive manner, taking into account the objectives and the strategies of the drug policy. In addition to this official report, there are many other ways to make the results widely known (presentations, articles, etc.). One important rule to follow in order to get people interested in the subjects discussed is to limit the scope to issues that will be of interest to the audience: the part they are responsible for, can influence or are (directly) concerned by. Therefore, presentations, reports, etc. should be tailored to specific audiences and results should be provided in a clear and visually attractive way. Reporting and presenting the results of a monitoring exercise is not difficult — but it does require some effort. Three mistakes are to be avoided, when you bring your work to the attention of others: · do not present results that are not relevant: people will feel that they are of no concern to them; · do not present results in a chaotic or "complicated" way: people will be lost; · do not present results in a boring way: people will fall asleep; make graphics, limit the information given, present results not indicators, etc. Step 17: Use the results for action No monitoring or evaluation exercise can be considered useful unless it leads to action. Results of monitoring NDP implementation should lead to action on at least three levels: the central level, the "specialist" audiences and/or intermediate levels, and the health facility level. At each of these levels, there are three types of action: recognize what is well done and encourage its continuation, correct what is not well done, and change what does not work (if necessary after further investigation into the reasons why it does not work). Central level At central level (MOH policy-makers) the results from the indicators can be used to define (new or additional) strategies, to update them, to reallocate resources (human as well as other) or to adjust plans and targets. Recognition of persons/ units/departments which are performing well is important in order to motivate them, as is identification of bottlenecks and problem areas. Problems may require additional research to understand better the underlying reasons for them before corrective action can be taken. Chapter IV: Methodology for indicator calculation 43 "Action" at central level can mean either action by the central level (this will usually be needed when structures are not in place) or stimulating other levels in the health care system to take action or to improve the quality of their action. Indicators for monitoring national drug policies 44 The "specialist" audiences and/or intermediate levels This refers to all those levels and structures that are involved in implementation of (part of) the NDP: drug inspection, central medical stores, private pharmacies, manufacturers etc. They can take action and start to improve their work after being stimulated to do so by the central level. However, they can also take action (within the limits of their means and responsibilities) for improvement themselves. Wide dissemination of the results is therefore important: it can help create awareness at different levels that performance is sub-optimal, initiate discussions on how to improve, and eventually lead to action. The health facilities Health facilities are not different from the previous group: they are responsible for implementing part of the NDP (and certainly not the least important part!). And they can do their part in improving the performance of some indicators (on "rational" prescription, e.g. PR 9 and OT 9), whilst other indicators may only partly depend on them (e.g. PR 29 concerning the average stockout duration at remote health facilities. Those stockouts can occur for various reasons — the central procurement unit may not perform very well, but bad stock management at the facilities may also be the cause). For individual health facilities to improve their performance, they need to be given feedback on how they are performing and how/where they can (should) improve. The more specific this feedback is, the more helpful it will be to the health facility concerned, and the more likely it is that action will be taken. Furthermore, feedback should be given in a constructive way (that is, focusing on how to improve), rather than "telling them what they do wrong". Step 18: Evaluate the whole process The monitoring unit should review and evaluate the full process with the staff involved. The best approach for this evaluation is to analyse systematically during a seminar all the procedures used in each phase, including sampling techniques, methods for data collection, training of data collectors, organization of surveys, etc. All obstacles and difficulties encountered during the process should be reviewed and ways to address them in future work should be identified. This activity can provide a good opportunity for further discussion on the ways to set up a more regular monitoring system. Chapter IV: Methodology for indicator calculation 45 CONDUCTING SURVEYS Although most of the data for calculating the indicators can be obtained from the existing monitoring systems and from reviews of documents/records and interviews at central level, data for some process and outcome indicators will need to be collected in health facilities and drug outlets through special surveys. These surveys can be organized to collect data for several indicators at the same time. This section provides the list of indicators which require surveys, guidelines on survey design and implementation including sampling issues, and gives examples of sampling procedures. List of indicators which require surveys For the proposed list of indicators, the data for nine process indicators and nine outcome indicators will need to be collected through special surveys in the majority of countries. Process indicators PR9: Number of drugs from the national essential drugs list (EDL) prescribed, out of total number of drugs prescribed. PR10: Number of drugs from the national essential drugs list (EDL) sold, out of total number of drugs sold. PR27: Average time between order and delivery from central store to remote facilities in the last year, out of average time between order and delivery in the past three years. PR29: Average stockout duration for a basket of drugs in a sample of remote facilities in the last year, out of average stockout duration for the same basket in the past three years. PR30: Value of a basket of drugs, out of CIF/ex-factory value of the same basket. PR31: Average expenditure per prescription, out of average expenditure per prescription in the past three years. PR32: Value of a basket of drugs, out of value of the same basket in the year of reference. PR33: Number of prescribers having direct access to a (national) drug formulary, out of total number of prescribers surveyed. PR35: Number of prescribers who have attended at least one training session in the last year, out of total number of prescribers surveyed. Indicators for monitoring national drug policies 46 Outcome indicators OT1: Number of drugs from a basket of drugs available in a sample of remote health facilities, out of total number of drugs in the same basket. OT2: Number of drugs at the lowest price from a basket of drugs, out of total number of drugs in the same basket. OT3: Average retail price of standard treatment of pneumonia, out of the average retail price of a basket of food. OT4: Value of a basket of drugs, out of the value of the same basket with the cheapest drugs. OT5: Number of drugs/batches that failed quality control testing, out of the total number of drugs/batches surveyed. OT6: Number of drugs beyond the expiry date, out of the total number of drugs surveyed. OT7: Average number of drugs per prescription. OT8: Number of prescriptions with at least one injection, out of the total number of prescriptions surveyed. OT9: Number of children under five with diarrhoea receiving antidiarrhoeal drugs, out of the total number of children under five with diarrhoea surveyed. Survey design and implementation The various steps which have been described above (see pages 29 to 43) apply also to surveys and need to be carefully followed. However, additional specific steps are necessary and are reviewed below. Selecting the type of facilities The data needed for calculating the 18 indicators should be collected in two types of facilities: F drug outlets (public and private); F health facilities (public, private and remote). Table 1 gives the sites for which data should be collected for the 18 indicators requiring surveys. Chapter IV: Methodology for indicator calculation 47 Table 1: Survey sites for the 18 indicators Indicators Private drug outlets Public drug outlets Private health facilities Public health facilities Remote health facilities PR9 l l PR10 l PR27 l PR29 l PR30 l PR31 l l PR32 l l PR33 l l PR35 l l OT1 l OT2 l OT3 l l OT4 l OT5 l l OT6 l l OT7 l l OT8 l l OT9 l Selecting the type of survey Most of the data will be collected through cross-sectional surveys carried out in drug outlets and health facilities. Data will be collected from current patients as they present to the drug outlet or from prescribers/dispensers on the day of the data collector's visit. For process indicators PR27 and PR29, data will be collected from existing records and inventory controls. The literature shows13 that individual health providers tend to exhibit more or less consistent practices over time. Therefore, a sample drawn at one point in time will provide basically the same results as a sample that covers a longer period. However, since data will generally be collected over a short period, they may suffer from bias due to seasonality, variations in staffing, inconsistencies in the supply cycle, or the fact that providers are aware that their 13 For more information, see the document How to investigate drug use in health facilities (WHO/DAP/93.1), in which some of the methodological principles used in this manual are described. Indicators for monitoring national drug policies 48 behaviour is being observed. Data collectors should be trained to try to guard against these possible sources of bias. Drawing a sample To estimate indicators accurately, it is important to follow specific procedures for drawing samples of health facilities and drug outlets. These procedures will vary depending on the context of each country and the availability of data. The selection of samples should be made in such a way that each sampling unit has an equal chance of being included in the sample. In this way, selection bias can be avoided and the study population will be representative of the reference population. However, there are often major logistical constraints in carrying out surveys, such as transport, time, budget and lack of data collectors with specific skills. The best design for a particular survey therefore depends not only on statistical theory, but also on the practical aspects of collecting the data. Different strategies for sampling drug outlets and health facilities can be used: Simple random sampling This is the simplest form of probability sampling. To select a simple random sample, the first step is to make a numbered list of all the units from which the samples are to be drawn (sampling frame), e.g. all the drug outlets or health facilities. The next step is to decide on the size of the sample, i.e. the number of units that need to be randomly chosen from the sampling frame (see page 48). Then the required number of sampling units is selected using a "lottery" method or a table of random numbers (see Annex 3). The lottery method assigns numbers to all the units; these numbers are then mixed and the required number drawn at random (without replacement). With a table of random numbers, each sampling unit is assigned a number and the required numbers are selected from the table. All units with these numbers constitute the samples. Simple random sampling ensures that the indicators are unbiased, but may not be the most efficient procedure. Systematic sampling Sometimes the most convenient way of obtaining a sample is by choosing the sampling units directly from the sampling frame (e.g. taking every n drug outlet from a list of all drug outlets). Ideally a random number is used to decide where to start. The sampling interval will be calculated by dividing the total number of drug outlets existing in the country by the desired sample. For example, if a systematic sample is to be selected from 360 drug outlets existing in the country and a sample size of 20 has been chosen, the sampling interval will be 360/20 = 18. In other words, one in every 18 drug outlets will be included in the sample until 20 drug outlets have been obtained. This strategy is used in the model sampling procedures described below (see pages 49 to 58). Chapter IV: Methodology for indicator calculation 49 Stratified sampling In this type of sampling, units (e.g. health facilities/drug outlets) are put into groups according to a characteristic (such as urban/rural area), and the sample is apportioned among these groups according to a set plan, chosen to ensure their representation in the sample. Stratification, along or combined with systematic sampling, can achieve very efficient designs. This strategy is used in the model sampling procedures described below (see pages 49 to 58). Cluster sampling It may be difficult or impossible to take a random sample of sampling units (e.g. health facilities/drug outlets) in the study population, either because a list of all the drug outlets or health facilities does not exist, or because of other logistical difficulties (e.g. visiting drug outlets which are scattered over a large area may be too time- consuming). However, when a list of groupings of sampling units — clusters — is available (e.g. districts or provinces) or can be easily compiled, then a random sample of clusters can be selected. Within the clusters that are finally selected, the sampling units (e.g. drug outlets, health facilities) are listed and sampled. Multi-stage sampling A multi-stage sampling procedure is carried out in phases and usually involves more than one sampling method. A first stage sampling could, for instance, be cluster sampling of districts and the next stage, sampling of drug outlets within the selected districts. The strategy proposed as an example in this manual is a multi-stage sampling using systematic sampling for selection of drug outlets and health facilities. Determining the sample size One difficult aspect of designing a sample is deciding how many health facilities, drug outlets, prescribers and prescriptions to include. The sample size is usually a compromise between what is desirable and what is feasible. The sample size should be the smallest one that will give an estimate of proportion within the desired degree of precision. The size of the sample is also determined by the availability of time, human resources, transport and money. Countries with large human and financial resources may wish to make their own decision on a representative sample size. In such cases this should be done with the support of statisticians. The sample size depends on the degree of precision needed and the anticipated proportion of the characteristics under study. For countries with limited human and financial resources and major logistical constraints on carrying out surveys, a recommended sample size of health facilities, drug outlets, remote health facilities, prescribers and prescriptions has been determined: Indicators for monitoring national drug policies 50 F For collecting data for indicators PR9, PR10, PR31, OT7 and OT8, 20 drug outlets or health facilities should be selected randomly and 30 prescriptions or drugs sold per health facility or drug outlet should be collected, amounting to a total sample size of 600 prescriptions or 600 drugs sold (see details on pages 49 to 58). For indicator OT9, it is suggested that the first 5 prescriptions seen for children under 5 years old with diarrhoea should be collected in 20 health facilities, amounting to a total sample size of 100 prescriptions. Because the treatment practices of individual providers are consistent and similar among providers within the same facility, in-facility sources of variation will tend to be reduced, and after a certain point, adding prescriptions to a sample within a facility provides very little new information. The principal source of variation will tend to be differences in practice between health facilities. Increasing the number of facilities in a sample will be the best way to obtain more accurate and reliable estimates and will be better than increasing the number of prescriptions sampled within facilities.14 Because of the substantial variations in practice among facilities for many indicators, it would be hazardous to generalize about a large population of facilities from a sample that includes fewer than 20 facilities. F For collecting data for indicators PR33 and PR35, it is suggested that 40 public sector health facilities should be selected randomly (see details on pages 49 to 58). Within these facilities a minimum of 100 prescribers should be included in the sample to achieve any statistically significant comparison from one year to the other. F For collecting data for indicators PR27, PR29, PR30, PR32, OT1, OT2, OT3, OT4, OT5 and OT6, it is suggested to use the previously selected 20 private drug outlets, to randomly choose 20 of the 40 previously selected public sector health facilities and to randomly select 20 remote health facilities (see details on pages 49 to 58). Proposed sampling procedures The following procedures are intended to assist national managers in selecting representative samples of health facilities, drug outlets and prescribers, for collecting data needed for the calculation of the 18 indicators. They are designed as examples for countries with limited resources. However, as stated previously, countries can use other methods and sample size as long as the samples are representative of the country's characteristics. These procedures are based on the assumption that owing to logistical constraints, surveys should preferably be conducted in a limited number of regions, to the extent that an acceptable sample can be drawn. The selection of the regions where the surveys will be conducted depends on the sites of the surveys: private drug outlets, public drug outlets, health facilities and remote health facilities. 14 See footnote 13. Chapter IV: Methodology for indicator calculation 51 Therefore the procedures described below include two stages: the selection of the regions where the surveys will be conducted and the selection of a representative sample of drug outlets, health facilities, prescribers and prescriptions. Select the regions where the surveys will be conducted Selection of regions for the surveys in private drug outlets In most developing countries, private drug outlets are mainly in the capital city area and in the major urban areas. This should be taken into account when selecting regions in which to conduct the survey. First, divide the country into geographical units based on administrative regions/districts, with one unit being the capital city area/region. Each unit should include at least one significant urban area (according to the size of the population of the country). Therefore a rural region/district should be merged with one adjacent region/district which includes a significant urban area. A basic principle should be to merge a rural region/district with an adjacent one where there is a reference hospital or an active regional capital with trading activities. Five to 20 geographical units is an acceptable range. Second, owing to the particular distribution of private drug outlets in most developing countries, it is suggested to select a sample of geographical units as follows: (a) one geographical unit will be the capital city area/region, where a large number of private drug outlets (G0) is usually concentrated; (b) three geographical units will be randomly selected, with a probability for each unit to be selected proportionally to the number of drug outlets per geographical unit. To achieve this, list all private drug outlets in the country, excluding those located in the capital city area/region, and number these drug outlets from 1 to n2. Use a table of random numbers (see Annex 3) to draw a number between 1 and n2 and select the geographical unit which corresponds to that number. Repeat the procedure until three different geographical units are selected (G1, G2 and G3) (see Figure 1). Indicators for monitoring national drug policies 52 Figure 1 Example of selection of geographical units for determining a sample of private drug outlets Selection of regions for the surveys in public sector drug outlets and health facilities The selection of regions where the surveys will be conducted in public sector drug outlets and health facilities can follow two different procedures: (a) In order to facilitate the logistics and to reduce time and cost, the regions sampled for the private drug outlets (the capital city region G0 plus the three geographical units G1, G2 and G3) can be selected. Such a choice is acceptable for countries with limited human and financial resources. But it is based on the assumption that the distribution of public health drug outlets and health facilities is the same as that of private sector drug outlets. This is rarely the case. However, in order to simplify the procedures, it is wise to use this method for the first years in most developing countries. (b) Because the public sector drug outlets and health facilities are usually not concentrated in the major cities, the grouping of regions in geographical units based on major cities is not necessary. Therefore the administrative division Total number of private drug outlets in the country (n) Total number of private drug outlets in the country (n) List of private drug outlets in the rest of the country (n2) List of private drug outlets in the rest of the country (n2) List of private drug outlets in the capital city region (n1) List of private drug outlets in the capital city region (n1) Randomization G1 G2 G3 G1 G2 G3G0 G0 Chapter IV: Methodology for indicator calculation 53 in regions/districts can be used for establishing the list from which a sample of four regions will be chosen. The decision to automatically include the capital city region/district is the responsibility of the central team. It will depend on the country's context. However, in some countries, such choice would reduce the logistical problems and is therefore relevant. In most of the countries, depending on the size of the regions/districts, four to five geographical units are normally sufficient. For selecting the geographical units, list all the drug outlets/health facilities of the country and number them from 1 to n. Use a table of random numbers (see Annex 3) to draw a number between 1 and n and select the region which corresponds to that number. Repeat the procedure until four different geographical units are selected. Selection of regions for the surveys in remote health facilities Remote health facilities can be defined as health facilities located more than 100 km from the capital city or from any city with more than 100,000 inhabitants. In order to facilitate the logistics and to reduce time and cost, the regions sampled for the public sector drug outlets/health facilities can be selected for the remote health facilities, as these are mainly public sector health facilities and some may have already been selected by randomization when selecting health facilities. In conclusion, it is suggested to use the same regions for all the surveys, at least during the first years of the monitoring process. This will reduce logistical problems and budget. Select a representative sample of facilities where information will be collected Private drug outlets In order to collect the data needed for calculating process indicators (PR9, PR10, PR30, PR31 and PR32) and outcome indicators (OT2, OT3, OT4, OT5, OT6, OT7 and OT8), a sample of at least 20 private drug outlets is required (see page 49). Some countries may prefer to select a larger sample to get more accurate figures. However, such a sample size can be considered acceptable for providing the data needed to calculate indicators with a reasonable level of accuracy in most developing countries (especially those with a low income and few resources). For selecting such a sample of private drug outlets, which should take into account specificities of the drug sector, different procedures could be used. The following is an example which is reasonably easy to implement. After selecting the geographical units (see page 50), list all private drug outlets located in the capital city area/region (list L1) and number these drug outlets from 1 to n1. Indicators for monitoring national drug policies 54 The proportion (p = n1/n) of private drug outlets in the capital city area/region (n1) out of the total number of private drug outlets in the country (n) should be used for defining the size of the sample of drug outlets which will be selected from the capital city area/region (sa). For a total sample (S) of 20 drug outlets, the number of drug outlets which will be selected from the capital city area/region will be sa = p x 20. Then the number of drug outlets which will be selected from the other geographical units will be sb = 20 – sa. Use a table of random numbers to draw a number between 1 and n1 in the list L1. The number obtained will correspond to the first drug outlet selected. Then move down the list using a sampling interval of i1 = n1/sa to find out the second private drug outlet. When you reach the bottom of the list, go back to the top and repeat the operation. Use the same method until you select the required number of private drug outlets in the capital city area/region (sa). This can be considered an acceptable sample of private drug outlets in the capital city area/region where data for calculating indicators can be collected. Figure 2 Proposed sampling procedure for selecting a sample of 20 private drug outlets Total number of private drug outlets in the country (n) Total number of private drug outlets in the country (n) List of private drug outlets in the capital city region (n1) List of private drug outlets in the capital city region (n1) List of private drug outlets in the three selected geographical units (n2) List of private drug outlets in the three selected geographical units (n2) Sample of private drug outlets in the capital city region (sa = p x 20) Sample of private drug outlets in the capital city region (sa = p x 20) Sample of private drug outlets in the three selected geographical units (sb = 20 – sa) Sample of private drug outlets in the three selected geographical units (sb = 20 – sa) Sample of 20 private drug outlets (S = sa + sb) Sample of 20 private drug outlets (S = sa + sb) Randomization (p = n1/n) Randomization Chapter IV: Methodology for indicator calculation 55 List all private drug outlets of the three selected geographical units (list L2) and number these drug outlets from 1 to n2. Use a table of random numbers to draw a number between 1 and n2 in the list L2. The number obtained will correspond to the first drug outlet selected. Then move down the list using a sampling interval of i2 = n2/sb to determine the second private drug outlet. When you reach the bottom of the list, go back to the top and repeat the operation. Use the same method until you select the required number of private drug outlets in the three selected geographical units (sb). Such a sample can be considered an acceptable sample of private drug outlets in the selected regions/districts where data for calculating indicators can be collected. Then add the sample of drug outlets located in the capital city area/region (sa) to the sample of private drug outlets in the selected regions/districts (sb) for obtaining a sample (S) of 20 private drug outlets. Using this methodology, such a sample can be considered an acceptable sample of private drug outlets where data for calculating indicators can be collected (Figure 2). If when collecting data a drug outlet does not exist any more, replace it by the next one on the list. Public sector drug outlets and health facilities In order to collect the data needed for calculating process indicators (PR33, PR35) a sample of at least 40 health facilities is required (see page 49); for process indicators (PR9, PR31, PR32) and outcome indicators (OT3, OT5, OT6, OT7, OT8, OT9), only a sample of 20 public drug outlets/health facilities is needed (see page 49). This second sample can be selected from the first using simple randomization, as in many countries most public sector drug outlets are located within health facilities. Some countries may prefer to select a larger sample to get more accurate figures. However, such a sample size can be considered acceptable for providing data to calculate the indicators with a reasonable level of accuracy in most developing countries (especially those with a low income and few resources). For selecting such a sample of public drug outlets and health facilities, which should take into account specificities of the drug and health sectors, different procedures could be used. The following is an example which is reasonably easy to implement (Figure 3). After selecting the regions (see pages 50 to 52), a sample of public sector drug outlets and/or health facilities can be selected using the same procedures as for private drug outlets. Once a sample of 40 health facilities has been obtained, randomly select 20 public drug outlets and/or health facilities for the indicators where this size of sample is sufficient. Indicators for monitoring national drug policies 56 Figure 3 Proposed sampling procedure for selecting a sample of 40 health facilities and 20 public drug outlets The question of the respective proportion of urban and rural drug outlets and/or health facilities has to be addressed only if it is considered that practices are different in the two situations. In this case a stratified sampling strategy should be used to ensure a minimum number of drug outlets and/or health facilities from urban and rural areas. In the four selected geographical units, list all the public sector drug outlets or the health facilities located in urban areas (L1) and those located in rural areas (L2). Using the same strategy as above in each area, select a sample of drug outlets or facilities for each area. Remote health facilities In order to collect the data needed for calculating process indicators PR27 and PR29 and outcome indicator OT1, a sample of 20 remote health facilities is required Total number of public sector health facilities/ drug outlets in the country (n) Total number of public sector health facilities/ drug outlets in the country (n) Sample of 40 public sector health facilities/drug outlets (S = sa + sb) Sample of 40 public sector health facilities/drug outlets (S = sa + sb) List of public sector health facilities/drug outlets in the capital city region (n1) List of public sector health facilities/drug outlets in the capital city region (n1) 20 public sector health facilities/drug outlets 20 public sector health facilities/drug outlets Sample of public sector health facilities/drug outlets in the capital city region (sa = p x 40) Sample of public sector health facilities/drug outlets in the capital city region (sa = p x 40) Sample of public sector health facilities/drug outlets in the three selected geographical units (sb = 40 – sa) Sample of public sector health facilities/drug outlets in the three selected geographical units (sb = 40 – sa) List of public sector health facilities/drug outlets in the three selected geographical units (n2) List of public sector health facilities/drug outlets in the three selected geographical units (n2) Randomization Randomization Randomization (p = n1/n) Chapter IV: Methodology for indicator calculation 57 (see page 49). Some countries may prefer to select a larger sample to get more accurate figures. However, such a sample size can be considered as acceptable for providing data needed to calculate the indicators with a reasonable level of accuracy in most developing countries (especially those with a low income and a population of less than 20 million). For selecting such a sample of remote health facilities, which should take into account specificities of the health sector, different procedures could be used. The following is an example which is reasonably easy to implement. Figure 4 Proposed sampling procedures for selecting a sample of 20 remote health facilities After selecting the regions (see pages 50 to 52) list all the remote health facilities in these regions. Consider the remote health facilities already selected in the previous selection of health facilities (sa) as part of the sample of remote health facilities (S). Number the remaining ones from 1 to n1. Use a table of random numbers to draw a number between 1 and n1. The number obtained will correspond to the first remote health facility selected. Then move down the line using a sampling interval of i = n1/sb (sb is the number of remote health facilities which have to be selected according to List the remote health facilities in the selected geographical units (n) List the remote health facilities in the selected geographical units (n) List the remaining ones (n1) List the remaining ones (n1) List the remote health facilities already selected (sa) List the remote health facilities already selected (sa) S = sa + sb S = sa + sb Sample of remote health facilities (sb = 20 – sa) Sample of remote health facilities (sb = 20 – sa) Randomization Indicators for monitoring national drug policies 58 the size of sample of remote health facilities needed S = sa + sb). When you reach the bottom of the list, go back to the top and repeat the operation. Use the same method until you select the required number of remote health facilities in the selected geographical units. They can be considered an acceptable sample of remote health facilities in the selected regions/districts where data for calculating indicators can be collected (Figure 4). If when collecting data a health facility is not operational, replace it by the next one on the list. Select a sample of data sources Prescriptions or drugs sold In order to collect the data needed for calculating process indicators (PR9, PR10 and PR31) and outcome indicators (OT7 and OT8), a sample of 600 prescriptions or 600 drugs sold is required (see page 49). The following is a procedure which is easy to implement. In each of the 20 drug outlets/health facilities already selected, take the first 30 prescriptions or 30 drugs sold. Although the selection of the 30 first prescriptions or drugs sold is convenient, it may introduce some bias. If the number of prescriptions or customers buying drugs anticipated per day is high, you may choose to take one prescription or one customer in every three. On the contrary, if it is difficult to obtain 30 prescriptions or drugs sold in one day, you may return the following day to complete the data. However, a reasonable time limit should be set for staying at one unit (see box). Example 7: Data collection in Guinea A data collector in one remote facility reported: "I have spent two full days in this health centre now, and so far I have seen 5 prescriptions only." Obviously, it would not be very realistic to require the data collector to stay until having collected 30 prescriptions in this health centre, which would take more than 2 weeks. Possible actions to reduce the risks of being short of data in any one facility include: · On the local market day, the number of patients can be substantially higher, so consider surveying remote facilities on that day. · In cases where there is a good, reliable system of recording all prescriptions, data recorded the previous day could be added to data collected during the data collector’s visit, to make 30. Chapter IV: Methodology for indicator calculation 59 Prescribers In order to collect data needed for calculating process indicators PR33 and PR35, a sample of at least 100 prescribers is required (see page 49). The following is a procedure which is easy to implement. Select all prescribers up to five in each of the 40 selected health facilities. If there are more than five prescribers in some health facilities, list all of them and select five randomly. If there are fewer than five prescribers, include all of them in the sample. Drugs for quality control testing In order to collect data needed for calculating outcome indicator OT5, a sample of at least 20 drugs is needed. The following is a procedure which is easy to implement. The monitoring unit should indicate on the data collection form which of the drugs from the basket should be collected at each outlet. When visiting the drug outlet, the data collector should pick the selected drug randomly. CALCULATING THE VALUE OF A BASKET OF DRUGS Selection of a basket of drugs Various methods can be used to constitute a basket of drugs for monitoring trends in prices. However, for a basket of drugs to be representative of the national drug consumption, a number of principles should be respected in the choice and the number of the drugs. Choice of drugs The drugs to be included in the basket should be selected from among the essential drugs which are most used in the country. This selection should be carefully made according to the following steps: F the drugs to be included in the basket should be from the national list of essential drugs (EDL). Essential drugs are those that satisfy the health care needs of the majority of the population and should be available at all times in adequate quantity and in the appropriate dosage form. Most countries have a national list of essential drugs; the other countries can use the model list of WHO; F drugs to be included in the basket should also be selected on the basis of their general usefulness and their wide availability internationally. The advantage of using the same selected list (see Table 2 for a model of a basket of drugs) is that comparison could be made among different countries; Indicators for monitoring national drug policies 60 F some drugs (for example, acetylsalicylic acid and chloroquine) are used with different dosages. For each of the drugs, countries should select those which are most commonly used and on the basis of local availability and need; F criteria for selecting drugs for the basket should include the concept of affordability for the majority of the population. For example, praziquantel, although useful in many developing countries and especially many of the poorest ones, should not be included in the selection as it is an expensive drug which cannot be afforded by many people. Number of drugs The number of drugs to be included in the basket cannot be defined at global level as it depends on many variables which are country-specific (level of precision required, type of consumption, pricing policy, etc.). The ideal would be to follow the price of as many products as possible and eventually of all the products. Experience shows that such a decision implies expensive, lengthy procedures for selecting the basket and for monitoring the price trends. However, for calculating the indicators in the manual, it is recommended that the number of drugs in the basket should be limited, on the condition that these drugs are (i) on the national list of essential drugs, (ii) among the most important therapeutically and (iii) among the most used. If these principles are respected, a basket of 10 essential drugs can be considered sufficient to calculate indicators PR21, PR22, PR30, PR32 and OT4 in a developing country with limited resources. Such a basket is also suitable for building a price index to monitor price trends. An example of such a basket of drugs is given in Table 2. Table 2: Example of a basket of drugs acetylsalicylic acid tablet, 300 mg aluminium hydroxide tablet, 500 mg amoxicillin tablet, 250 mg chloroquine tablet, 150 mg (as phosphate or sulfate) ferrous salt tablet, equivalent to 60 mg iron mebendazole chewable tablet, 100 mg metronidazole tablet, 500 mg paracetamol tablet, 500 mg phenoxymethylpenicillin tablet, 250 mg (as potassium salt) sulfamethoxazole + trimethoprim tablet, 400 mg + 80 mg Chapter IV: Methodology for indicator calculation 61 Calculate the value of the selected basket Different methods exist to calculate the value of a basket of drugs for a given year. For use as a basis for a monitoring system on price trends, the price of each of the drugs of the basket should be weighted by its level of consumption. The method proposed in the manual is simple to implement and scientifically acceptable. It is based on principles used everywhere to calculate price indexes. The main problem with this method is the need to know with some precision the drug consumption for the year used as reference year (Y0), as this consumption will be used as a basis for calculating the price index for the following years. In many developing countries, such data on drug consumption are not reliable. In these situations, the value of the basket can be based on a reasonable estimation of the consumption of each of the selected drugs. This reference basket will then be used every year until the quality of the data collection improves. A clear increase or decrease in the consumption of certain drugs of the basket, and/or therapeutic changes, will necessitate its modification. How to calculate the value of the basket of drugs for the year of reference - For a given year (for instance 1997), take the first 20 causes of consultations and the prevalence of each cause (p1, p2,., p20, i.e. pi). This can be obtained nationally if a health information system exists or from a representative sample of health facilities. The sample proposed for collecting data for certain indicators can be used for this purpose. - For each of the causes, the standard treatment (or the most common treatment) used in the country should be noted (name of the drug under INN, standard dosage, length of the treatment). - Select 10 essential drugs among those most used for the treatment of the 20 first causes of consultations. For each drug, calculate an estimated consumption by cause of consultation (Q1, Q2,., Q20, i.e. Qi) by multiplying the number of units (tablets, vials, etc.) included in the standard treatment (a1, a2,., a20, i.e. ai) by the prevalence (pi); then add all the quantities obtained for each drug: Qi = Sai x pi = a1 x p1 + a2 x p2 + . + a20 x p20 - Calculate the average price for the given year of each of the drugs of the basket (P1, P2. P10, i.e. Pi) from a survey in a representative sample of drug outlets (the sample proposed for collecting data for certain indicators can be used for this purpose). According to the indicator, the calculation will be done for different prices: CIF/ex- factory price, international reference price or retail price. Indicators for monitoring national drug policies 62 - The value of the reference consumption for each drug of the basket is obtained by multiplying the consumption Qi by the price Pi. For the year Y0, the value of the basket will be: V0 = SPi x Qi = P1 x Q1 + P2 x Q2 + . + P10 x Q10 This figure will be designated arbitrarily as the basis 100 of the value of the consumption calculated for the year of reference Y0. How to calculate a drug price index A drug price index can be defined as the value of a basket of drugs calculated for a year Yx relative to the value of the same basket in the year of reference Y0. The formula for the price index will be: x/o i x i i o i I = P Q P Q 100 å å x x x This means that the price index will give a precise representation of the price trend every year, 100 being the value of reference of the consumption calculated for the year Y0. Chapter IV: Methodology for indicator calculation 63 CALCULATING THE VALUE OF A BASKET OF DRUGS Cause of consultations Name of drugs No. 1 No. 2 No. 3 No. 4 No. 5 No. 6 No. 7 No. . No. 16 No. 17 No. 18 No. 19 No. 20 Quantity of drugs consumed in millions of units Qi = Sai x pi Average unit price for the year Pi x Consumption value of each drug for the year Vx = S Qi x Pix 1. . a1 a2 a3 a4 a5 a6 a7 a. a16 a17 a18 a19 a20 Q1 P1 Q1 x P1 2. . b1 b2 b20 Q2 P2 Q2 x P2 3. . c1 c2 c20 Q3 P3 Q3 x P3 4. . d1 d2 d20 Q4 P4 Q4 x P4 5. . e1 e2 e20 Q5 P5 Q5 x P5 6. . f1 f2 f20 Q6 P6 Q6 x P6 7. . g1 g2 g20 Q7 P7 Q7 x P7 8. . h1 h2 h20 Q8 P8 Q8 x P8 9. . i1 i2 i20 Q9 P9 Q9 x P9 10. . j1 j2 j20 Q10 P10 Q10 x P10 Prevalence p1 p2 p20 - - S Qi x Pi (See Part 2 – Chapter 5) hapter V Detailed presentation of indicators Structural indicators Process indicators Outcome indicators C Chapter V: Detailed presentation of indicators 63 STRUCTURAL INDICATORS These 50 indicators provide qualitative information on the basic structures (including mechanisms and systems) that are considered necessary for implementing a national drug policy. The indicators check whether the basic structures under each key component are present in the country. They do not evaluate the functioning of these structures. For example, these indicators check whether quality control facilities exist, but not whether they work efficiently. The performance of the structures is assessed with the process indicators. The structural indicators monitor the main aspects of the seven key components of drug policy: F The establishment of appropriate legislation and regulation. F The selection of essential drugs and the registration process. F The importance of maintaining a significant drug allocation in the health budget and developing relevant financing policy in the public sector. F The improvement of procurement procedures in the public sector. F The strengthening of drug distribution and logistics in the public sector. F The establishment of a drug pricing policy in both public and private sectors. F The role of information and continuing education programmes to improve drug use. The structural indicators are answered "Yes" or "No" on the basis of information usually available at the central level, with a "Yes" response intended to be positive. Many negative responses would suggest that improvements are required in institutional capacity if the drug sector is to make significant progress towards achieving the overall policy objectives. For certain indicators and according to specific country needs, the monitoring unit can decide to also collect some written statements to understand the situation better or to quantify some indicators. The results of the structural indicators should be analysed together with the results of the process indicators and this in the framework of each of the seven key strategies/components. For instance, to better understand the situation and the progress accomplished in the implementation of a selection of essential drugs and registration of all the drugs (component no. 2), it is necessary to analyse together structural indicators ST12 to ST18 and process indicators PR8 to PR13. For further explanation, see page 20. The structural indicators can be used for assisting national and international decision-makers in formulating strategies and designing interventions to improve the pharmaceutical sector. They can be used in comparing the implementation of pharmaceutical policies in different Indicators for monitoring national drug policies 64 countries. Structural indicators can also be used for advocacy purposes for increasing government and donors' support to the drug sector. Based on field experience with this manual, structural indicators can be collected in a few days if adequate access exists to key personnel in the pharmaceutical sector (public and private). A model summary form is provided in Annex 1 (Summary Form 2). In this chapter, each indicator is described as follows: Definition What is the content of the indicator? Use: What will this indicator measure? Why is this indicator important? Description: What are the definitions of key terms? What is the scope of the indicator? How can the results be interpreted? Sources and methods of data collection and indicator calculation: What are the main sources and methods of data collection? How should the indicator be calculated? Limitations: What are the main limitations of the indicator? Chapter V: Detailed presentation of indicators 65 Legislation and regulation Indicator ST1: Is there an official national drug policy document updated in the past 10 years? Use: To assess the political commitment and the capability of the government to define the objectives of its pharmaceutical policy and the activities to be undertaken for achieving these national objectives. A national drug policy, which covers both the public and the private sectors, is an expression of goals for improving the supply and use of drugs, the priorities among these goals and the main strategies for attaining them. It provides a framework for action. A written statement of the national drug policy is important, not only as it provides a comprehensive and detailed framework for all pharmaceutical development, but also because it explicitly demonstrates the full commitment of the government and the ministry of health. Description: The national drug policy document is an officially approved document which should be widely available. It should contain not only the key objectives of the national policy but also the main strategies proposed by the government for achieving these objectives. Even if key objectives remain the same for long periods of time (e.g. to improve accessibility, affordability, quality and rational use of drugs), the national drug policy document should be updated at regular intervals to take into account changes in the national and international pharmaceutical markets and to adapt strategies accordingly. A national drug policy document is different from a Drug Act. The formulation of a drug policy should be followed by enactment of appropriate legislation to provide a legal basis and make the policy enforceable. Sources and methods of data collection: The national drug policy document should be available from the ministry of health and/or from the national drug authority. In certain cases, the national drug policy is not a separate document but is part of the national health policy document. Limitations: Some countries (especially developed countries) have no official national drug policy document, although the main components of a policy are in place. In this case, the indicator will not be meaningful if taken in isolation. On the contrary, countries can have an updated document and yet still have a pharmaceutical sector where no major strategies are implemented for improving the situation. Indicators for monitoring national drug policies 66 Legislation and regulation Indicator ST2: Is there drug legislation updated in the past 10 years? Use: To assess the existence of a legal framework for the pharmaceutical sector. Because of the characteristics of drugs which make them different from other goods, government should enact legislation to ensure drug safety, efficacy and quality and to regulate production, marketing and dispensing. Description: Drug legislation describes the legal conditions under which pharmaceutical activities should be organized in line with the national drug policy. It covers activities such as drug importation, distribution, production, registration and sales practices. It should clarify what is permissible and what is not in the field of pharmaceuticals as well as laying down who may manufacture or import drugs, and who may prescribe them. It concerns both public and private sectors. Drug legislation is normally voted in by Parliament. The continuing evolution of national and international pharmaceutical markets should lead government to adapt its national drug policy to those changes and therefore to update the law regularly. Sources and methods of data collection: The national drug legislation documents are usually available from the ministry of health and/or from the national drug authority. Limitations: The existence of drug legislation does not imply that the law is fully implemented. Other structural indicators will give more information on the presence at national level of basic structures for ensuring the application of the law (see particularly ST4, ST5, ST7, ST8 and ST9). Some process indicators will give additional information on how the various structures function (see particularly PR1 to PR7). Chapter V: Detailed presentation of indicators 67 Legislation and regulation Indicator ST3: Have regulations based on the drug legislation been issued? Use: To assess if there are regulations governing the standards and procedures for carrying out the provisions of the law. Description: Regulations form the second stage of legislative procedures and are specifically designed to provide the legal machinery to achieve the administrative and technical goals. Most pharmaceutical activities have to be covered by regulations. These describe, for example, what the obligations of the professionals are, their responsibilities and the penal sanctions if they do not respect them. Drug regulation has to be published just after or at the same time as drug legislation. Drug regulation and legislation cannot exist separately. Regulations apply to the same fields as drug legislation; this indicator should therefore be analysed together with ST2. The answer to the indicator is "Yes" if the main aspects of the legislation are covered in the regulations. Sources and methods of data collection: Drug regulatory documents are usually available from the ministry of health and/or from the national drug authority. Limitations: The existence of a drug regulation does not necessarily mean the regulation is fully implemented. Other structural indicators will give more information on the presence at national level of basic structures for ensuring the application of the regulation (see particularly ST4, ST5, ST7, ST8, ST9 and ST11). Some process indicators will give additional information on how the various structures function (see particularly PR1 to PR7). Indicators for monitoring national drug policies 68 Legislation and regulation Indicator ST4: Is there a drug regulatory authority whose mandate includes registration and inspection? Use: To assess the capability of the government to set up a regulatory body able to control the circulation of drugs in the market and to ensure that legislation and regulation are respected. Pharmaceutical products should be safe, effective and of acceptable quality, and should correspond to health needs. This calls for adequate systems for registration and inspection at national level. Description: Experience shows that a regulatory body is necessary to control the activities of the pharmaceutical sector. This body can be an autonomous technical body outside the ministry of health but with a clear mandate from the government (see, for example, the Food and Drug Administration in the United States of America, the Medicines Control Agency in the United Kingdom and l'Agence nationale du Médicament in France) or it can be a division in the ministry of health. However, in many developing countries such regulatory bodies are very weak, or even non-existent. WHO has produced guidelines for establishing small national drug regulatory authorities which can be adapted to the local context. Among the essential functions of such an authority, registration of drugs and inspection are of the utmost importance. Registration of drugs, based on well-established criteria, is a tool to improve control of what is or should be available on the national market. The drug inspection body should support and, where necessary, enforce adherence to the minimum standards in public as well as private institutions. In certain countries, the inspection body is separate from the drug unit in charge of registration, laws, etc. Sources and methods of data collection: Official documents describing the mandate of the various bodies in charge of registration and inspection are normally available from the ministry of health and/or from the national drug authority. Limitations: The existence of a drug regulatory authority does not necessarily mean that it works properly. Other structural indicators will give more information on the presence at national level of basic structures for registration and inspection activities (see particularly ST5, ST7, ST8, ST9, ST11, ST16, ST17 and ST18). Some process indicators will give additional information on how registration and inspection are performing (see particularly PR1 to PR7, PR12 and PR13). Chapter V: Detailed presentation of indicators 69 Legislation and regulation Indicator ST5: Is there a licensing system to regulate the sale of drugs (wholesalers, pharmacists, retailers)? Use: To assess if a system has been set up to control who imports, distributes and sells drugs at country level. Drugs are not like other goods. For instance, if badly procured, stored or used, they can become dangerous. This is why their distribution and sale should be regulated. In most countries, drugs can only be imported, distributed or sold by a person who has a licence granted by the drug authority. Description: Licences are usually granted for a limited period of time, which will vary according to the type of licence and the administrative and technical capacity of the drug authority. The licensing system sets out provisions on who should import drugs, what qualifications people in the importing agency should have and who should dispense and sell drugs. Although experience suggests that drugs should be dispensed by professionals with pharmaceutical knowledge and the ability to advise patients, the development of primary health care in many countries has resulted in a greater role for nurses and village health workers in dispensing drugs in the public sector or in community pharmacies or private drug outlets. The licensing system should incorporate all these practices, including what drugs may be dispensed at different levels of the health care system and minimum standards on storage, inventory control, premises, record-keeping procedures, etc. It should include both public and private sectors. Sources and methods of data collection: Official documents (regulations) stipulating the conditions for licensing and application forms should be available from the ministry of health and/or from the national drug authority. Limitations: The effectiveness of the licensing system is dependent on the functioning of the inspection system. Additional information on this aspect of the pharmaceutical sector will be provided by other structural indicators (see particularly ST7 and ST8) and by some process indicators (see particularly PR1, PR2 and PR3). Indicators for monitoring national drug policies 70 Legislation and regulation Indicator ST6: Are pharmacists legally entitled to substitute generic drugs for brand name products? Use: To assess government commitment to promoting one of the major strategies for making drugs more affordable to the patient — generic substitution — by providing a legal framework. Drugs marketed under generic name are usually less expensive than drugs marketed under brand name. Description: Generic substitution is defined as the practice of substituting a product, whether marketed under a trade name or generic name, by an equivalent product, usually a cheaper one, containing the same active ingredient(s). In a country, generic substitution by all pharmacists or only by some of them (the ones in the public sector, or the ones in the private sector, etc.), or by other paramedical personnel (e.g. nurses), can be allowed through laws or regulations. In certain cases pharmacists can only substitute if the prescriber has indicated this. This indicator belongs to the series of indicators which make it possible to assess the legal aspects of a national drug policy (component no. 1); therefore the answer to the indicator is "Yes" only when the substitution right is included in the legislation or the regulations. Sources and methods of data collection: The legal right to substitute is described in the drug legislation and regulations. These documents are normally available from the ministry of health and/or at the national drug authority. Limitations: This indicator does not measure whether generic substitution is actually practised. In fact, even when generic substitution is legally permitted, it is not always practised unless accompanied by a pricing policy which provides incentives to the pharmacists and also by a large promotion campaign among the general public. Chapter V: Detailed presentation of indicators 71 Legislation and regulation Indicator ST7: Are there legal provisions for penal sanctions? Use: To assess government's commitment to compelling the pharmaceutical sector to comply with legislation and regulation. Description: Legislation and regulation should specify the sanctions that will apply in the event of failure to conform with any provision of the law. Sanctions can be administrative and/or penal according to the level of the offence. Information about these sanctions should be disseminated in the pharmaceutical sector. The national drug regulatory authority or the ministry of health should control compliance with the law through their inspection activities. Sources and methods of data collection: The list of penal sanctions should be available from both the ministry of health and the ministry of justice. The national pharmaceutical association should also be able to provide such documents. Limitations: The existence of a list of penal sanctions does not necessarily mean that in practice sanctions are implemented. Some process indicators will give more information on the level of implementation of sanctions (see particularly process indicators PR3 and PR7). Indicators for monitoring national drug policies 72 Legislation and regulation Indicator ST8: Is there a checklist for carrying out inspections in different types of pharmaceutical establishments? Use: To assess the effectiveness of the inspection system. The existence of simple guidelines on how to inspect various types of pharmaceutical establishments shows the commitment of the drug inspection body to perform its tasks and to improve its effectiveness. Description: Effectiveness of an inspection can be improved by standardizing the procedures. Simple checklists should be developed at national level taking into account international procedures (GMP, etc.). These lists assist the inspectors in controlling the different types of establishments (including producers, wholesalers and retailers in public and private sectors) and in ensuring that regulations are fully implemented and enforced. Sources and methods of data collection: Checklists for carrying out inspection activities are normally available from the ministry of health and/or from the national drug authority and/or in the inspection unit. Limitations: Even if standardization of the inspection procedures is important, the key factors for improving the quality of the inspection system remain the level of training and the commitment of the inspectors. These can partially be measured by some process indicators (see particularly PR1 to PR7). Chapter V: Detailed presentation of indicators 73 Legislation and regulation Indicator ST9: Are there any institutions within or outside the country where quality control is carried out? Use: To assess if there is a reliable mechanism for ensuring that drugs produced, entering or circulating in a country are of acceptable quality. This calls not only for adequate regulations, good manufacturing practices, an effective drug registration and inspection system but also for a quality control system capable of analysing and carrying out regular checks of all drugs used within the country. Description: Quality control can be done both within and outside the country. Many developing countries have set up small national quality control laboratories which test locally produced drugs as well as imported drugs. If there is no reliable control laboratory, drugs can also be tested outside the country by an independent laboratory on a regular basis. The absence of any institution within or outside the country where quality control is carried out suggests that drugs are sold without any guarantee of quality and/or efficacy for the patients. However, quality control is not the only way to ensure the quality of drugs. For example, a well-organized registration system and the systematic use of international documents such as GMP certificates and certification schemes concerning the quality of drugs are also important ways of ensuring drug quality in a country. Sources and methods of data collection: If a national quality control laboratory exists, it should be visited and information should be obtained on the type of quality control performed. If there is no national quality control laboratory, information on the types of controls carried out outside the country should be available from the ministry of health and/or the national drug authority. Limitations: This indicator should be analysed with caution and will in some cases need to be accompanied by written statements on what kind of quality assurance system is in place. Indeed, in certain cases there is a quality control laboratory, but it does not function properly or only for certain drugs (e.g. drugs from the public sector). In other cases, there is no laboratory, but the other aspects of the quality assurance system are functioning very well. Indicators for monitoring national drug policies 74 Legislation and regulation Indicator ST10: Is the WHO Certification Scheme on the Quality of Pharmaceutical Products Moving in International Commerce used systematically? Use: To assess if the government uses all available tools to ensure drug quality. Government has the responsibility for monitoring and controlling the pharmaceutical substances it allows to be imported. The WHO certification scheme can play an important role in this endeavour. Description: The WHO certification scheme can be used for imported drugs to ensure that they are of good quality. Under the certification scheme, the exporting country must certify that the drug is registered and authorized for sale in that country. The competent authority of the exporting country must also certify that the manufacturer's facilities are inspected regularly. The scheme also provides the product information which is issued in the exporting countries. It does not provide assurance on the quality of each batch. The scheme can be used for all imported products, both for the public and the private sectors. Sources and methods of data collection: Information on the use of the WHO scheme can be obtained from interviews with the main importers, including the central procurement unit. In certain cases, the scheme will not be used by all the importers and this can be noted in the final report. Limitations: The validity of the WHO certification scheme depends on the quality of the inspection system of the exporting country, as the inspectors are the ones assessing if GMP are followed by the producers. It depends also on the seriousness of the national drug authority in completing the certificate. Chapter V: Detailed presentation of indicators 75 Legislation and regulation Indicator ST11: Are there controls on drug promotion based on regulations and consistent with the WHO Ethical Criteria for Medicinal Drug Promotion? Use: To assess if the tools for better control of drug promotion are in place. Controlling the marketing, presentations and types of medicines can play an important role in preventing irrational use of drugs. It is the specific responsibility of government to develop measures to ensure that promotional practices involving drugs are in keeping with acceptable ethical standards. Description: The controls on drug promotion and advertising can be of various types: drug promotion can be controlled before the advertisement or the promotion is launched or it can be done retrospectively on the basis of a set of rules and regulations. If drug registration is functioning properly, the guiding principle is that information and promotional activities should be consistent with the terms and conditions of product approval. To enforce adherence to ethical criteria requires their inclusion in legislation and provisions for sanctions. To assess rules and regulations drawn up at national level, the ethical criteria for medicinal drug promotion, developed by WHO with an international group of experts, can be used. These criteria constitute general principles for ethical standards which could be adapted by governments. They apply to prescription and non- prescription drugs ("over the counter drugs") and to all informational and persuasive activities by manufacturers and distributors, the effect of which is to induce the prescription, supply, purchase and/or use of drugs (promotion, advertisements, medical representatives, etc.). Among other criteria, promotion should be reliable, accurate, truthful, informative, balanced, up to date and in good taste. Sources and methods of data collection: Information for the indicator will be available through review of documents (laws, regulations, etc.) to assess if regulations exist and through interviews with officials from the ministry of health and/or national drug authority to assess if controls are implemented. Indicators for monitoring national drug policies 76 Essential drug selection and drug registration Indicator ST12: Is there a national essential drugs list (EDL)/formulary using INN officially adopted and distributed countrywide? Use: To assess the existence of a key element of an efficient pharmaceutical supply system: the selection of drugs to meet the health needs of the population. Evidence suggests that drug selection and prioritization bring more advantages than disadvantages to public health. By focusing on fewer drugs, the efficiency of the procurement process can be improved, distribution is simplified, and provision of objective information is facilitated. Description: A national essential drugs list is defined as a booklet containing all the drugs approved for use in the public sector. In certain cases, there is one booklet which contains all the drugs agreed for all health care levels. In others, there are lists/booklets by level of use (tertiary, secondary, primary care). The booklet may contain additional information on each of the drugs. In certain countries the essential drugs lists may also apply to the private sector. For the indicator to be positive, the list should be officially approved by the ministry of health, should be written using INN and distributed widely in the public sector. The international nonproprietary name (INN) is the shortened scientific name based on the active ingredient; WHO is responsible for assigning INN to pharmaceutical substances. Sources and methods of data collection: The national essential drugs list/formulary is usually available from the ministry of health. Information on the use of INN and on the distribution of the list can be obtained from interviews with health personnel. Chapter V: Detailed presentation of indicators 77 Essential drug selection and drug registration Indicator ST13: Is there an official drug committee whose duties include updating the national essential drugs list (EDL)? Use: To assess if there is a formal mechanism in place to develop, adapt and update the national essential drugs list. To be well accepted by prescribers and the public the list should be revised regularly. A committee which will systematically collect information on new drugs, real health needs and demands from consumers will be an important tool in securing adherence to the list. Description: The committee for updating the national list of essential drugs can be a specific committee set up for this purpose or it can be a committee with other functions, such as development of standard treatment guidelines, quantification of drug needs, etc. The members of the committee should be officially appointed and include experts in clinical medicine, pharmacology, pharmacy and, where appropriate, nursing. Sources and methods of data collection: The official documents setting up the functions of the committee and its composition are usually available from the ministry of health and/or from the national drug authority. These functions are often included in the drug regulations. Limitations: The indicator does not measure the functioning of the committee. To provide useful information, it should be used in conjunction with structural indicators ST12 and ST14. Indicators for monitoring national drug policies 78 Essential drug selection and drug registration Indicator ST14: Has the national essential drugs list (EDL)/formulary been updated and distributed countrywide in the past five years? Use: To assess the continuous adaptation of the national essential drugs list to pharmacological, therapeutic and other changes. The list, which is often the heart of national drug policy and strategies for rational drug use, should be revised regularly in order to respond to evolving needs and pharmaceutical advances and to be well accepted by prescribers. Description: This indicator should be read in conjunction with structural indicators ST12 and ST13. If the answer to ST12 is "No", the response to this indicator will also be "No". If the answer to ST12 is "Yes", the additional information given by this indicator relates to the date of the last revision of the list. It is suggested, in WHO documents, that the list be revised every two years; however, in a number of countries this will be difficult to achieve. Five years seems a more realistic target, particularly in the least developed countries. Sources and methods of data collection: Information on the date of the last revision is usually available from the ministry of health and is normally given on the document containing the national essential drugs list. Chapter V: Detailed presentation of indicators 79 Essential drug selection and drug registration Indicator ST15: Do drug donations comply with the national essential drugs list (EDL)? Use: To assess the capacity of the government to ensure that organizations which donate drugs comply with the national drug policy. Drug donations often constitute a problem as they do not always match needs and in certain cases may even conflict with overall government drug policies. In many situations, donations containing only the most essential drugs included in national drugs lists would be the most appropriate. Description: The indicator should: (1) assess the existence of guidelines/regulations produced by the government which require that drug donations should consist of drugs included in the national essential drugs list; (2) check if the guidelines are respected for the main drug donations. Sources and methods of data collection: The information can be obtained through review of legal or administrative documents available from the ministry of health and through interviewing key staff at the ministry, in the central procurement unit and in the main NGOs and international organizations receiving or sending drug donations. If guidelines do not exist, it will still be feasible, through interviews, to assess if major drug donations comply with the national essential drugs list. Indicators for monitoring national drug policies 80 Essential drug selection and drug registration Indicator ST16: Are there formal procedures for registering drugs? Use: To assess if a proper registration procedure exists for drugs on the market. Drug registration is mandatory to ensure that drugs available on the market are of acceptable quality, safety and efficacy. To be effective this registration should follow a set of procedures. Description: The formal procedures for registering drugs can differ from one country to another but would most likely include the following elements: (i) well-established criteria (clauses on INN, cost, need, etc.) reflected in the regulations; (ii) provision in the legislation for temporary registration of drugs obtained by tender; (iii) availability of an application form for registration which should be completed by the manufacturer/importer and signed by the national drug regulatory authority; (iv) a list of the required documentation which should be submitted with each application; (v) a mechanism to assess the application form and a formal notice of approval or rejection. Sources and methods of data collection: The information will be available from the national drug authority and can be obtained through interviews and reviewing various drug registration documents. Chapter V: Detailed presentation of indicators 81 Essential drug selection and drug registration Indicator ST17: Is there a drug registration committee? Use: To assess if one of the mechanisms for proper drug registration is in place. In most well- organized registration systems, a committee is used to assist the national drug authority to assess the documentation accompanying each application for registration and prepare a brief appraisal for the ministry of health. Description: The drug registration committee is, in general, an advisory committee to the national drug regulatory agency on matters relating to registration of drugs and cancellation or suspension of such registration, based on evaluation of the quality, safety, efficacy and usefulness of the products. It should consist of people with the widest possible education, knowledge and experience in the field of pharmacy, medicine and pharmacology, such as pharmaceutical chemistry, pharmaceutical formulation, internal medicine, toxicology and clinical pharmacology. It should be granted the authority to request more information or documentation if necessary, to ensure the quality, efficacy and safety of the product and to seek the assistance of specialists in the various medical or pharmaceutical disciplines if needed. Sources and methods of data collection: Information on the drug registration committee is available from the ministry of health and/or from the national drug authority. Indicators for monitoring national drug policies 82 Essential drug selection and drug registration Indicator ST18: Is drug registration renewal required at least every five years? Use: To assess if there is a mechanism to regularly review the drugs which are allowed on the market. In most countries, manufacturers are requested to renew the registration of their drugs at regular intervals to ensure that the national drug authority is aware of the drugs available on the market (some may not be marketed any more). In the case of application for renewal of registration of a drug, the manufacturer/importer should be allowed to refer to previously submitted documentation and information on the product. Description: The provision about the period of time for which a drug is registered should be explicitly stipulated in the law/regulations. Sources and methods of data collection: The information will be available from the national drug authority and can be obtained through interviews and reviews of various legal and administrative documents (regulations, registration procedures, etc.). Limitations: In certain cases, the provision exists in the law/regulations but is not applied. Chapter V: Detailed presentation of indicators 83 Drug allocation in the health budget/public sector financing policy Indicator ST19: Is the public drug budget spent per year more than 20% of the ministry of health operating budget spent per year for the last three years? Use: To assess the commitment of the government to support the financing of drugs, with the ultimate objective of greater accessibility and equitable supply. For many years, the drug budget represented an important share of the ministry of health budget; however, many countries faced with the economic crisis had to reduce the share of the drug budget. It is still considered important to secure a contribution from the government (for instance through a percentage of the ministry of health budget) in order to ensure that the poorest are not deprived of drug treatments. The figure of 20% is based on review of country experiences; this figure can be adapted at national level according to the policy and targets of the country. Description: Public drug budget is defined as the total amount of money spent on pharmaceutical products by the government through the ministry of health, other ministries and hospitals. Public finance is understood as general government revenues and compulsory health insurance (sometimes known as social insurance) that is either publicly managed or heavily regulated by governments (see footnote 7, page 18). The operating budget of the ministry of health includes all expenditures which are not investments (i.e. salaries, drugs, national programmes, etc.). For this indicator, only the drug budget and the operating budget really spent should be taken into account. Sources and methods of data collection: The information is available, in most countries, in documents published by the ministry of finance and through interviews with government officials, including the procurement unit staff. The estimate must include drug budgets from all ministries. The indicator is obtained by dividing the sum of the drug expenditure for the last three years by the sum of the ministry of health operating budget spent for the same years, as below: Public drug budget spent in the past 3 years ––––––––––––––––––––––––––––––––––––––––––– MOH operating budget spent in the past 3 years x 100 The answer to the indicator is "Yes" if the figure obtained is above 20%. Indicators for monitoring national drug policies 84 Drug allocation in the health budget/public sector financing policy Indicator ST20: Is the public drug budget spent per capita per year more than US$ 1.00 per year for the last three years? Use: To assess the commitment of the government to support the financing of drugs, with the ultimate objective of greater accessibility and equitable supply. One US dollar per year has frequently been given as the lowest possible amount for minimum coverage of the population with essential drugs. It is still considered important to secure a contribution from the government in order to ensure that the poorest are not deprived of drug treatments. Description: Public drug budget is the same figure as in the previous indicator and is defined as the total amount of money spent on pharmaceutical products by the government through the ministry of health, other ministries and hospitals. Public finance is understood as general government revenues and compulsory health insurance (sometimes known as social insurance) that is either publicly managed or heavily regulated by governments (see footnote 7, page 18). In countries where the annual increase in population is high, it is important to get accurate annual estimates of this population. Sources and methods of data collection: The information is available, in most countries, in documents published by the ministry of finance and through interviews with government officials, including the procurement unit. The estimate must include budgets from all ministries. An estimate of the population for each year can be obtained from the planning office. The indicator is obtained by dividing the sum of the drug expenditure for the last three years by the sum of the population for the same years. Chapter V: Detailed presentation of indicators 85 Drug allocation in the health budget/public sector financing policy Indicator ST21: Is the public drug budget spent for national hospitals less than 40% of the total public drug budget spent for the last three years? Use: To assess the commitment of the government to a primary health care policy. Since the beginning of the 1980s, most countries have adopted a primary health care policy, which entails a reallocation of resources towards primary and secondary levels. For drugs, this means that the main part of the budget should go to primary health care institutions, rather than to national hospitals. In most countries it has been shown that it is common for 40% of the total drug budget to be directed to national hospitals — a proportion which is too high. Description: Public drug budget is defined as the total amount of money spent on pharmaceutical products by the government through the ministry of health, other ministries and hospitals. Public finance is understood as general government revenues and compulsory health insurance (sometimes known as social insurance) that is either publicly managed or heavily regulated by governments (see footnote 7, page 18). The operating budget of the ministry of health includes all expenditures which are not investments (i.e. salaries, drugs, national programmes, etc.). National hospitals are the main referral hospitals, normally situated in the capital city and in the main towns. In certain cases, the national hospitals have a separate budget. This separate budget should be added to the public drug budget as long as it is also funded from government revenues. Sources and methods of data collection: The information is available, in most countries, in documents published by the ministry of finance and through interviews with government officials. The estimates for public drug expenditure and for the national hospitals' drug expenditure must include expenditures from all ministries. The indicator is then obtained by dividing the sum of public drug expenditure for the national hospitals by the total public drug expenditure, and multiplying the result by 100. Indicators for monitoring national drug policies 86 Drug allocation in the health budget/public sector financing policy Indicator ST22: Has the public drug budget spent per capita increased in the last three years? Use: To assess the commitment of the government to support the financing of drugs with the ultimate objective of greater accessibility and equitable supply. The government drug budget, which is often very limited, should not decrease in real terms. With a growing population it should, on the contrary, increase over the years in order to cover at least the most vulnerable groups. Description: Public drug budget is defined as the total amount of money spent on pharmaceutical products by the government through the ministry of health, other ministries and hospitals. Public finance is understood as general government revenues and compulsory health insurance (sometimes known as social insurance) that is either publicly managed or heavily regulated by governments (see footnote 7, page 18). It should take into account the inflation rate (real terms) in order to obtain comparable figures for the last three years. Sources and methods of data collection: The information is available in most countries in documents published by the ministry of finance and through interviews with government officials, including the procurement unit. Data on population can be obtained from the planning office. The following data are needed: public drug expenditures for the current year and the previous three years (the estimate must include expenditure from all public sources at central and local levels); population for the current year and the previous three years; and the inflation rate for the current year and the previous three years. Donations should be included if they have been accounted for in the budget. It happens that drug donations are valued at a very high price (for instance, pharmaceutical companies often give drugs under brand name, and the price of these drugs may be much higher than existing alternatives on the market). In this case, it is desirable to calculate the value of the donation at the average price of the same product on the international market. Chapter V: Detailed presentation of indicators 87 Drug allocation in the health budget/public sector financing policy Indicator ST23: Are there any financing systems in addition to the public drug budget that contribute to the provision of drugs in the public sector? Use: To assess if there are other mechanisms to increase the provision of drugs in the public sector and therefore to increase coverage. Many countries have developed financing systems aimed at collecting additional revenues at the health facility level in order to buy drugs and cover a limited number of expenditures (cf. Bamako Initiative). In general, these systems allow patients to have access to essential drugs at a lower price than in the private sector. However, these schemes should not be the only source of finance for health care. Description: Financing systems are defined as any systems which, in the public sector, contribute to the provision of drugs by charging patients or the community. These systems can include out-of- pocket payments (e.g. user charges) or voluntary health insurance (see footnote 7, page 18). They can cover all or part of the costs of the drugs. The indicator should be analysed in conjunction with indicators ST19 and ST20. Sources and methods of data collection: The information can be easily obtained through interviews at the ministry of health. Indicators for monitoring national drug policies 88 Public sector procurement procedures Indicator ST24: Are drugs usually procured in the public sector through competitive tender? Use: To assess the capacity of the public sector to procure essential drugs at low cost. Many countries pay prices for drugs that are far above those on the international market because of the absence of a centralized purchasing system and procedures for tendering. By using such a tender system, the cost of drugs can be considerably reduced. Description: Competitive tender is defined as a procedure for procuring drugs which puts a number of suppliers into competition. Purchasing is done on the basis of quotations submitted by the suppliers in response to a public notice. Depending on the country situation, tenders can be open or restricted to a more limited list of already known suppliers, including local producers. Restricted tender is preferred by many countries as it is easier to manage, and evidence shows that it provides drugs of quality at low cost. The term "usually" quoted in the definition of the indicator means that most of the drug procurement (in value and in volume) is done by tender. Sources and methods of data collection: Information is available from the ministry of health or from the central procurement unit (CMS, etc.). Chapter V: Detailed presentation of indicators 89 Public sector procurement procedures Indicator ST25: Is there a system for monitoring supplier performance? Use: To assess the capacity of the procurement unit to effectively manage the procurement procedures. In order to obtain drugs of good quality and to be sure they will arrive in time, it is important to select suppliers carefully. A system which monitors the performance of suppliers over a number of years allows the identification of the best suppliers and assists in guaranteeing the smooth supply of drugs of acceptable quality. Description: A system for monitoring supplier performance is defined as a system which provides information on the past performances of each supplier. It should include information on delivery time, adherence to delivery instructions, packaging and labelling, expiring products policy, quality of the pharmaceutical products supplied, etc. Sources and methods of data collection: The information is available from the procurement unit and can be obtained through interviews and review of the system, in order to check if most of the necessary data are included. Indicators for monitoring national drug policies 90 Public sector procurement procedures Indicator ST26: Is most of the tendering done under international nonproprietary name (INN)? Use: To assess the commitment and the capacity of the public sector to procure drugs at low cost. The use of the international nonproprietary name is essential in drug procurement as it allows all the suppliers which manufacture a given substance to participate in the tender, and therefore reduces prices through increased competition. Description: Tender is defined as a procedure for procuring drugs which puts a number of suppliers into competition. Purchasing is done on the basis of quotations by the suppliers submitted in response to a public notice. INN is the shortened scientific name based on the active ingredient. WHO is responsible for assigning INN to pharmaceutical substances. The answer to the indicator is considered to be "Yes" if the vast majority of tendering measured in value (80% and above) is done under INN. Sources and methods of data collection: The information is available from the procurement unit and can be obtained through interviews and review of the most recent major tenders. Chapter V: Detailed presentation of indicators 91 Public sector procurement procedures Indicator ST27: Does the procurement unit receive foreign currency in less than 60 days (from request to release)? Use: To assess if the government is committed to efficient and relatively fast procurement of drugs. In a well-functioning procurement system, drugs should be bought from reliable suppliers by tender (see indicators ST24, ST25 and ST26). In addition, the foreign exchange needed should be made available in good time to the procuring unit. Otherwise, suppliers will not trust the procurement unit, supply will take a long time and shortages may occur. Description: In most countries, the procurement unit needs to request foreign currency from the government to pay for drugs. This indicator looks at the interval of time between the moment the request is made and the moment the foreign exchange is allocated. Sources and methods of data collection: The information is available from the procurement unit and from the ministry of finance, and can be obtained through interviews with the procurement staff and review of relevant documents. Limitations: In countries where the central procurement unit operates a revolving fund and can obtain foreign currency on the financial market and in countries where all drugs are procured locally, the indicator is not valid. Indicators for monitoring national drug policies 92 Public sector procurement procedures Indicator ST28: Is procurement in the public sector limited to drugs on the national essential drugs list (EDL)? Use: To assess the willingness of the government to implement good drug procurement practices and to rationalize the public sector drug supply system. A national list of essential drugs includes all the drugs needed in a given country to cover most therapeutic needs. Therefore procurement in the public sector should be limited to the list. In addition, by focusing on a reduced number of drugs, the efficiency of the procurement process can be significantly improved, discounts for bulk purchase obtained and quality control analysis undertaken more easily. Description: Drugs procured for the public sector include drugs for all health care levels. This means that procurement done directly by hospitals, by projects supported by bilateral or multilateral organizations or by vertical programmes (e.g. TB) should be reviewed and should, for the indicator to be positive, be in line with the national essential drugs list. Sources and methods of data collection: Information is normally available from the procurement unit and from the main procuring bodies (hospitals), and can be obtained through interviews and review of orders. Limitations: If the national essential drugs list has not been revised for a long time, it may be that some new drugs are procured which are necessary although not on the list. Rather than looking at the details, this indicator should look at the general policy in terms of procurement. If there is not a national list of essential drugs but only lists for certain levels of the health care system, the indicator will not be valid (see particularly ST12 to ST15). Chapter V: Detailed presentation of indicators 93 Public sector procurement procedures Indicator ST29: Is the average lead time (from order to receipt at central level) less than eight months? Use: To assess the efficiency of the procurement procedures. This indicator provides additional information on the procurement procedures. The procurement agency, if it performs well, should obtain drugs procured by international tender in a reasonable period of time. Experience shows that a period of longer than eight months between the time drugs have been ordered and delivery is indicative of poor performance by the procurement unit, as it increases the pressure on major stock items and the risk of shortages. Description: The lead time is defined as the number of months between the moment tenders have been issued and the moment drugs arrived in the country. It includes the bidding process, the selection of suppliers, etc. This average lead time should be calculated on the basis of main orders during the last three years. Sources and methods of data collection: Information is available from the procurement unit and can be obtained through interviews and review of the main orders during the last three years. Limitations: This indicator is not valid if, for a given reason, a lead time exceeding eight months has been formally agreed between the supplier and the procurement unit. In addition, delays can be due to the slow release of foreign currency and therefore are not the responsibility of the procurement unit (see indicator ST27). Indicators for monitoring national drug policies 94 Public sector procurement procedures Indicator ST30: Is procurement based on a reliable quantification of drug needs? Use: To assess the efficiency of procurement procedures. A far more complex task than selecting the drugs is deciding on the quantities needed. The procurement unit should be able to supply the correct quantities of drugs in order to prevent surplus and shortages at health facility level. In many countries, this quantification is not always easy because of poor recording systems, limited budget, irrational drug use, etc. Description: Reliable quantification of drug needs is defined as a careful evaluation of the quantities needed of each drug, based on either adjusted past consumption or anticipated pattern of diseases and standard treatment. In order to get a more precise answer, one can look at performances in previous years and see if shortages were common. If this is the case, one should try to identify the reasons for shortages, and one of them can be unreliable drug quantification. In countries where there is no central procurement unit but regional units, the indicator can be slightly modified and become "Is procurement based on a reliable quantification of drug needs in the majority of these units?". In the results, the indicator can be presented with detailed information on the number of units where procurement is based on a reliable quantification of drug needs. In countries where drugs are bought directly by the health facilities, the indicator can be obtained from a sample of health facilities, for instance when conducting surveys for process or outcome indicators. Data Collection Form 2 (Annex 1) can be adapted and the result will be presented as a percentage: percentage of health facilities surveyed where procurement was based on a reliable quantification of drug needs. Sources and methods of data collection: Information is available from the procurement unit and/or th

View the publication